MXPA06007738A - Method and apparatus for controlling the insertion of additional fields or frames into a first format picture sequence in order to construct therefrom a second format picture sequence - Google Patents

Abstract

The major TV systems in the world use interlaced scanning and either 50Hz field frequency or 60Hz field frequency. However, movies are produced in 24Hz frame frequency and progressive scanning, which format will be used for future digital video discs to be sold in 50Hz countries. In 50Hz display devices the disc content is presented with the original audio pitch but with repeated video frames or fields in order to achieve on average the original video source speed. However, the frame or field insertion is not carried out in a regular pattern but adaptively in order to reduce visible motion judder.

Description

METHOD AND APPARATUS FOR CONTROLLING FIELD INSERTION ADDITIONAL OR FRAMES IN A FIRST FORMAT SEQUENCE OF IMAGE, TO BUILD A SECOND SEQUENCE OF IMAGE FORMAT FROM IT DESCRIPTION OF THE INVENTION The invention relates to a method and an apparatus for controlling the insertion of additional fields or frames in a first image format sequence for example 24 progressive frames per second in order to build from them a second one. sequence of image format you have, for example 25 frames per second.

BACKGROUND The major TV systems in the world use interlaced scanning and either a 50 Hz field frequency (for example in Europe and China for PAL and SECAM) or 60 Hz or almost 60 Hz field frequency (for example in United States and Japan for NTSC), indicated as 50i and 60i, respectively. However, the films are produced at a frame frequency of 24 Hz and progressive scan, indicated as 24p, which value, when expressed in interlaced format, would correspond to 48i. Up to now, the film conversion, from 24p to 60Hz interlace presentation, is handled by a decrease by "pull 3: 2" as shown in Figure 2, in which the decrease by pull 3: 2 of a field is insert by field repetition every five fields. The interlaced ILF fields are derived from the original ORGFF movie frames. From a first original film frame OFR1 three output fields OF1 to OF3 are generated, and from a third original film frame 0RF3, three output fields OF6 to OF8 are generated. From a second original film frame OFR2 two output fields OF4 and OF5 are generated, and from a fourth original film frame OFR4 two output fields OF9 and OF10 are generated and so on. It is desirable that the distribution means have a unique video format and an audio track which are susceptible to being reproduced throughout the world instead of the current situation where there is at least one version at 50 Hz and one at 60 Hz of each packaged media title, for example DVD. Because many sources consist of 24 fps (frames per second) movies, this 24 p format is preferably the desired format for such single format video tracks, which format therefore needs to be adapted at the time of playback for presentation correctly in presentation devices, in countries with 5.0 Hz as with 60 Hz. The following solutions are known for conversion from 24p to 25p or 50i or, more generally, for conversion to 25 fps: - 4.2% faster re-playback: this changes the length of the content and requires a costly real-time audio track conversion and therefore is not applicable for consumer products. It is true that the diffusion of a current movie and the DVD apply this solution for video, but the conversion of speed or audio step required is dealt with beforehand by the content provider side so that on the consumer side there is no conversion is required - audio step. DVD video discs sold in countries at 50 Hz contain audio data streams that are already encoded so that the DVD playback decoder automatically transmits the correct speed or step of the audio signal. apply a regular field / frame duplication scheme: this solution leads to an unacceptable regular movement jump and, therefore, practice is not applied. apply compensated frame rate conversion: this is a generic solution to such conversion problems which is very expensive and therefore not applicable to consumer products.

DESCRIPTION OF THE INVENTION Currently, a conversion of an original 24p format video film and audio data stream to a 50 Hz interlace display is carried out by re-playing the film approximately 4% faster. However, this means that in the 50 Hz countries, the artistic content of the film (its duration, the passage of the voices), is modified. Field / frame repeat schemes similar to 3: 2 pulldown are not used because they show unacceptable motion jump artifacts when applied in a regular manner, such as the insertion of an additional field every 12 frames. A problem to be solved by the invention is to provide a framework or field insertion scheme for converting the 24p format to the 25 fps format in an improved manner and thereby minimizing motion jump artifacts. This problem is solved by the method described in claim 1. In claim 2 an apparatus using this method is described. The characteristics of a current movie scene such as a global movement, brightness level, intensity and scene change places are evaluated in order to apply duplicate or repeated frames / fields in subjectively non-obtrusive places. In other words, the invention uses readily available information about the source material to be converted from 24p to 25 fps to adaptively insert repeated fields / frames in non-equidistant places where the resulting insertion artifacts are minimal. Advantageously, the invention can be used for all frame rate conversion problems where there is a small difference between the source frame rate and the target frame rate. If the frame rates differ greatly, such as in the 24 fps conversion at 30 fps, there is hardly any freedom for movement in fields or time frames to be repeated. The invention facilitates the computationally inexpensive conversion of image sequences of 24 fps format at 25 fps (example values) with minimized movement jumps. In principle, the method of the invention is suitable for controlling the insertion of additional fields or frames within a first image format sequence in order to build from it a second image format sequence, whose frame frequency is constant and is greater than that of the first format image sequence, the method includes the steps of: - determining places of fields or frames in the first sequence of image format in which the insertion locations of a field or frame corresponding additional causes a minimum visible movement jump in the second image format sequence; inserting in the first sequence of image format a field or a frame in some of said places at non-regular field or frame insertion distances in such a way that the average distance between any of the adjacent frames corresponds to that of the second sequence of format image; - - presenting the first image format sequence together with the fields and / or frames inserted in a non-regular manner in the format of the second image format sequence. In principle, the apparatus of the invention is suitable for controlling the insertion of additional fields or frames in a first image format sequence in order to build from them a second image format sequence, the frame frequency of which is constant and is larger than the first image format sequence, the apparatus includes a means which is adapted to determine the positions of the fields or frames in the first image format sequence at positions which the insertion of a field or corresponding additional frame causes a minimum visible movement jump in the second image format sequence, and to insert in the first image format sequence a field or a frame in some of the places at field or frame insertion distances not regular in such a way that in total the average distance between any of the adjacent frames corresponds to that of the second image format sequence n, and to represent the first image format sequence together with the fields and / or frames inserted in a non-regular manner in the format of the second image format sequence. The further advantageous embodiments of the invention are described in the respective dependent claims.

DRAWINGS The exemplary embodiments of the invention are described with reference to the accompanying drawings, which show: in Figure 1, a simplified block diagram of a disc player of the invention; in Figure 2, the application of the pulldown by 3: 2 in a 24p source image sequence to provide an image sequence 60i; in figure 3, a regular pattern of repeated frames; in figure 4, a regular pattern of repeated fields; in figure 5, a line of ... time for regular frame repetition, according to figure 3; in figure 6, example movement jump tolerance values of a video sequence; in Figures 7a and 7b, irregular example temporary locations for field or frame repetition and the resulting variation display delay; in figure 8, the frame or field repetition distance expressed as a function of the video delay and movement jump tolerance; in Figure 9, the frame or field repeat distance function of Figure 8, by means of which the maximum and minimum video delay depends on the required degree of lip synchronization; in figure 10, format frames of 2-4 fps that - include a repeating frame without motion compensation; in figure 11, a frame output of 25 fps format related to figure 10; in Figure 12, 24 fps format frames that include a repeating frame with motion compensation; in Figure 13, a transmitted frame format of 25 fps related to Figure 12.

EXEMPLARY MODALITIES Figure 1 presents a disk unit that includes a PEC of capture stage in error correction that reads a 24p format of a video and audio signal encoded from a disk D. The output signal passes through of a track buffer and a demultiplexer stage TBM to a VD? C video decoder and an ADEC audio decoder, respectively. A CTRL controller can control PEC, TBM, VDEC and ADEC. A user interface Ul or an IF interconnection between a TV receiver or a screen (not shown) and the disc player is used to switch the playback output in either 24 fps or 25 fps mode. The IF interconnection can automatically verify which mode or modes can be processed and presented by the TV receiver or the screen. The playback mode information is automatically derived from the characteristics data (ie, data about which the presentation mode is available on the TV receiver or on the screen) received by the connecting interconnection IF by a wire, by radio waves or optically to the TV receiver or the presentation device. The feature data can be received regularly by the IF interconnection or by sending a corresponding request to the TV receiver or the presentation device. As an alternative, the information of the reproduction mode is introduced by the user interface Ul upon presentation of a corresponding request for a user. In the case of a 25 fps output from the VDEC video decoder the CTRL controller, or the VDEC video decoder itself, determines from the characteristics of the decoded video signal in which of the temporary places it is to be repeated a field or frame by the video decoder. In some embodiments of the invention these temporary locations are also controlled by the audio signal or signals that come from the ADEC audio decoder, as explained in the following. Instead of a disc player, the invention can also be used in other devices, for example, a digital receiver / decoder or a digital TV receiver, in which case the front end which includes the disk unit and the memory Intermediate track is replaced by a tuner for digital signals. Figure 3 shows a regular pattern of repeated frames where one frame is repeated every 24 frames, that is, in tn, tn + l, tn + 2, tn + 3, etc. seconds, to obtain a known conversion of 24p at 25 fps. Figure 4 shows a regular pattern of repeated fields in which a field is repeated every 24 fields, that is, in tn, tn + 0.5, tn + l, tn + 1.5, tn + 2, etc. seconds to get a known conversion of 24p at 25 fps. This kind of processing is applicable if the display device has an interlaced output. The number of locations on the time axis when a jump occurs is doubled, but the intensity of each "jump instance" is halved compared to the frame repeat. The upper fields are derived from the first, third, fifth, etc. line of the indicated frame of the source sequence and the lower fields are derived from the second, fourth, sixth, etc. line of the indicated frame of the source sequence. Figure 5 shows a timeline for a regular frame repeat according to Figure 3, with the markers at the temporary places tn tn + l, tn + 2, tn + 3, etc. seconds where the frame repetition occurs.

- In order to carry out the 'adaptive insertion of the invention of the repeated fields or frames in non-equidistant (or irregular) locations, corresponding control information of the locations is required. The content information and the image signal characteristics about the source material are available as soon as the image sequence is compressed by a scheme such as MPEG-2 video, MPEG-4 video or MPEG-4 video part 10, which will supposedly be used not only for the dissemination of the current generation and packaged media such as DVD, but also for future media such as discs based on blue laser technology. The characteristics of image signal or information that is useful in the context of this invention are: the generated or transmitted motion vectors, scene change information generated by an encoder, - information of brightness or average intensity, which can be derived from analyzing the transformed coefficients of DC, • average texture resistance information, which is. it can be derived from analyzing the transform coefficients of AC.

- Such image signal characteristics can be transferred from the day encoder to a disc or via broadcast to the decoder as MPEG user data or private data. Alternatively, the video decoder may collect or calculate and provide said information. In order to take advantage of the moving vector information, the set of motion vectors MV for each frame is collected and processed so that it can be determined if a current frame has visibly large moving areas, since those areas suffer from the greatest part of the movement jump when frames or fields are duplicated. To determine the presence of such areas, the length of the average absolute vector AvgMVi can be calculated for a frame as an indication for a panning motion: ^ VX- \ VY- \ AvgMV = YY Wr v \, (1) VX - VY x = 0 y = where "i" indicates the frame number, "VX" and "VY" are the number of motion vectors in the x direction (horizontal) e and (vertical) of the image. Therefore VX and VY are typically obtained by dividing the image size in the respective direction between the block size for motion calculation. If the motion vectors within a point 1 of frame with different frame of reference in temporary distance different to the current frame, additionally a normalization factor RDistx is required, and for this distance:. rx-i vr-i \ MV I In another embodiment, using a more complex processing, a movement segmentation of each image is calculated, that is, one or more groups of adjacent blocks are determined that have motion vectors of similar length and direction in order to detect areas of movement sufficiently large multiples with different directions of movement. In this case, the average movement vector can be calculated, for example, by: «Groups Y? VgMVc • Group size c Avg? MV¡ '= c = l n" Groups, "(2a) Group size c where AvgMVc is the length of the average movement vector for the group "c" identified. Advantageously, this approach eliminates the effect of motion vectors for small objects that move randomly within an image that is not a member of any identified block group movement and that do not visibly contribute significantly to the movement jump. The processing can take into account weighting factors for AvgMV_ where the movement areas are strongly textured or have defined edges, since this also increases the visibility of the movement jump. The information about the texture resistance can be reviewed more conveniently from a statistical analysis of the transmitted or reproduced AC transform coefficients transmitted for the prediction error. At first, the texture strength can be determined from analysis of an original image block, however, in many cases, such strongly textured blocks, after encoding using compensated motion prediction will also have a higher predictive error energy in their AC coefficients, compared to less textured blocks. The jump jump tolerance MJT in a specific temporal location of the video sequence can therefore be expressed as: MJT = f (AvgMV, texture resistance, edge resistance) (3) with the following general characteristics: given values fixed texture resistance and edge resistance, MJT is proportional to 1 / AvgMV; given fixed AvgMV values and edge strength, MJT is proportional to 1 / (texture strength); Given fixed values of AvgMV and strength of texture, MJT is proportional to 1 / (edge resistance). Figure 6 shows example movement jump tolerance values MJT (t) over a source sequence. Preferably, the current size of the movement jump tolerance value influences the distribution, as shown in Figure 7a, of the inserted repeated frames or the fields within the resulting 25 fps sequence, i.e. the FRD distance of repetition of frame or field. The early or delayed insertion of repeated frames causes a negative or positive delay of the audio track in relation to the video track, as indicated in Figure 7b, that is, a variable display delay for video. A maximum tolerable video delay in relation to the audio in both. addresses is considered when the mapping of a motion jump tolerance MJT is applied for the frame or field repeat FRD distance. A possible solution for this control problem is shown in Figure 8. The frame or field repeat FRD distance is expressed as a function of the video delay VD and the MJT tolerance of the motion jump: FRD = f (VD , MJT), (4) with the following general characteristics: Given a fixed value of VD, FRD is proportional to / MJT; Given a fixed value of MJT, FRD is proportional to l / VD; This relationship can be expressed in a characteristic of FRD = f (VD) that changes depending on the tolerance value of the movement jump, as it is the case in figure 8, which favors more prolonged separations than the optimum between repeated and fenced frames in case of a low tolerance of jump of movement (that is to say, a high degree of movement) and favors shorter separations than the optimal ones in case of a tolerance of jump of movement high (for example, degree of movement lower than the average) . The field or optimal frame repeat distance is displayed as FRDopt- The maximum allowable video delay is shown as VDmax. The maximum allowable video delay in the negative direction is shown as VDmin. Since a short-frame freeze effect in places of scene change is not considered uncomfortable, the scene change information generated by a video encoder (or by a video decoder) can be used to insert one or more Repeated fields or frames in such places, the number of repetitions depends on the current degree of video delay. For the same reasons, repeated fields or frames can be inserted after an extinction sequence to black, in a target extinction sequence or an extinction to any color. All of such unique locations have a very high value of MJT. Remarkably repeated frames can be used in such places even if other fields containing image can only be repeated in order to reduce the intensity of movement jump in individual places. Generally, repeated frames and repeated fields can coexist in a converted image sequence. The delay joints typically accepted for perceived lip synchronization need only be observed if at least one speaker is actually visible within the scene. Therefore, the delay between the presentation of audio and video may become larger than the aforementioned unions as long as the speaker is not visible. This is the case typically during fast moving scenes. Therefore, an additional control can be carried out, as shown in Figure 9 wherein the video delay links VDmin and VDmax are changed or subjected to uniform transition between: acceptable lip synchronization values VDmiIlLipSyn < , and VDmaxLipSyr? c if the speed of short sound peaks (which are caused by special events such as a door being whipped) are detected and a scene with slow or static movement is also detected; otherwise, the larger VD values are VDmin and VDmax. A speech detection can be varied, for example, in the case of the multiple channel audio used mostly when evaluating the center channel in relation to the left and right channels, since the speech in the movies is encoded mainly within the central channel . If the central channel shows a sudden energy distribution with respect to time that is significantly different from the energy distribution in the left and right channels, then the probability that speech is present is high. The entirety of the above controls for adaptively determining the local frame repeat distance work for a single pass through a video sequence. However, the control of the invention benefits from an encoding processing - 0 of two passes as it takes place in many professional MPEG-2 encoders. In that case, where the first pass is used to collect the movement intensity curve, the scene cut locations and the count, number and location as well as the length of the scenes that require close lip synchronization, black frames, etc. , then a modified control can be applied that not only takes into account the information available for the current processed frame and its passage, but also for past and future neighboring frames: FRD (i) = f (VD, MJT (ik) .. MJT (i + k)), (5) where "i" indicates the current frame number and "k" indicates a reference number that refers to the adjacent frames. A general characteristic of each such function is that FRD is increased if MJT (i) is smaller than the surrounding MJT values and decreases if MJ (i) is larger than the surrounding MJT values. The related image signal characteristics can be transferred as MPEG user data or private encoder data via a disk or via a broadcast signal to the decoder. In another embodiment of the invention, compensated motion interpolation of the frames can be applied under specific circumstances instead of repetition of the frames without computational expense. Such compensated motion interpolation can make use of the motion vectors transmitted for the current frame. In general, these motion vectors are not suitable for motion compensated frame interpolation since they are optimized for optimal prediction gain instead of indicating the true motion of a scene. However, if a decoder analysis of the vectors of motion received shows that homogeneous panning of the scene occurs, a highly precise frame can be interpolated between a previous frame and the previous frame. Panning means that all motion vectors within a frame are identical or nearly identical in length and orientation. Therefore, an interpolated frame can be generated by moving the previous frame by half the distance indicated by the average motion vector of the current frame. It is assumed that the previous frame is the frame of reference for the compensated motion prediction of the current frame and that the interpolated frame is placed equidistantly between the previous frame and the current frame. If the prediction framework is not the previous framework, an appropriate scaling of the mean movement vector will be applied. The corresponding considerations are valid for the case where an approach can be determined from the received motion vectors. An approach is characterized by vectors of zero movement at the center of the approach and an increasing length in the movement vectors directed to the center (opposite) around this center of approach, the length of the movement vector increases in relation to the distance from the center of approach. Advantageously, this kind of motion compensated interpolation generates an improved motion jump behavior compared to the repetition of a frame, as illustrated in Figures 10 to 13. Figure 10 in the 24 fps format and Figure 11 after The conversion of the format at 25 fps shows frames (indicated as vertical bars) with a movement path for a vertically moving object and an instance of frame repetition, which results in a "freeze frame". Figure 12 shows the insertion of an interpolated motion frame which, when presented at an increased frame rate of 25 fps, as shown in Figure 13, leads to a "slower motion frame" instead of a "freezing frame" The controls described above for frame or field repetition and interpolation for frame rate conversion can be applied, both, in the encoder and on the decoder side of an MPEG-2 (or similar) compression system since most of the side information is available on both sides, possibly except for a reliable scene change indication. However, in order to take advantage of the knowledge of the upper image sequence characteristics of the encoder, the locations for the fields or frames to be repeated or interpolated can be transported in the compressed 24 fps video signal.

(MPEG-2 or otherwise). The flags to indicate the temporal order of the fields (top_field_first) and the repetition of the first field for presentation (repeat_first_field) exist in advance in the syntax of MPEG-2. If it is required to indicate the conversion pattern for the conversions of both 24 fps at 30 fps and 24 fps at fps for the same video signal, one of two sets of indicators can be transported in a suitable user data field for each image. The values 24 fps and 25 fps and the other numbers mentioned in the above are values of examples which can be adapted correspondingly to other applications of the invention. The invention can be applied to: packaged media (DVD, blue laser discs, etc.), downloaded media including video as requested, almost video where requested, etc., - broadcast media. The invention can be applied to an optical disc player or an optical disc recorder, or a hard disk recorder, for example an HDD recorder or a PC, or a receiver / decoder or a TV receiver.

Claims (11)

1. Method for controlling the insertion of additional fields or frames within a first image format sequence having a frame frequency of, for example, essentially 24 Hz in order to build from it a second format image sequence whose frame frequency is constant and is greater than the first image format sequence, for example 50 Hz, the method includes the steps of: - determining locations of fields or frames in the first image format sequence in places in which the insertion of a corresponding additional field or frame causes a minimum visible movement jump in the second image format sequence; inserting in the first sequence of image format a field or a frame in some of the places at non-regular field or frame insertion distances such that the total average distance between any adjacent frame corresponding to said second format sequence of image; presenting the first image format sequence together with the fields or frames inserted in a non-regular manner in the format of the second image format sequence, characterized in that the insertion locations of the field or frame in the first image format sequence is control so that, in order to gain lip synchronization is perceived in the second image format sequence, the maximum image content delay caused by the insertion irregularity remains lower than the average in the case of a scene of slow or static movement and the audio information assigned to the first image format sequence is detected.

2. Apparatus for controlling the insertion of additional fields or frames in a first image format sequence in order to build from it a second image format sequence, the frame frequency for example essentially 24 Hz, the which is constant that of the first image format sequence, for example 50 Hz, the apparatus includes a means that is adapted to determine places or fields or frames in the first image format sequence in which the locations of the insert of a corresponding additional field or frame causes a minimum visible movement jump in the second image format sequence, and to insert in the first image format sequence a field or a frame in some of the places in field insertion distances or non-regular frames in such a way that in total the average distance between adjacent frames corresponds to that of the second sequence of image format, and to present the The image format sequence together with the fields or frames inserted in a non-regular manner in the format of the second image format sequence, characterized in that the field or frame insertion locations in the first image format sequence are controlled by the medium so that, in order to gain perceived lip synchronization, in the second image format sequence in maximum delay of image content caused by insertion irregularity is kept lower than an average in case of a motion scene slow or static and speech is detected in the audio information assigned to the first image format sequence.

Apparatus as described in claim 2, wherein the apparatus is an optical disc player or an optical disc recorder, or a hard disk recorder, for example an HDD recorder or a PC or a receiver / decoder or a TV receiver.

Apparatus as described in claim 2 or 3, wherein the apparatus is an optical disc player or an optical disc recorder or a hard disk recorder or a receiver / decoder, wherein the apparatus transmits either the first sequence of original image format or the second sequence of image format, choice which is controlled by the reproduction mode information received either automatically from an interconnection that is connected to a device that includes a presentation device, or that is received from a user interconnection.

Method as described in claim 1, or apparatus according to claims 2 to 4, wherein the speech in the audio information assigned to the first image format sequence is detected when evaluating in multi-channel audio, if the central channel in relation to the left and right channels shows a distribution of discharge energy over time and is significantly different from the distribution of energy in the left and right channels.

Method as described in claim 1 or 5, or apparatus as described in one of claims 2 to 5, wherein the first image format sequence is stored or recorded in a storage medium, for example a optical disc or a hard disk, or it is broadcast or transferred as a digital TV signal.

Method as described in one of claims 1, 5 and 6, or apparatus as is. described in one of claims 2 to 6, wherein the field or frame insertion locations in the first image format sequence are frames or fields that do not contain large areas of moving image content, the movement is determined when evaluating motion vectors.

Method as described in one of claims 1 and 5 to 7, or apparatus as described in one of claims 2 to 7, wherein the field or frame insertion locations in the first image format sequence are frames or fields in which the scene changes or is extinguished to black or extinguished to white or extinguished to any other color.

Method as described in one of claims 1 and 5 to 8, or apparatus as described in one of claims 2 to 8, wherein the inserted fields or frames are offset in motion before being transmitted in a second sequence. of image format.

Method as described in one of claims 1 and 5 to 9, or apparatus as described in one of claims 2 to 9, wherein the first image format sequence is an MPEG-2 image sequence and wherein the insertion of the fields or frames in the first image format sequence is controlled by evaluating indicators either to indicate temporal order of the fields or to indicate repetition of the first field for presentation, indicators which are transported in a first format sequence of image in a user data field for each image. -

11. Method for providing on the encoder side a decoder side control of the insertion of additional fields or frames in an MPEG-2 image sequence having a frame sequence of, for example, essentially 24 Hz in order to build from of the same an image sequence, the frame frequency of which is greater, for example 50 Hz, where the places of insertion of field or frame in the image sequence will be controlled by transported indicators so that, with In order to obtain perceived lip synchronization, the maximum delay of image content caused by the irregularity of insertion is kept lower than the average in case there is a scene with slow or static movement as well as speech in audio information assigned to the sequence of image, the method includes the-stage of inserting, for each image in the image sequence, in a user data field either indicators to indicate order of fields Individuals or indicators to indicate repetition of the first field for presentation.