DE102005012668A1 - Method for detecting missing frames in audio and video data streams in e.g. hard disk, involves making assumption by frames missing from located frame, where rough time of frames is smaller than or equal to time to output frames - Google Patents

Abstract

The method involves adding an offset of audio and video data streams to a rough time of frames such that the offset is determined from synchronization information in packet headers. The time and the offset are compared with the information for each located frame. The information are obtained from presentation time stamp and/or system clock reference. An assumption is made by the frames missing from the located frame when the time and the offset are smaller than or equal to the time stamp and/or clock reference, where the time is smaller than or equal to a time for outputting of the frames. An independent claim is also included for a device for detecting missing frames in audio and video data streams.

Description

Object of the invention

These The invention relates to a method for real-time or for later Detection of sections where in the digital recording or transmission of image and / or sound sequences by means of digital, compressed Video system streams consisting of linked video and audio tracks Parts of these sequences have been lost. This invention relates a method for improving such digital video system streams in which Lost image and / or sound sequences are detected.

Technical field of the invention

The digital recording of video data including the audio data associated therewith produces big ones Amounts of data. Those for their storage for consumer electronics products to disposal stationary mass storage (eg hard disk or optical media like DVD) at the time of filing this patent application usually neither one sufficient access speed, nor the necessary storage density, to an uncompressed storage of digital video data and if necessary, the associated Allow audio data at a sufficient resolution. Therefore, must be digital recorded video data is compressed before being stored or transferred become. This compression of both video and audio data is called encoding. Internationally standardized for this Purpose became the protocols and formats of the Motion Pictures Expert Group (MPEG, eg MPEG 1, MPEG 2, or MPEG 4). Industry standards, derived from these protocols with only minor changes, are z. DivX (based on MPEG 4) or the DVD Whitebook (standard for DVD's, based on MPEG 2). Bigger differences exist z. To Quicktime and MJPEG, though these formats too partly based on the MPEG standards or as part of the product improvement Have implemented properties of the MPEG standards. The names these standards or industry standards are partially registered Trademark.

core The video data formats is the compression method. Place next to it the video data formats also determine how the video belonging to a video and audio data in a file (for storing) or in a data stream (for transmission) (multiplexed) and define additional control information.

at the MPEG formats and most other digital video formats The video and audio data are initially separated as their own Video and audio streams digitized or digitally recorded (or hardware encoders at least buffered) and encoded. When encoding will be the video and audio data streams in turn into individual sections, which are referred to as frames (engl. there A video frame usually corresponds to a picture of the movie that as video was recorded while an audio frame corresponds to a fixed length sound sequence. The duration of the sound sequence contained in an audio frame is different depending on the one used Audio format varies, but it usually moves in the tens of milliseconds range.

Within the frame, the audio and video raw data is contained in compressed form. If the compression was done at a constant data rate, the frame has a fixed length, which can be calculated from the data rate contained in its header. If the compression has been done with variable data rate, the frame has a variable length. In this case, the detection of the start of a frame can be done with the data contained in the header. For example, the in 2 represented Picture Header, which is the header of a video frame, with a 24-bit synchronization bit sequence (0x000001), followed by the identification byte of the Picture Header (0x00). The MPEG format ensures that no bit sequence similar to the synchronization bit string can occur within the data stream. This ensures that if one searches within the video data stream for a synchronization bit sequence, one finds the beginning of a header. The headers of the audio streams also use a sync bit string, but this is different depending on the audio data format. Thus, in the Dolby Digital audio data format, the synchronization bit string is 16 bits long and has the value 0x0B77.

Before the digital video stored on a mass storage, terrestrial or broadcast by satellite, or via a transmission medium such. B. to broadcast the internet or a network will have to the frames of the audio and video streams are tied together again become. This binding of data streams is called multiplexing. At the Multiplexing will also provide additional control information, such as B. the playback time of a frame generated and the data stream added. The result of the multiplexing is a so-called system stream (System Stream), this is also here as bound digital video, bound Called stream or bound video.

The multiplexing method works with the MPEG formats or the formats derived therefrom, regardless of which Transmission or storage medium the video is multiplexed, in principle in the same way. However, depending on the transmission or storage medium for which the video is multiplexed, the individual parameters of this method can vary widely. So z. For example, in DVD-Video, only one video stream is allowed, while in the DVB-T format used for the terrestrial transmission of digital television, four video streams are typically multiplexed.

At the The individual video and audio streams are multiplexed which the system stream should consist of, first divided into packets of the same size. The size of the packages depending on the transmission or storage medium selected from certain allowable values. Each packet is preceded by a so-called "packet header", on the one "packetetized Elementary Stream Header "(PES header) follows. Behind these two headers will be data of the video or Inserted audio stream, until the package is full. Then a new package begins. To the hardware requirements to that for To limit the output of digital video devices used, the Packets of the various data streams to be connected alternately in a ratio corresponding to their data rate in the multiplexing written data stream written. This allows multiplexing as well z. B. in hardware encoder encoders in one step with the encoding and done in real time. For the detection of the headers generated during the multiplexing is performed by the the video data streams known synchronization bit sequence 0x000001 used.

The packet header is followed by the identification byte of the packet header (0xBA) on the 24-bit synchronization bit string (0x000001). The packet header ( 3 ) contains among other things the "SYSTEM CLOCK REFERENCE" (SCR). If a player evaluates this field, it will not start playing the first frame in that packet until a 27 MHz counter started at the start of the video has reached the value contained in the SCR. The SCR is a 48-bit field that follows just after the identification byte of the pack header. The first byte of the SCR field is a 0, further bits of the SCR field are pre-assigned a 1 to ensure that no second sync bit sequence can occur within the header. The value of the 27 MHz counter is divided by 300 and written to the front of the SCR, the remainder resulting from the division being written as a 9-bit number at the end of the SCR. The structure of the packet header can 3 be removed.

The Pack Header is followed by a Packetetized Elementary Stream Header (PES header, 6 ). This starts again with the 24-bit synchronization bit sequence (0x000001). This is followed by the 8-bit stream ID. Permitted stream IDs are the values 0xBD to 0xEF. Each stream contained in the bound stream is assigned a unique stream ID at which it can be identified. The stream ID also determines what the data stream is used for. In this case, the stream IDs 0xE0 to 0xEF for video streams and the streams 0xC0 to 0xDF for audio streams are reserved in one of the original audio formats of the MPEG protocol family. Furthermore, a padding data stream (Stream-ID 0xBE) is allowed for inserting stuffing data and two private data streams (Stream-ID's 0xBD and 0xBF). For example, DVDs use Private Stream 0xBD to wrap audio data in DTS (Digital Theater Sound), AC3 (Dolby Digital), and PCM audio formats not supported by the original MPEG standard.

in the Packetized Elementary Stream Headers are a set of fields allowed. Which of them actually is present, the bits in a 32-bit bit field, that as a "PES header Extension " becomes. This bit field directly follows the stream ID.

For the understanding of inventive method is just the field "PRESENTATION TIME STAMP "(PTS) important to the PES header. It is present if the 7th bit of the "PES header Extension "the Value 1 has. In this case, the PTS directly follows the PES headers Extension. The PTS indicates the time in milliseconds at which, measured from the start of the video, a player playing the first Start frame in the packet that precedes this PES header should.

The structure of the PES header can, as far as it is required for the inventive method, 6 be removed.

The invention underlying problem

The MPEG formats are consistent in that the video and audio streams on each possible transmission or storage medium can be in the same form, so far one does not fall within this format family in another format replaced. In contrast, the system flows differ depending on the medium, for that they were created in relation to the information added in the binding often considerably.

Take, for example, a digital video with a TV tuner card and a PC, so a program stream is generated. In order to burn this recording to DVD for later archiving, it must be at least rebounded because the format (VOB - Video OBject) used on DVD's for the system stream is different from that of the Program Streams is different. On the contrary, the video and audio data streams bound in the various system streams are mutually exclusive. So theoretically you do not have to encode them anymore, at least if you have already taken into account certain boundary conditions (eg support DVD's only a few resolutions) in the settings for the recording. This is particularly advantageous because the MPEG formats use lossy compression techniques so that quality degradation occurs each time the video and audio data streams are re-encoded.

Especially in the MPEG-2 format there are a whole range of use cases, in where such a format conversion of the system stream by renewed Multiplexing is required in which the video and audio streams are not absolutely have to be re-encoded. Because the MPEG-2 format especially common is used, it has a corresponding number of derivatives. Another one Example of the Necessity of such a format conversion is to use a digital in DVB-S or DVB-T format broadcast on one DVD to archive. If you have a video that you z. B. as a transport stream received on a DVB receiver with built-in hard drive, burn on DVD is due to the additional navigation data, which requires the DVD in Private Stream 2 (Stream-ID 0xBF), basically one new multiplexing as well as generating additional control information required.

Becomes the video is recorded on the PC (eg with an analogue TV card) and further processed there, the encoding, cutting and, if it is to be burned to DVD, the design of the menus and the Burning often with different programs. These programs run each internally demultiplexing the system stream and multiplexing him again after further processing. Some of these programs can even do not process system streams tied from video and audio streams. In this case the video must be demultiplexed before further processing and to the subsequent Storage be multiplexed again. The only one known to the author Program that deviates from this is an external hardware encoder instructed version of a video editing program.

So long in the audio and video streams included in the source video are missing, no frames, this demultiplexing is followed by renewed Multiplexing easily. However, if frames are missing in one of the data streams, are after demultiplexing followed by re-multiplexing Audio and video for a part of the video is no longer synchronous. Already a difference less than half a second is distracting.

reason for that is, that all information about the time at which to start playing a frame should be multiplexed only in the packet headers and the PES headers Videos are included. If a video is played with missing frames, Wait for the player or device to play until the time of play the next, is reached again existing frames. Until then, the picture remains stand briefly, or the sound is temporarily mute.

At the Demultiplexing becomes these timestamps or synchronization information removed, so they no longer correct when multiplexing again can be formed. Instead, the multiplexer calculates the timestamps from the number the one counted by him Frames and some other parameters such. B. the frame rate. Thereby The video may become out of sync after re-multiplexing displayed when frames are lost.

Unfortunately The lack of frames is very common. Below are some possible ones Reasons for described a loss of frames. This problem is made more difficult in that in a video, the video data significantly more bandwidth as audio data, so the probability for that Loss of video frames is much higher than the loss from audio frames. If there is a loss of frames, this will Effect is even worse, that video and audio frames are usually of different lengths.

A Compress to a digital video in MPEG format in real time is required when watching videos from an analog source (analogue Video recorder, analog cable TV, analogue terrestrial Television) should be digitized. For the compression of videos In real time, a particularly high computing power is needed. Furthermore is the computing time required for Compressing an image sequence of the video is necessary, much of the image content dependent, So do it yourself with the most powerful processors available It can not be ruled out that individual images are not compressed can be before the next Image is received. In this case, frames, usually mostly Video frames are discarded.

If an analog television signal is received via terrestrial antenna or satellite dish, interference, such. B. slightly delayed reflections on the ground or on obstacles, interference signals, and weather, for which especially the satellite reception is sensitive affect the image quality. In two widespread, examined by the inventor TV decoder analog-digital wall For example, clocks and sync information (eg, starts of generating the next frame / frame at the blanking interval) are generated from the video input signal. This is also necessary because using a separate time base already with minimal deviation from the time base of the television signal creates a continuous picture. But it also leads to that, because z. B. the blanking interval is not detected clean, frames may be missing in the output data stream generated by the digitization, or frames are discarded because no clock could be formed for the sampling of the signal. This behavior can be understood from the data sheets of these circuits (eg Conextant Bt829 and Philips SAA 7113) and has been confirmed by the inventor's tests with the SAA7113. The data sheets were published on the websites of the respective manufacturers at the time of writing of this patent application.

Of the o. g. Effect is exacerbated by that the video decoders do not process the audio themselves, but only generate an analog baseband audio signal that either from the sound card of a PC or with hardware encoders and standalone devices a specialized audio decoder chip (sometimes with the CPU these devices) is digitized. This often leads that in the audio signal and the video signal a different Number of frames missing.

A another source for disorders in the analog input signal, which is lost as described above Frames leads, can Fluctuation fluctuations or mechanical damage to the magnetic tape at an analog video recorder. These usually lead to that the output signal fails temporarily. Even if on a video cassette, which is to be digitized, bodies are present at which If the recording starts or ends, it can be played over it Put out to disturbances come in the video signal of the recorder. The already described peculiarity video digitizer chips then in consequence of these disturbances often to a frame loss, because of the digitizing circuit with the missing video or audio signal also no clock for recording can be formed and no blanking interval is detected, which is the generation to initiate a new video frame would. This problem is exacerbated by that the video head and the audio head on analog video recorders offset by the dominant VHS system, so that the frame losses in the audio and in the video even at different Jobs occur and do not have the same duration.

Also when receiving digital sources, frames may be disrupted during transmission get lost, z. For example, if digital television by antenna from the satellite (DVB-S) and in some regions also terrestrial receive (DVB-T) becomes. The for However, the interference of the transmission path described in the analog television reception apply also for the digital transmission. As long as the disturbances are so small that they are not in the demodulation in the receiver Cause bit errors, the reception quality is not affected. In the border area become individual macroblocks disturbed of the picture, This leads to block formation. Exceed the disturbances a certain amount, the reception is complete disturbed, or individual frames are lost, causing a loss of audio frames less likely than the loss of video frames.

problem

Of the Loss of frames occurs quite often, as already pointed out, and can have many causes. Most often he occurs when Videos are digitized from analogue sources, and the video signal of these sources is of inadequate quality.

digital Video recorders (eg PC's with TV tuner card, DVD recorder), the videos in a format of the MPEG protocol family record these usually record them as system streams. Because in the system streams sync information is also included with the help of formed from the received video signal independent time base of the recording device, Lost frames will initially affect not from.

at the further processing z. B. by cutting the video or Burning the video to DVD will trigger this synchronization information usually lost. additionally these are the total playing time of those lost in the audio streams Frames almost always different to the total playing time of the in the video data streams lost frames when recording or transferring digital video frames have been lost. As a consequence, that in the finishing process sound and picture for the entire rest of the video are asynchronous.

Of the The invention is therefore based on the problem of finding a method with that during the Recording or later It can be detected when a system stream is not further processed can, because of one of the o. g. Reasons frames have been lost are.

Further, further embodiments of the method according to the invention is based on the task of losing lost frames in the audio and video data streams inserting system streams later or in real time, or to subsequently correct their adverse effects on further processing, in particular the shift of picture and sound, or to completely prevent the loss of frames.

Further should the inventive method all formats in which MPEG video family digital video may be available may be applicable; independently of which medium the digital video is stored on or via which Medium and with which protocol it is transmitted.

Description of the invention

Is solved this task according to the invention by hereinafter referred to as a frame error correction method Method having the characterizing features of claim 1. After that is a stored or received, as a system stream and in a format of the MPEG protocol family or a related format present digital video by comparison of each frame associated Synchronization information with the sum of all playing times Frames of the same data stream that occurred in this frame are missing Examined frames.

at the method according to the invention the digital video is preferably treated as a data stream and serially read in and evaluated. The claims are however, not limited to this serial processing. By a serial processing results in the advantage that the inventive method independently of whether the video is stored as a file, from a network received or transmitted by radio will always be used in the same way.

at the following description of the features of the method according to the invention is based on the description of the function, data structure and nomenclature of the MPEG protocol family in the section "Technical Field of the Invention" of this Patent application referred. This data streams or Files consisting of at least one video and optionally one or multiple audio streams associated with each video stream and encoded in a format of the MPEG protocol family, as called digital video.

Corresponding the description in the section "Technical Field of the invention " assuming digital videos in an MPEG format that the beginning of a frame with a video stream using of the Picture Header. When encoded at a constant bit rate Audio streams can change the start position of the frame based on the fixed length of the frame otherwise it can also be calculated by its header and the compression bit sequence dependent synchronization bit string contained therein can be identified. The frame or picture header is part its video and audio stream and thus in the packages of the packed with digital videos. As a result, the frames can receive the information Packet header and Packetized Elementary Stream Headers (PES header) of the packet in which the header of the frame starts, be clearly assigned.

A characteristic feature of the method according to the invention is to identify the individual packets contained in a digital video and process them in the order in which they are stored or transmitted. The stream ID from the PES header (see 6 ) of the packet is to check whether it corresponds to the stream ID of a video or audio stream to be checked for missing frames. If this is the case, for the further processing of the packet a register set or a memory area is to be selected in which the information required by the method for processing this data stream is buffered.

Distinctive Feature of the method according to the invention it is still that from the headers of the package the time stamp of the packet and in the register file or memory area is cached associated with the data stream of the packet is. The time stamp of the package is the following to which the playing of the frame whose data in the Package included, should be started. Is more in a package as a frame, the time stamp is below the timestamp to understand the beginning of playing the first in the package Frames should be started.

Of the Timestamp of the package is thereby according to the characteristics indent 5 and indent 6 of claim 1 from the synchronization information in the fields "PRESENTATION TIME STAMP "(in the" PACKETIZED ELEMENTARY STREAM "header)" SYSTEM CLOCK REFERENCE "(off the "PACKET HEADER ") calculated. Without leaving the inventive teaching, too an optional plausibility comparison between these two values, and the more plausible one Value accepted become.

Another characteristic feature of the method according to the invention is the examination of the contents of the packets of the data streams to be examined. If one were to assemble the contents of these packages while maintaining their order, one could add the data stream contained therein recover. This thus formed data stream is according to feature indent 2 of claim 1 after the headers of frames and video frames in addition to GOP headers ( 4 ) and to search for headers containing the frame rate (eg, sequence header).

Also required for the inventive method is according to feature indent 1 of claim 1, the playback time of the individual frames of the data stream. For the audio data streams, the playing time of each frame has a fixed value that is different for each audio format. In order to determine the playing time of the currently examined frame, it is sufficient to determine the audio format in which the associated data stream exists and to store the frame rate assigned to it in a register. In order to determine the audio format, the stream ID can be used, if necessary also information on subformats (eg in the case of the MPEG-original audio formats) must be evaluated from the header of the audio frame. With video data streams, however, several different frame rates are possible, which are coded in different headers. Without limiting the method according to the invention, the image refresh rate of the video data stream z. For example, from the sequence header that is in 5 is shown determined.

at this search must also be taken into account a frame, picture, sequence or GOP header can be distributed over 2 packets. Therefore, as a minimum requirement to those for the inventive method device used to request that for each video or audio stream to be checked at least one shift register of the length of the largest one to be evaluated for the method Header is present in the respective data stream. In this register x, the data read sequentially from the packet is pushed into it. The selection of the register set of the data stream is controlled through the stream ID from the PES header of the packet in which the Frame starts. After each sliding operation is to check if the beginning of this register contains the identifier of a frame header. at Video streams is about it to check out whether the beginning of this register is the identifier of a GOP header or a sequence header contains. Optionally, a buffer may be provided in which at least 2 packets of each data stream to be examined cached can be and in this buffer after the start of the frame and, if necessary, after the start a GOP header or a sequence header. In audio frames, which are coded with constant bit rate, it is possible (with PCM necessary), to calculate the frame start from the frame size, being the size of a Frames in turn results from the bitrate.

• • Entsprechend Merkmal Spiegelstrich 2 des Patentanspruchs 1 ist der Zähler für die Anzahl der Frames des Datenstrom zu erhöhen.
• • Entsprechend Merkmal Spiegelstrich 3 des Patentanspruchs 1 ist bei Videodatenströmen zusätzlich der Zähler für die Anzahl der Frames des GOP ist zu erhöhen.
• • Bei Videodatenströmen muss das Feld „TEMPORAL SEQUENCE NUMBER” aus dem Picture Header gelesen und im dem gerade untersuchten Datenstrom zugeordneten Registersatz oder Kontext zwischengespeichert werden. Dieses Feld wird benötigt, um die wegen der innerhalb eines GOP möglicherweise von der Übertragungs- oder Aufzeichnungsreihenfolge abweichenden Abspielreihenfolge erforderliche Korrektur der Zeitberechnung aus der Framezahl vorzunehmen.
• • In den meisten Fällen beginnt das erste Frame eines jeden Datenstroms des digitalen Videos mit einem Offset. Dieser Offset dient dazu, Abspielgeräten oder -programmen die Zeit zu geben, sich zu initialisieren. Der Offset wird dadurch realisiert, dass dem ersten Frame jedes Datenstroms mittels der Synchronisationsinformationen in dem Packet- und PES-Header ein Zeitstempel zugeordnet wird, der größer als 0 ist. Ein sinnvoller Wert für diesen Offset sind 300 Millisekunden. Da es Multiplexer gibt, bei denen dieser Wert eingestellt werden kann, hängt es letztendlich von der Person, die das digitale Video erstellt hat, ab, welcher Wert vorgegeben ist.

When locating the start of a frame, the following operations characteristic of the method according to the invention are to be carried out:

• According to feature indent 2 of claim 1, the counter for the number of frames of the data stream is to be increased.
• • According to feature indent 3 of claim 1 is in video data streams in addition, the counter for the number of frames of the GOP is to increase.
• • For video streams, the "TEMPORAL SEQUENCE NUMBER" field must be read from the Picture Header and cached in the register set or context associated with the data stream being examined. This field is needed to correct for the time calculation from frame number required because of the play order that may be different within a GOP from the transfer or record order.
• • In most cases, the first frame of each stream of digital video begins with an offset. This offset is used to give players or programs the time to initialize. The offset is realized by assigning to the first frame of each data stream, by means of the synchronization information in the packet and PES headers, a time stamp which is greater than zero. A reasonable value for this offset is 300 milliseconds. Since there are multiplexers where this value can be set, it ultimately depends on the person who made the digital video, which value is given.

According to feature Indent 5 of the method according to the invention becomes the offset the data streams to be examined written in the register set or context associated with the data stream. The offset is determined when the beginning of the first frame a video or Audio data stream was detected. At this time corresponds the offset of the data stream is the timestamp of the packet in which the first frame starts.

• • Die Prüfung, ob vor dem Frame, dessen Header gerade ausgewertet wird, Frames verloren gegangen sind, ist entsprechend dem erfindungsgemäßen Verfahren beim Auftreten jedes neuen Frame-Headers auszuführen, wenn es sich um den ersten Frame-Header in dem Paket handelt. Dieses Verfahren wird im Anschluss an die Beschreibung der auf einen GOP-Header und einen Sequence-Header anzuwendenden Operationen beschrieben. Abhängig vom Ergebnis dieser Prüfung können weitere, in den Unteransprüchen beschriebene Operationen ausgeführt werden.

The offset can be different for each data stream. Therefore, a memory cell is required for each video and audio data stream to be examined, in which the offset of the respective data stream is noted.

• The check whether frames have been lost before the frame whose header is being evaluated has to be executed according to the inventive method upon the occurrence of each new frame header, if it is the first frame header in the packet. This method will be described following the description of the operations to be applied to a GOP header and a sequence header. Depending on the result of this test Further operations described in the subclaims can be performed.

In the video data streams the MPEG protocol family groups frames into groups which are referred to as GOP (Group of Pictures). Within a GOP is allowed to save or transfer the frames in a different order, as the order in which they should be displayed. This special feature of video streams must for the inventive method considered become. As explained in more detail later in this description compares the inventive method the accumulated runtime of the data stream that has occurred so far Frames formed runtime with the timestamp of the currently checked frame. For video streams This accumulated runtime must be one from the playhead of the frame within the GOP calculated value to be corrected before the comparison can be made with the timestamp of the frame.

Around the correction value resulting from the play order of video frames later to determine the transmission or storage position of the frame within the GOP as well as its play position be known. Without departing from the inventive teaching, various Procedures used for this purpose. The easiest way is it, at the beginning of each GOP a counter To start with every picture header, this is the frame header in video streams, increased by 1. This counter must for any video stream being examined for missing frames, be present once.

The beginning of a GOP is indicated in the video streams by the GOP header. The GOP header is in 4 shown. It begins with a 24-bit synchronization bit string (0x000001), followed by the identification byte of the GOP header (0xB8). The GOP header is just like the picture header part of the video data stream, so it is not created until the multiplexing. In the search for the GOP header, apart from the value of the identification byte, the same procedure as for the search for a picture header (this is used as a frame header for video data streams).

• • Der Zähler für die Anzahl der Frames des GOP ist zurückzusetzen.

When locating the start of a GOP in a video stream, perform the following operations:

• • The counter for the number of frames of the GOP must be reset.

The sequence header is in 5 shown. It begins with a 24-bit synchronization bit string (0x000001), followed by the identification byte of the sequence header (0xB3). The sequence header is just like the picture header and the GOP header part of the video data stream, so it is not created until the multiplexing. The search for the sequence header, apart from the value of the identification byte, can be used in the same way as for the search for a picture header or a GOP header.

• • Aus den Codes für die Bildrate des Videodatenstromes im Feld Framerate des Sequence Headers ist gemäß der Tabelle in 5 die Anzahl der pro Sekunde abzuspielenden Bilder zu ermitteln, aus der wiederum die Abspieldauer eines Bildes ermittelt werden kann.

When locating the start of a sequence header in a video stream, perform the following operations:

• • From the frame rate codes of the video stream in the Frame Rate field of the Sequence Header, refer to the table in 5 determine the number of images to be played per second, from which in turn the playing time of an image can be determined.

T_F:

Abspielstartzeitpunkt (Zeitstempel) des Frames (aus dem Feld SCR des Packet-Headers oder aus dem Feld PTS des PES-Header des Paketes berechnet, in den der Beginn des Frames fällt),

Fps:

Framerate bei Audiodatenströmen (abhängig von der Codierung des Datenstroms) oder Bildrate (aus dem Sequence-Header)

T_Offs:

Der Offset (Abspielstartzeitpunkt/Zeitstempel) T_F des ersten Frames des Datenstroms

N_F:

Die Anzahl der bisher aufgetretenen Frame-Header einschließlich des Headers des gerade ausgewerteten Frames

With the information obtained from the various headers during sequential reading of the data streams as described above, the inventive method for checking for missing packets can be executed at the end of each first frame header in a packet. At this point, the following information is available for the frame being examined:

T _F :

Playback start time (timestamp) of the frame (calculated from the field SCR of the packet header or from the field PTS of the PES header of the packet in which the beginning of the frame falls),

fps:

Frame rate for audio data streams (depending on the encoding of the data stream) or frame rate (from the sequence header)

T _Offs :

The offset (playback start time / timestamp) T _{F of} the first frame of the data stream

N _F :

The number of frame headers that have occurred so far, including the header of the currently evaluated frame

TSN:

Abspielposition des aktuell bearbeiteten Frames im GOP (aus dem Picture Header des gerade ausgewerteten Frames entnommener Parameter Temporal Sequence Number)

N_GOP:

Die Anzahl der bisher in dem aktuellen GOP gezählten Frames Mit diesen Informationen kann nun die Überprüfung, die Hauptgegenstand des erfindungsgemäßen Verfahrens ist, erfolgen. Dazu wird die Zeit berechnet, die benötigt worden wäre, um alle bisher aufgetretenen Frames des Datenstromes hintereinander abzuspielen. Diese Zeit wird im folgenden als

T_R:

Rohzeit

bezeichnet. Für Audiodatenströme kann sie wie folgt berechnet werden: TR = (NF/Fps) + TOffs For video frames, the following information is also available:

TSN:

Play position of the currently edited frame in the GOP (parameter Temporal Sequence Number taken from the Picture Header of the currently evaluated frame)

N _GOP :

The number of frames previously counted in the current GOP With this information, the review, the main object of the method according to the invention is now carried out. For this purpose, the time is calculated, which would have been needed to play all previously occurred frames of the data stream in a row. This time will be referred to below

T _R :

Rohzeit

designated. For audio data streams, it can be calculated as follows: T R = (N F / Fps) + T offs

For video frames, the position of the frame in the GOP must be considered. Here the raw time is calculated as follows: T R = ((N F - N GOP + TSN) / Fps) + T offs

Now it can be checked if frames have been lost in front of the position of the frame being checked. This is the case if the raw time T _{R is} greater than the play start time T _F calculated for that frame from the packet header or PES header of the packet in which the beginning of the frame falls.

The above description of the method according to the invention is in the form of a flow chart in 7 for a video stream. In 1 a flow chart for the application of the method according to the invention to an audio data stream is shown.

Advantages of the invention

The Advantages of the method according to the invention are that it's the possibilities to detect if a video is incorrectly digitized or transmitted digitally has been extended. Missing frames in digital video are common Error as described in the section "Technical field of the invention", can have a whole range of causes. The personal experience of the applicant is that, in the digitization of VHS video cassettes, when they reach a certain age, lost frames can not be avoided no matter how high the quality for the Digitization used devices are and how carefully you proceed. In addition, there is a lack of conventional procedure Only make frames noticeable when watching a digital video further processed.

Especially due to the different lengths of video and audio frames to lead Conditions during which the digitization frames are lost after further processing almost always a loss of synchronization of image and sound. A lack of synchronization of image and sound affects but very disturbing out.

While with analog video a disturbance only for short time, the same video after digitizing and further processing be useless, since the lost synchronization between Audio and video starts at the point where frames are missing, and continues until its end. It is particularly annoying when one analog video has been digitized, then deletes it, and only after the subsequent processing notes that digitization has failed. This is possible avoid with the method according to the invention.

In the dependent claims are further advantageous embodiments of the method according to the invention specified, with the help of which missing frames at least so far correct that they do not have any loss during further processing the synchronization of image and sound effect more. This has the The advantage that analog videos can still be "saved" in this way, if error-free digitization is no longer possible.

Advantageous embodiments the method according to the invention

A advantageous embodiment of the invention is in claim 2 specified. The development of the invention according to this claim allows to determine how many frames are in front of the one being examined Frame are lost. This is necessary because of disturbances, which prevent the correct digitization or transmission, quite also lead to it can, that several frames are lost.

According to claim 2, the number Nv of the missing frames for audio data streams with the formula Nv = (T F - (N F / Fps) + T offs ) · FPS and be determined by rounding to an integer value. For video frames, the position of the frame in the GOP must be considered. Here can Nv with the formula Nv = (T F - ((N F - N GOP + TSN) / Fps) + T offs ) · FPS and be determined by rounding to an integer value.

The specified in claim 3 advantageous embodiment of the invention is required to detect even after the detection of the first frame, before a single frame or a block of frames, even more lost frames. If one did not correct the registers of the method after detecting such an error, a loss of frames would be detected every other frame. This correction can be made according to the invention by adding the number of calculated according to the method according to claim 2 number of lost frames Nv to the counter of the previously occurred frames N _F. This realization requires u. U. one Rounding error, however, is very easy to implement, especially when integrating the method according to the invention into a hardware solution. An alternative advantageous embodiment of the invention according to indent 2 of claim 3 is to increase the register or the variable for the offset of the data stream in which the loss of frames was discovered in order to increase the playing time of the lost packets. This also achieves that a continuation of the comparison is possible.

object a further advantageous embodiment of the invention according to claim 4 is the loss of synchronization between picture and sound the further processing of digital videos in whose data streams frames are lost, prevent. These are according to feature Indent 1 of this claim the synchronization information, which exists only in the Packet and PES headers of the bound video are, and can be lost in further processing, in the video and audio data streams conform to format z. B. as user data. inserted. After further processing of the digital video is then one more Post-processing required, which in the course of further processing possibly incorrectly set synchronization information with help corrects the information inserted in the video and audio streams.

at Video streams can you z. B. exploit that the MPEG formats the insertion of user data allow. This user data is z. B. to insert the Information with which encoder the video was compressed, used. User data can in video streams anywhere outside of an area enclosed by an MPEG header. They start with a user data header. the user data header exists from the 24-bit synchronization bit sequence (0x000001) and the 8-bit identifier of the user data header (0xB2). Regarding the Content of the user data, there is only the limitation that this is not the 24-bit synchronization bit sequence (0x000001) as it signals the beginning of the next MPEG header, with which the user data block ends. To the synchronization information from the package headers to secure the system stream, which lost during further processing would go So you can create a user data block. It's enough from every package that contains a picture header or a frame header starts the contents of the PRESENTATION TIME STAMP fields in the PES header or / and SYSTEM CLOCK REFERENCE in the packet header of the package, as Insert user data. additionally have to these two values for the previous frame in the stream will be saved to recognize later to be able to if the video was cut. In addition, an identifier is required at this user data block as one of the inventive method generated record can be detected. For this identifier can z. Legs String can be used, an extra checksum over the String and the saved data can be the probability increase detection.

The Feature indent 2 of claim 4 is required for Restore synchronization after further processing of the digital video. In this further processing (eg authoring as a DVD or editing the video), the data streams are usually demultiplexed and subsequently again multiplexed. It usually go u. a. the first time Generating the bound video generated timestamp lost. were these as described in feature indent 1 of claim 4 stored in the data stream as recovery information, can with The fields PRESENTATION TIME STAMP in the PES header and / or SYSTEM CLOCK REFERENCE in the packet header of all packets of the data stream, from which they were taken. Is in a block of recovery information to a video or audio stream are found in the package headers of the package in which this block ends the values contained in this block for the fields PRESENTATION TIME STAMP or / and SYSTEM CLOCK REFERENCE written. All following Packet headers of the stream with the same stream ID will be in corrected the same way until the next block with recovery information Is found. If the digital video was cut, this must be Value, however, be corrected. This can be recognized by it, though the saved synchronization information of the last frame in currently processed frame differ from the synchronization information, the for the previous frame was saved, deviates. Then the synchronization information, which is written to the further processed system stream the difference between these two values will be corrected.

The removal of the added in the audio and video data streams according to feature indent 1 of claim 4 additional correction information after binding or multiplexing the video is very complex. This is especially true if the video has been multiplexed to DVD for burning. In this case, the navigation parameters for the menus of the DVD would have to be adjusted as well as the playheads and hands that were generated for the search or play backwards and forwards at a multiple of the normal speed. However, since the inserted data is packaged in the MPEG format compliant headers that are ignored by the player or program, it is not necessarily necessary to remove.

meaningful However, it is only this correction information in the data stream to write when the synchronization information in the PES or / and packet headers change. This often leads that significantly less overhead due to the correction information arises, especially hardware encoders in the packet or PES header of all Frames that belong to a GOP enter the same time stamp. This is not just the video frames affected. Also, all audio frames associated with this GOP are getting the timestamp from the video frames of the GOP.

Of the Overhead due to the correction data can be further reduced, if the paste the additional Correction information according to feature indent 1 of claim 4 for reducing the amount of data only occurs when a frame has occurred before which frames are lost went. Whether such a frame is present, with the in claim 1 specified method can be determined.

Become Correction information only before frames from which frames are lost gone, inserted, can also be used instead of the absolute information (the timestamp of the frame) the difference between the actual and the expected timestamp in the block inserted for later correction be registered. During post-processing after further processing of the digital video must then be the timestamp of the frames of the data stream to increase the correction value written in the data stream. For this purpose, in the package header of the package in which the block with the correction information ends that contained in this block Values for the fields PRESENTATION TIME STAMP or / and SYSTEM CLOCK REFERENCE written. All following packet headers of this data stream will be corrected in the same way until the next block with correction information Is found. In this case, however, it must be ensured that the inserted Correction data, the further processing should be a cutting of the Videos include, not deleted become.

A Another advantageous embodiment of the invention is in the claim 5 indicated. This also serves the loss of synchronization between Picture and sound in the processing of digital video, in their data streams Frames are lost, prevent. She should, however can be used if post-processing is not possible after further processing. This can be z. B. be the case when further processing with a Authoring program for a DVD takes place, which is the caching of the burn DVD certain files on disk not allowed, but the Data burns directly to DVD. In contrast to the in claim 4 specified method, it is also possible here, in the context of further processing to perform a new encoding (which, however, must be discouraged because of the associated loss of quality).

According to claim 5 are pre-encoded to achieve this goal or at runtime the parameters of the data stream encoded frames in the data stream inserted. For audio data streams this is not a problem as it is in the MPEG audio compression formats no dependence between the individual packages. Holding precoded Frames in non-volatile memory the for execution the method according to the invention used device offers itself, since there are very few parameters where audio and video streams can vary.

at Audio streams the number of frames to be pre-encoded only affects the sampling rate (allowed Values 32.441, 48 kHz and 44.1 kHz) and the data rate (16 permissible values). Considered only those found in the field of consumer electronics Variants of the MPEG audio compression method MPEG 1 Layer 1, MPEG 1 Layer 2, and MPEG 1 Layer 3, results in a manageable number of frames, which you have to precompress. something similar applies to the audio compression methods used by the MPEG formats derived industry standards. These include Dolby Digital, PCM and Digital Theater Sound. So z. B. at Dolby Digital sampling rates 32, 48 kHz and 44.1 kHz allowed, so like 38 different values for the data rate. Because of these properties, with the insertion of precompiled frames connected much less effort than to recompress all or part of the data stream as you it can save itself in implementing an encoder. The Processing time is also reduced in this way, this In turn, reduces the performance requirements the for the inventive method used device.

If pre-encoded frames are inserted, they should not be noticeable to the observer / listener. This can z. For example, a silent frame (ie, a sound sequence in which no sound is heard) or a repetition of the last image so that the image freezes are used. Since ana logen source material disturbances that cause the loss of frames in digitizing, are often seen as a picture dropouts or continuous stripes, even a subjective improvement in the quality compared to the analog output material can be achieved often. Without departing from the inventive teaching, but can also frames are involved, such. For example, a fault icon, an error message, or a beep to report the problem.

at Video streams have to also only a few frames are precompressed, at least if you look at the resolutions limited, the DVD player support. These are 720 × 576, 704 × 576, 352 × 576 and 352 × 288 at PAL (in Europe and others) such as 720 × 480, 704 × 480, 352 × 480 and 352 × 240 at NTSC (in the US and others). Other parameters of the video data stream, such as B. the aspect ratio or the frame rate, have no effect on the frames to be inserted. The resolution the video data stream can, for. B. from the system header to be corrected Data stream are read.

On the first look problematic, however, appears that the frames of video streams from each other depend, because in each GOP only one picture, the I-frame, is present, which is the contains complete image information. All other pictures contain only the information in which they differ from the I-frame or other images of the GOP. To replace a missing image, it makes sense to look at it restrict, to freeze the picture. In this case you can insert a picture that opposite does not change the previous picture. If you compare 2 frames completely different, but generated under the same boundary conditions Videos with the same parameters and the same encoder compressed, and picks out a section in each video, In which nothing moves, one realizes that of others Frames depending Frames (B-frames or P-frames) are the same for both videos. So you can remove frames from a video while still in motion and thus a frame in a still sequence of another video replace without anything changing, although both videos completely show different still pictures. One can with the taken out Frame even replace a frame in a moving sequence. In this Case, however, freezes the picture. Is the removed frame elsewhere in the GOP, as the inserted, you may need the reference on the frame or frames, to which the inserted frame contains the delta information, to adjust. The parameter Temporal Sequence Number in the headers of the frames of the GOP to be corrected.

Around So you can replace missing frames additional B or P frames, preferably B frames, inserted into a video stream be without removing other frames. However, this is increasing the number of frames in the GOP and its composition over others GOPs of the video. Especially if the source material for a DVD is determined, in which a GOP 15 or 18 frames (for PAL or NTSC), you have to hurt u. U. the specifications. In practice, however, it has been proven that the tested hardware and software DVD player such overlong GOP's without problems could play. Therefore, that such overlong GOP's within certain limits are uncritical, also speaks that on the market rental and purchase DVD's are available, which do not comply with the DVD-Whitebook prescribed GOP lengths. Farther has a commercial DVD authoring program that has been tested by the Applicant and that has the on DVD to burning video files on DVD conformity checks, and only re-encoded, if the video files in this test fails, video files with inserted B-frames without re-encoding on DVD-burned

Yet is the method in claim 5 only in its application Audio frames complete standard compliant. Aim of the method described in claim 6 It is therefore, even missing frames in video streams standard compliant to correct. In claim 6 is therefore proposed that to restore the synchronization between picture and sound possibly missing audio frames in the audio data streams as in claim 5 described inserted and in those places where video frames are missing, which are among the missing video frames synchronous audio frames are removed again. Since audio and video frames can be different lengths, must if necessary, the number of video frames to be removed per video frame to be removed Audio frames are varied to achieve that playing time the distant audio frames and the lost video frames about the same are long.

Corresponding Claim 7 may be a mixing of the methods of claim 4, 5, and 6 be advantageous. So it may be z. B. be useful, the procedure according to claim 4 on video data streams and the method according to claim 5 on audio data streams apply.

Further advantageous embodiments of the invention are mentioned in the claims 8 and 9. According to claim 8, it makes sense to use the device used for the inventive method simultaneously for demultiplexing the digital video. This is advantageous because the method according to the invention requires the search for the frame and packet headers as well as the streamwise analysis in the evaluation, and thus only has to be supplemented slightly to fulfill the function of a demultiplexer. Furthermore, it is much easier to insert missing frames into a video or audio stream than they would into a linked video, taking into account the changes in the packets involve them.

at many occurring in the context of digital video processing Tasks such. B. cutting, burning to DVD or encoding the system stream, as the digital video is usually present, be demultiplexed. Therefore, it is according to claim 8 also useful, the inventive method to integrate into such an application.

Also makes sense according to claim 9 the integration of the method according to the invention into an integrated Video digitizing circuit, since you are safe at this point that may be the video before checking for missing frames yet may have been further processed. The reason for this is that the inventive method only works as long as the digital video is not yet demultiplexed and the encoder used to generate the system stream has set the synchronization information correctly.

A advantageous embodiment of the invention is in claim 10th specified. This embodiment makes use of that when digitizing analog Videos where the analog video recorder, such B. a VHS VCR that has interrupted recording with high Probability lost frames in the digital output data stream walk. The investigations carried out in the context of the preparation of this patent application VHS video recorders of various types and makes have been found in places where the recording was interrupted, some few frames are lost. If an older recording was dubbed, arises in the digitization of the place where the new recording ends, a sequence with a significant number of in very short intervals occurring missing frames. This agrees with theoretical considerations to the restrictions the architecture of analog VCRs that record the video signal to record a magnetic tape. Cause for This observed and predicted behavior is that of an analogous VCR when inserting the cassette tape from the cassette pulls out and by means of several pulleys around the video head drum wraps. This process is not very repetitive, so that when starting a recording in the middle of the band this Record not exactly at the end of an image already on the tape is present in the tape. The preparation of the This can usually be output signals, so also in the examined Recorders, do not compensate. This is true even with high quality devices with digital signal processing. At the signal generated by the VCR this causes disruptions the blanking interval, so that the electronics used for digitization usually does not recognize the beginning of a new image and thus omits frames. This Effect is only very short-term at the beginning of a recording, and lasts in usually exactly one frame long. At the end of a recording or when dubbing an old record this effect is reinforced by the fact that the Record the pictures at an angle to the band takes place while the erase head horizontal over the video recorder the band led will and over out locally offset to the video head is attached. This will end up with a Recording that dubbed an older recording, a whole series of pictures disturbed. This has the consequence that Digitizing in this area a whole series of frames get lost.

With the method according to the invention can according to the o. g. versions Even after digitization, bodies are detected at which in the analog recording new films or video footage begin was cut. These brands determined in this way can according to dependent claim 10 for the setting of chapter marks for the creation of DVD menus with an authoring program used for burning video DVD's or as crop marks for the cutting of the digitized material will be proposed.

example

following The invention will be explained with reference to two applications of the closer. Show it:

8th : Representation of the use of the method according to the invention for the correction of errors that have arisen during the digitization of a partially dubbed VHS video

9 , The protocol of the error correction of in 8th illustrated use case

10 , Representation of the use of the inventive method for correcting errors that occur when the recording is started in a digital video recorder before starting to play the video to be digitized.

11 The protocol of the error correction of in 10 In the case of consumer electronics, the VHS system for recording video sources such as video signals has been used by end users. B. enforced television broadcasts. At times, this system was also used by amateur filmmakers. VHS has been on the market for about 30 years at the time of filing this patent application.

VHS is an analog Vi de-recording procedure, which is now successively replaced by digital video recording. First vendors have already removed VHS VCRs from their product range. This results for end users who have pre-recorded VHS tapes, the problem that for the recorded video in the foreseeable future, no players will be available more. Especially when personal memories are recorded on these cartridges, there is a desire to transfer their contents to a digital medium. In addition, the quality of analog video recorded on magnetic tapes used with VHS video cassettes decreases with time. Since this is not the case with digitally recorded videos, this transmission creates an additional benefit for the consumer. This will increase the demand for solutions that help consumers and service providers to digitize analog video.

As already described elsewhere in this patent application, can from the properties of the prior art and for playing analog video and recording digital video Devices are closed, that digitizing videos a process is where short-term disturbances of the source material lead to it can, that after processing the digital video sound and picture is not are more in sync. Already a slight time offset of image and However, sound is perceived as very disturbing.

In the 8th and 10 Applications shown represent two situations in which the digitization of analog video using the prior art associated with a u. U. qualitatively no longer satisfactory results, and in which the inventive method can be used in conjunction with its subclaims to improve the result. Both applications describe situations in which the output signal of an analog video player is disturbed so much that a digital recording device can no longer decode it cleanly and therefore rejects frames in the output data stream it generates. Such disorders can have many causes, ranging from poor quality of the media to be dubbed due to aging. The two selected use cases have the advantage that they are reproducible.

In the first, in 8th In this case, it demonstrates how the transitional area affects the digitization of a video at the end of a new recording that has been dubbed an older, slightly longer recording. If a VHS VCR continues to play at the end of a recording, a disturbed video signal will be output for about one second before the older recording is output smoothly. The reason for this has already been addressed in the description of subclaim 10.

8th shows how the sequence used to evaluate this use case was generated. In step 1) a longer sequence is recorded on a VHS video cassette. Then the cassette is spooled back and then in step 2) the beginning of this sequence is dubbed with a shorter sequence. Subsequently, the video cassette is rewound again in step 3), played back and recorded digitally in MPEG-2 format.

In the evaluation of the recorded video with the inventive method according to claim 1, to 3 and its correction with the method according to claim 4, the in 9 generated log generated. At the same time, the digitally recorded video was visually evaluated image by image using a video editing program. In this case, the later recorded recording ends after 1 minute, 1 second and 14 frames a 40 ms (equivalent to a runtime of 61,560 ms). The image of the older recording is stable at 1 minute, 12 seconds and 18 frames (equivalent to a runtime of 72,720 ms). At this point the transition area ends. According to protocol in 9 was discovered by the inventive method, the first missing video frame after 62,083 ms runtime and the last missing video frame after 72,766 ms. The runtimes issued in the protocol are those entered in the PTS by the digital recorder (in this case a TV tuner card for PC's with hardware encoders). Of these runtimes, 400 ms offset and the lost frame time (in this case 40 ms per frame as recorded in PAL) have to be deducted. At the end of the recording is still missing another video frame, but that does not matter here.

Overall, 19 video frames and 16 audio frames were not recorded as a result of the transients in the transition area. Especially since the playback time of the audio frames is 24 ms (they were encoded as MPEG I Layer 2 audio data) and the video frames 40 ms (the video was available as a PAL signal with 25 frames / s), there is a temporal difference between audio and video Offset of 360 ms from this transition range. However, this temporal offset only becomes visible or audible after burning to DVD. To optically control this, in step 5) both the recorded and the corrected file produced by the method according to claim 4 were burnt to DVD and subsequently considered tet. It was found that even this short offset of 360 ms in the uncorrected video is already noticeable.

The Method according to claim 4 was therefore selected for this application because it gives a result that does not quite conform to the relatively narrow ones Specifications of the DVD Whitebook is, and therefore clarified in experiments had to be whether the DVDs produced with it on different types of DVD players runnable are.

In the second, in 10 In the case of the application shown, it is demonstrated which effects it may have if the recording with the digital recording device is started in step 1) before the playback of the playback device used as the signal source is started in step 2). In this case, a VHS VCR was used as the signal source whose antenna input had not been connected. Therefore, as long as the VCR has not played anything, it has output a noise signal at its output.

The evaluation of the digitally recorded video for the second application is as in 10 Step 3) shown evaluated by the method according to the invention according to claim 1 to 3. In the correction of the digital video with the method according to claim 4, the in 11a to 11c generated log generated. At the same time, the digitally recorded video was visually evaluated image by image using a video editing program. In this case, the output of the video played sets at 28 seconds and 9 frames a 40 ms (equivalent to a runtime of 28,360 ms), the picture is interference-free after another 9 frames (equivalent to a runtime of 28,720 ms). According to protocol in 11a to c, the first missing video frame after 521 ms runtime and the last missing video frame after 29.392 ms were discovered by the method according to the invention. The runtimes issued in the protocol are those entered in the PTS by the digital recorder (in this case a TV tuner card for PC's with hardware encoders). Of these runtimes, 400 ms offset and the lost frame time (in this case 40 ms per frame as recorded in PAL) have to be deducted.

Looking closer, you can see that the TV tuner card loses frames because of the lack of input signal during the time nothing is playing. In the area in which the image stabilizes at the start of playback, an accumulation of lost frames is observed. In total, 54 video frames and 40 audio frames are lost in the switch-on phase. Since in addition to the higher number of video frames lost, the playing time of the audio frames is 24ms (they are encoded as MPEG I Layer 2 audio data) and the video frames are 40ms (the video was a PAL signal at 25fps) Audio and video a time offset of 1,200 s from this transitional area. However, this temporal offset only becomes visible or audible after burning to DVD. To visually control this, were in 10 Step 4) burned both the digitally recorded as well as the corrected file produced by the method according to claim 4 on DVD and then viewed. It was found that the offset of 1200 ms in the uncorrected video is very disturbing.

The DVD burned in step 4 was a total of 4 commercial and an open-source DVD authoring program. At all 5 on this Way created DVD's video and audio were asynchronous with uncorrected files. 3 of the commercial, tested authoring programs had the property the material basically to re-encode. Also with these programs were video and audio for the uncorrected files asynchronous. Apparently encode You for every frame of the input file is a frame for the output file, without itself to take care of the synchronization information. In one of the tested Programs that the video basically re-encoded, had the non-existent synchronization between video and audio on one of the uncorrected files even led to that a directly behind recorded, flawless file also had lost the synchronization between image and sound.

Economic and technical potential

The quality from analog recorded video material, especially if the Records are already several years old, or if on a tape multiple records are present, not sufficient for digitization and subsequent digital finishing with state-of-the-art techniques be.

As shown in the examples, the DVD-making products available on the market and known to the applicant do not sufficiently take into account the particular problem of lost frames in the digital recording of low-quality analogue sources. By contrast, the method according to the invention offers the possibility of "saving" such recordings. As VHS video recorders in the consumer sector have dominated the video recording and reproducing market for more than 30 years and are now being replaced by digital techniques, a large number of videocassette tapes are expected for their owners or owners Digitization is intended. A high proportion it is meanwhile aged so much that a treatment of their quality z. B. may be required with the inventive method.

Claims (10)

A method of detecting missing frames in audio and video streams bound (multiplexed) for transmission or storage to a system stream (system stream, eg, a transport stream, program stream, DVD-Video object) and in a format of the MPEG protocol family (Motion Pictures Expert Group, eg MPEG 1, MPEG 2, MPEG 4 etc.) or related formats (eg DivX, DVD-Whitebook, DVB-T, DVB-S , DVB-C), as well as for repairing such corrupted by missing frames and bound from video and audio data streams, characterized in that - the playing time of the frames of the data streams to be examined from the control parameters of the respective data stream are calculated the system stream is searched for packets belonging to one of the data streams to be examined and containing the beginning of one or more frames, and in each audio and video stream to be examined the number d he frame Frames) of the data stream starting from the start, as in each video data stream the number of frames of the data stream is counted beginning with the start of the GOP, - in audio frames based on the previously counted frames and the duration of the frame determined from the audio data format , for video frames based on the previously counted frames, the display position of the image from the "TEMPORAL SEQUENCE NUMBER" field from the "Picture Header" and the frame rate, the time referred to below as the raw time to which the frame is added as the previous frames are continuously output output is added to the raw time of the frame, this offset being determined from the synchronization information from the packet headers of the first packet of the system stream in which the first frame of this data stream begins, and this synchronization information being from the field "PRESENTATION TIME STAMP" in the "PACKETIZED ELEMENTA RY STREAM "header or / and from the" SYSTEM CLOCK REFERENCE "field are taken from the" PACKET HEADER ", - for each found frame the raw time of the frame plus offset is compared with the synchronization information of the frame, this synchronization information from the" PRESENTATION TIME STAMP "and / or the" SYSTEM CLOCK REFERENCE ", which in turn are taken from the first packet of the system stream in which data of the frame currently being examined is contained, and - is assumed to be one or more frames missing before the examined frame if raw time plus offset is less than the "PRESENTATION TIME STAMP" or / and the "SYSTEM CLOCK REFERENCE" of the first packet of the currently considered frame plus a security that is less than or equal to the time for the output of a frame. Detection of Missing Frames (Frames) in audio and video streams according to claim 1, characterized in that upon detection one or more frames lost before the frame under consideration the number of lost frames is determined, in which the difference of timestamps determined from the synchronization information and the sum of raw time and offset by those determined for the data stream Playback time of a frame divided and rounded to an integer (usually rounded). Detection of Missing Frames (Frames) in audio and video streams according to claim 1 and claim 2, characterized, that upon detection of one or more missing frames after the optional one execution one of the corrections described in the further claims - the offset the data stream is increased by the play time of the lost frames, or - of the for the Calculation of the raw time used offset data stream by the playback time the lost frames increased becomes. Method for detecting missing frames in audio and video data streams according to claims 1 to 3 and for correcting the adverse effects of these missing frames, characterized in that - before the further processing of the digital video, the synchronization information obtained from the fields "PRESENTATION TIME STAMP "in the" PACKETIZED ELEMENTARY STREAM "header or / and from the" SYSTEM CLOCK REFERENCE "field from the" PACKET HEADER "of the packet of the system stream in which a frame starts, in the second and all subsequent frames of the data stream also the synchronization information from the "PRESENTATION TIME STAMP" fields in the "PACKETIZED ELEMENTARY STREAM" header or / and from the "SYSTEM CLOCK REFERENCE" field from the "PACKET HEADER" of the system stream packet in which the previous frame starts, format conforming to the MPEG protocol of the digital Videos (eg as user data together with an identifier) this user data block can be recognized as a data record generated by the method according to the invention) may be inserted before the frame in the data stream, this insertion either before each individual frame, only before the frames before which the content of the synchronization information changes, or optionally together with the playtime corresponding to the lost frames only before the frames from which lost frames were detected, and - in a second step, which is only possible after further processing of the digital video, the synchronization information in the fields "PRESENTATION TIME STAMP" in the "PACKETIZED ELEMENTARY STREAM "header or / and" SYSTEM CLOCK REFERENCE "from the" PACKET HEADER "of each packet of the system stream generated during further processing using the information inserted prior to further processing. Detection of Missing Frames (Frames) in audio and video streams according to claim 1 to 3 and to correct the adverse effects this missing frame, characterized, that - instead of each frame lost a frame, preferably when viewing or playing is not noticeable (for example, a silent audio frame or a B frame of a video stream in which a still picture is encoded is) inserted in the data stream be, with the inserted Frames during the processing of the data stream to be checked using parameters extracted from this data stream be generated, or - in the for the inventive method created device at least one frame, preferably When viewing is not noticeable, for each of those from this device supported formats the audio and video streams permissible combination of parameters for the encoding are relevant (such as sampling frequency or resolution) stored are, and that from these reproached frames to the parameters the data stream matching frame instead of every lost frame inserted into this data stream becomes. Detection of Missing Frames (Frames) in audio and video streams according to claim 1 to 3 and to correct the adverse effects this missing frame, characterized, that - in the Audio streams the system stream with the method according to claim 5 missing audio frames added be, and - out the audio data streams the system stream those audio frames are removed, their playing period overlaps with the playing period of missing video frames, and - the number which removed from the audio stream due to lost video frames Audio frames either at the beginning or at the end of the block to be removed Audioframes is reduced so that the sum of play times the previously occurred lost video frames from the sum of the play times the previously removed audio frames by a maximum of the playing time of a different audio or video frames. Detection of Missing Frames (Frames) in audio and video streams and to correct the adverse effects of these missing ones Frame according to one or more of claims 1 to 6 characterized in that that on each of the data stream types present in the system stream (eg. Audio or video frames) or any of those present in the system stream Data formats (eg MPEG 1 Layer 2 or Dolby Digital with an audio data stream) from the method according to claim 4, 5 or 6 a separate Method used to correct the effects of lost frames can be. Detection of Missing Frames (Frames) in audio and video streams according to claim 1 to 3 and for the optional correction of the disadvantageous Effects of these missing frames after one or more of the claims 4 to 7, characterized in that it is part of a demultiplexer or another product such as An authoring product for DVD's, a video editing product for digital Videos, or an encoder whose functionality is a demultiplexing of system currents is required. Device for detecting missing frames in audio and video streams according to claim 1 to 3 and for the optional correction of the disadvantageous Effects of these missing frames after one or more of the claims 4 to 8, characterized, that she - as a standalone solution, or - as part an integrated video digitizing circuit realized is. A method or apparatus for detecting missing frames in audio and video data streams according to claims 1 to 3 and for optionally correcting the adverse effects of said missing frames according to one or more of claims 4 to 8, characterized in that the timestamps are missing Frames have been detected and possibly corrected, - displayed for evaluation purposes, and / or - exported for further processing as a file or on a removable medium, and / or - proposed as crop marks for editing the video, and / or - proposed as chapter marks for the design of menus of a DVD or other interactive medium.