KR20030027253A - Method and apparatus of file formatting and synchronization for multimedia streaming service - Google Patents

Abstract

PURPOSE: A method and an apparatus for formatting and synchronizing files for multimedia streaming service are provided to generate a file including synchronization information to omit a synchronization process at a system level and transmit multimedia streams regardless of a network transmission protocol. CONSTITUTION: When encoded video/audio data is received(S102), the size of video/audio unit with respect to the video/audio data is determined. The header of a stream file is generated according to the determined video/audio unit size(S109). Video/audio unit duration is checked to judge if video unit duration has passed(S111). When the unit duration has passed, the header of video/audio unit is generated(S114). Multimedia video/audio data including the generated video/audio unit header is combined with the header of the stream file, to generate one multimedia stream file.

Description

File formatting and synchronization method for multimedia streaming service and apparatus therefor {Method and apparatus of file formatting and synchronization for multimedia streaming service}

The present invention relates to a file formatting and synchronization method and apparatus for a multimedia streaming service, and more particularly, to a method and apparatus for generating a stream file suitable for an internet streaming service and a local player and synchronizing a generated stream file. That is, the present invention includes synchronization information and order information during file creation and playback (streaming) to improve video / audio synchronization, and multimedia data are multiplexed on the time axis to assist other devices. The synchronization process is easy and quick. In addition, the present invention relates to a method for minimizing the synchronization technique from a server or a player providing a stream service and controlling a synchronization element according to user demand and system performance by providing a method of performing synchronization during file generation and reproduction. In addition, the present invention relates to a file formatting and synchronization method for a multimedia streaming service and a device for selecting a synchronization level according to a system resource level so that synchronization can be selected according to a system.

Currently, there are many problems to be solved in order to provide a multimedia stream service through the Internet. In particular, in providing a stream service, generation of a multimedia stream file having synchronization information and synchronization of video / audio data included in the generated multimedia stream file have been proposed as important issues.

Currently, Microsoft's ASF. Stream file format for providing multimedia streams over the Internet. There is RealPlayer RealMedia file.

ASF consists of Header Object, Data Object and Index Object, and RealMedia file consists of Header Section, Data Section and Index Section. It is composed of a similar structure to each other.

The header object and the header section represent the overall file characteristics. The header object and the header section have attribute information about multimedia data included in the file, such as attributes, stream attributes, and content attributes. The data object and the data section are stored in the data group as the actual multimedia data is stored, and the time values for the transmission time and the presentation time are recorded.

Index objects and index sections are used to access specific parts of the video / audio stream.

The construction and use of such stream files is very complicated at present and generates unnecessary data information. In addition, due to the different formats, a lot of difficulties come when configuring the system. In addition, since the file format simply uses a time stamp value for synchronization information, and synchronization information for video / audio is not correlated with each other, synchronization becomes more difficult to implement. Therefore, a new file format containing synchronization information is needed.

Currently, systems that provide most multimedia streaming services perform synchronization with time stamps. Currently, the synchronization methods performed by the systems providing the multimedia streaming service include a method of checking the synchronization of each received video / audio data and performing buffering and skipping according to the phase change of each data. However, this method requires performance evaluation for the system where the actual synchronization is performed, and causes a waste of system resources due to time check. In addition, the difference between the time value between the time of encoding the video / audio data and the time of decoding the video / audio signal is kept constant, or the difference between the time value between the video / audio decoding time and the actual representation time is kept constant. If the difference exceeds or falls below a certain threshold, skipping frames or synchronizing repeatedly generates many calculations and necessary repetitions to keep the time constant, and causes jitter due to skipping.

When the reference stream is set to be synchronized with a more accurate stream by setting the reference stream among the video / audio data, the overall synchronization is not matched when the reference reference stream is not synchronized.

In addition, when the average value of the bit ratr of video / audio is calculated and the average data amount of video and audio is mixed in the order of video / audio, the synchronization is performed with the average amount of video and audio data. Does not fit. Therefore, the synchronization in the multimedia streaming service requires that the synchronization in the files constituting the actual multimedia be harmonized with each other to produce a true synchronization performance, increase the efficiency of the synchronization, simplify the implementation and waste resources for synchronization. Will be able to prevent.

In order to provide various multimedia services through the Internet, a proper format for video / audio in a stream format is defined. Currently, the Internet provides various types of multimedia data combining video and audio, and users are also demanding and using the multimedia data.

In providing multimedia services through the Internet, the most important problems include bandwidth, network multimedia data processing capability, and synchronization.

In particular, in the synchronization related technology, existing multimedia processing technologies provide synchronization information for multimedia data, but accurate synchronization processing is difficult in actual implementation and operation.

Therefore, there is a need for methods that can easily perform synchronization while providing synchronization information. Currently, most systems suffer from the difficulty of performing synchronization at the system level when transmitting or receiving video / audio data synchronization. In addition, synchronization at the system level has a problem that it is impossible to perform accurate synchronization in the absence of information provided by the video / audio.

Accordingly, the present invention has been made to solve the above-mentioned problems according to the prior art, an object of the present invention, by generating a file containing the synchronization information to skip the synchronization process at the system level (server or media player) The present invention provides a file formatting method and apparatus for a multimedia streaming service for performing a multimedia stream transmission regardless of a network transmission protocol such as TCP / IP or RTP / RTCP.

In addition, another object of the present invention is to perform a synchronization upon playback after file transfer to minimize the technology of synchronization from the server or player providing a stream service and to control the synchronization element according to user requirements and system performance A file synchronization method for a service and an apparatus thereof are provided.

In addition, another object of the present invention is to provide a synchronization while reducing the overhead incurred in the system, by appropriately adjusting the synchronization level according to the performance of the system and the degree of available resources, thereby eliminating the neutral point between resource waste and actual synchronization. The present invention provides a method and apparatus for file formatting and synchronization for a multimedia streaming service. In fact, because the video / audio processing routines are affected by the scheduling and time management of the operating system to perform a new synchronization method must be provided by a new file format that provides synchronization.

1 (a) is a diagram showing the configuration of a synchronized three-layer streaming file for audio / video output synchronization according to the present invention, (b) is a diagram showing the detailed configuration format of the multimedia data shown in (a) .

FIG. 2 (a) is a diagram showing the detailed configuration format of the file header shown in FIG. 1 (a), and (b) is a diagram showing the detailed configuration format of the video / audio unit shown in FIG. 1 (b).

3 is a detailed block diagram of a file formatting apparatus for a multimedia streaming service according to the present invention;

4 is a flowchart illustrating an operation of a file formatting method for a multimedia streaming service according to the present invention.

FIG. 5 is a block diagram illustrating a detailed block configuration of a file synchronization device for a multimedia streaming service according to the present invention. FIG.

6 is a flowchart illustrating an operation of a file synchronization method for a multimedia streaming service according to the present invention.

According to one aspect of the stream file formatting method for a multimedia streaming service according to the present invention for achieving the above object, when the encoded video / audio data is input, the size of the video / audio unit for the input video / audio data Determining; Generating a header of a stream file according to the size of the determined video / audio unit; After generating the file header, checking a video / audio unit duration to determine whether a video unit duration has elapsed; When the unit duration has elapsed, generating a header of the video / audio unit; Generating one multimedia stream file by combining the header of the generated stream file with the multimedia video / audio data including the generated video / audio unit header.

The determining of the size of the video / audio unit may include determining whether there is a sync unit user input for determining the size of the video / audio unit from the user; a) determining that the size of the video / audio unit is generated according to the user input when there is a synchronization unit user input, and b) generating a file header if the synchronization unit user input is not present. Determining whether to determine the size of the audio unit; As a result of the determination, when the size of the video / audio unit is determined on a network packet basis, determining the size of the video / audio unit according to network protocol characteristics.

The determining of the size of the video / audio unit may include calculating a video frame time value corresponding to at least one network frame by comparing the determined size of the video / audio unit with a bit rate of the input video / audio data. step; Determining a video / audio unit duration value using the calculated time value of the video frame, and then generating a file header according to the determined file duration value, wherein generating the header of the stream file comprises: If not generated, performing buffering on the video / audio data until the file header is generated.

The multimedia stream file may include a file header and multimedia data. The multimedia data may include video / audio data necessary for a playback process, and include synchronization information and order information. A first field indicating a file header full length; A second field for determining a size of a video / audio unit constituting a multimedia data portion with a value determined by a user request or a characteristic of a network protocol; A third field indicating descriptive information about the video format as information about the video format of the multimedia data portion; A fourth field indicating a version of the video player; A fifth field indicating the number of video frames to be processed per second in Frame / Second; A sixth field representing a bit rate occurring per second of the actual video stream; A seventh field indicating descriptive information about the audio format as information on the audio format of the multimedia data portion; An eighth field indicating a player version to process audio; A ninth field indicating a number of samples of audio frames to be processed per second; And a tenth field indicating a bit rate occurring per second of the actual audio stream.

The video / audio unit may include a type field for identifying a video / audio unit, a unit length field indicating an overall size of each video / audio unit, and a plurality of video / audio frames. The / audio frame includes a first field indicating an order of the video / audio frame and including order information related to network transmission; A first field indicating a value generated by the video / audio frame encoder in the actual time at which each video / audio frame should be represented; The third field includes the actual length of the video / audio data and the actual video data and the audio data.

Meanwhile, according to an aspect of a method of synchronizing a stream file for a multimedia streaming service according to the present invention, when a formatted multimedia stream file is received, file header information and an FPS value of the received stream file are extracted, and an audio sampling process is performed. Determining a playback time of each frame in the video / audio unit; Detecting video / audio data using the extracted file header, and then identifying system resources to determine at least one video / audio data synchronization check layer; a) when the synchronization check layer is determined, after checking whether the layer can be inspected according to the determined synchronization layer, unit duration check and video / audio unit are detected according to the check result, and b) the extracted video / audio FPS. Determining a video / audio playback time value from the video, FPS and audio synchronization thresholds to compare whether the playback time is included in the threshold region, and c) comparing time stamp values of each frame of the video / audio with each other; As a result of the comparison, when the playback time of each video / audio frame is not included in the critical region and the video / audio frames coincide with each other, it is determined that video and audio data have been synchronized and decode the detected video / audio data. Providing to the device.

The providing of the detected video / audio data to a decoding apparatus may include: checking a video / audio unit duration when a first layer inspection is possible as a result of checking whether the layer is inspectable; Detecting a video / audio unit when the video / audio unit duration has elapsed as a result of the inspection; If a video / audio unit is detected, determining whether a second layer inspection is possible; If possible, determining a video / audio playback time value per second from the extracted video / audio FPS, and determining a synchronization threshold for each frame in the video / audio unit; If the synchronization threshold is determined, comparing the video / audio frame playback time to determine whether a playback time is included in the determined threshold region; Determining whether the third layer inspection is possible when the video / audio playback time is included in the threshold region as a result of the determination; Comparing a time stamp value of each frame of video / audio when the third layer inspection is possible; As a result of the comparison, if the video / audio time stamp values match, determining that the video data and the audio data are synchronized, and transmitting the detected video audio data to the decoding apparatus.

Further, according to an aspect of a stream file formatting apparatus for a multimedia streaming service according to the present invention, an apparatus for formatting video / audio data provided from an encoder into a stream file for a multimedia streaming service, the video encoded from the encoder Unit size determining means for determining a size of a video / audio unit with respect to the input video / audio data, when the audio data is input; File header generating means for generating a header of the stream file according to the size of the video / audio unit determined by the unit size determining means; Unit duration processing means for checking a video / audio unit duration after generating the file header to determine whether a video unit duration has elapsed; Video / audio unit header generating means for generating a header of the video / audio unit when the unit duration has elapsed; And multimedia stream file generating means for generating a multimedia stream file by combining the multimedia video / audio data including the generated video / audio unit header and the generated stream file header.

The unit size determining means may include: a synchronization unit user input processing unit that determines the size of the video / audio unit according to the user input when there is a synchronization unit user input for determining the size of the video / audio unit from the user; If there is no sync unit user input, the network protocol processor may be configured to determine the size of a video / audio unit in packet units according to network protocol characteristics.

The unit size determining unit may further include: a video / audio playback time calculating unit configured to compare a size of the video / audio unit with a bit rate of input video / audio data and to determine a video frame playing time value corresponding to at least one network frame; And determining a video / audio unit duration value using the calculated reproduction time value of the video frame, and providing a determined file duration value to the file header generating means.

According to one aspect of the apparatus for synchronizing a stream file for a multimedia streaming service according to the present invention, when a formatted multimedia stream file is received, file header information and an FPS value of the received stream file are extracted, and an audio sampling process is performed. Input processing means for determining a playback time of each frame in the video / audio unit; First extracting means for extracting multimedia data using the file header extracted from the input processing means; Second extraction means for extracting unit header, frame header and video / audio data in the extracted multimedia data; Layer selection means for identifying the extracted video / audio data and system resources and selecting at least one video / audio data synchronization check layer; When the synchronization check layer is selected, the layer is checked according to the determined synchronization layer and then the video / audio playback time value is determined from the extracted video / audio FPS to determine the video, FPS and audio synchronization thresholds. Synchronization threshold range processing means for determining whether time is included in the threshold region; Time stamp processing means for detecting and comparing time stamp values of each frame of the video / audio; As a result of the comparison, when the playback time of each video / audio frame is not included in the critical region and the video / audio frames coincide with each other, it is determined that video and audio data have been synchronized and decode the detected video / audio data. Synchronization processing means provided to the apparatus.

The synchronization processing module detects the video / audio unit when the first layer inspection is possible, and when the video / audio unit duration has elapsed, as a result of checking whether the layer can be inspected. Detection unit; When the video / audio unit is detected by the detection unit, it is determined whether the second layer inspection is possible, and when the second layer inspection is possible, the video / audio playback time value is determined from the extracted video / audio FPS per second, and the video / audio unit is determined. A synchronization threshold range processor for determining a synchronization threshold for each frame in the audio unit; When the synchronization threshold is determined, it is determined whether the playback time is included in the determined threshold region by comparing the video / audio frame playback time, and if the third layer check is possible when the video / audio playback time is included in the threshold region. The apparatus further includes a comparison judgment unit.

The synchronization processing unit may further include: a time stamp comparison unit comparing the time stamp value of each video / audio frame when a third layer inspection is possible in the comparison determination unit; As a result of the comparison, when the video / audio time stamp value is matched, the video data and the audio data are determined to be synchronized, and further comprising a transmission unit for transmitting the detected video audio data to the decoding apparatus.

On the other hand, in order to perform the stream file formatting method for the multimedia streaming service according to the present invention, a program of instructions that can be executed by the digital processing apparatus is tangibly embodied, and one of the recording media that can be read by the digital processing apparatus. According to an aspect, if encoded video / audio data is input, determining a size of a video / audio unit for the input video / audio data; Generating a header of a stream file according to the size of the determined video / audio unit; After generating the file header, checking a video / audio unit duration to determine whether a video unit duration has elapsed; When the unit duration has elapsed, generating a header of the video / audio unit; And generating one multimedia stream file by combining the multimedia video / audio data including the generated video / audio unit header and the generated stream file header.

In addition, a program of instructions that can be executed by a digital processing apparatus is tangibly embodied in order to perform the method of synchronizing a stream file for a multimedia streaming service according to the present invention, which is a type of recording medium that can be read by a digital processing apparatus. According to an aspect, if a formatted multimedia stream file is received, extracting file header information and an FPS value of the received stream file, and determining a play time of each frame in the video / audio unit through an audio sampling process; Detecting video / audio data using the extracted file header, and then identifying system resources to determine at least one video / audio data synchronization check layer; a) when the synchronization check layer is determined, checking whether the layer can be inspected according to the determined synchronization layer, and then checking the unit duration and the video / audio unit according to the check result, and b) the extracted video / audio FPS. Determining a video / audio playback time value from the video, FPS and audio synchronization thresholds to compare whether the playback time is included in the threshold region, and c) comparing time stamp values of each frame of the video / audio with each other; As a result of the comparison, when the playback time of each video / audio frame is not included in the critical region and the video / audio frames coincide with each other, it is determined that video and audio data have been synchronized and decode the detected video / audio data. It is to perform the steps provided by the device.

Hereinafter, a file formatting and synchronization method and a device for a multimedia streaming service according to an exemplary embodiment of the present invention will be described in detail with reference to the accompanying drawings. In the first embodiment, a file synchronization method and a device for a multimedia streaming service will be described separately in the second embodiment.

First, the configuration of the three-layer streaming file applied to the present invention will be described.

1 (a) is a diagram showing the configuration of a synchronized three-layer streaming file for audio / video output synchronization according to the present invention, (b) is a diagram showing the detailed configuration format of the multimedia data shown in (a) 2 (a) is a diagram showing the detailed configuration format of the file header shown in FIG. 1 (a), and (b) is a diagram showing the detailed configuration format of the video / audio unit shown in FIG. 1 (b). .

The name of the multimedia streaming file shown in FIG. 1 is called a synchronized three-layer streaming file. This file has a simpler structure than other file formats to apply to both the Internet streaming service and the local player.

The synchronized three-layer streaming file shown in FIG. 1 consists of a file header and multimedia data.

The file header shown in FIG. 1 (a) is information required by a receiver for playing a synchronized three-layer streaming file and includes description information and format information about streams included in the file.

The multimedia data portion includes video / audio data necessary for the reproduction process and includes synchronization information and order information. As shown in FIG. 1, the multimedia data portion is composed of video / audio units.

The video / audio unit represents a unit of video / audio to be processed per unit time at the receiving end, and the size of the video / audio unit to be processed varies according to the user's request and the network. The multimedia data portion consists of these consecutive video / audio units.

The file header and the multimedia data will be described in more detail with reference to FIG. 2.

The file header includes information necessary for playing the synchronized three-layer streaming file in a playback stage that receives and plays the synchronized three-layer streaming file as described above.

As shown in FIG. 2, the file header includes a file header length field, a video / audio unit duration field, a video format info field, a video version field, an fps field, a video bit rate field, an audio format info field, an audio version field, and a sampling. It may consist of a Frequency field and an Audio Bit Rate field.

The File Header Length field is a field representing the total length of a file header, and the Video / Audio Unit Duration field is a value determined by user demand or characteristics of a network protocol to determine the size of a video / audio unit constituting a multimedia data part. Used as a value for That is, if the user who creates the file determines that the unit of the unit is one second, video / audio data to be processed in one second is included.

If there is no user request, the system determines the size of the data to be transmitted to the network 1 packet through the characteristics of the network protocol, converts the size into a time value, and determines the video / audio unit duration field.

The Video Format Info field is information about a video format of a multimedia data part and is technical information about a video format. And the Video Version field contains the video processing handler (player) version.

The FPS field indicates the number of video frames to be processed per second in Frame / Second. Video Bit Rate represents the bit rate that occurs per second of the actual video stream.

Audio Format Info is information about an audio format of a multimedia data part and is technical information about an audio format. And the Audio Version field contains the player version to process the audio. The Sampling Frequency field indicates the number of samples of audio frames to be processed per second, and the audio bit rate indicates the bit rate occurring per second of the actual audio stream.

Meanwhile, a configuration of a video / audio unit of multimedia data will be described.

The video / audio unit is composed of a Type field for largely distinguishing a video / audio unit, a Unit Length field indicating the total size of each video / audio unit, and a plurality of video / audio frames.

The plurality of video / audio frames includes a sequence number, a time stamp, a data length, and video / audio data.

Sequence Number indicates the sequence of video / audio frames and may be used as a sequence number related to network transmission.

The Time Stamp records the value generated by the video / audio frame encoder in the actual time each video / audio frame should be represented.

The Data Length represents the actual length of the video data or the audio data, followed by the actual video data and the audio data to form the video / audio unit.

Hereinafter, the apparatus for formatting / synchronizing the streaming file as described above and a method thereof will be described in each embodiment.

(First embodiment)

3 is a block diagram illustrating a detailed block configuration of a file formatting apparatus for a multimedia streaming service according to an embodiment of the present invention. Referring to the configuration, the encoding block 10 includes an encoding unit 10 and a stream file generating unit 100.

The encoding unit 10 includes a video encoder 11 and an audio encoder 12. The stream file generating unit 100 includes a synchronization unit user input processing module 101, a network protocol processing module 102, and a file header generation unit. Module 103, unit duration processing module 104, video / audio unit header generation module 105, video / audio frame header generation module 106, audio detector 107 per unit time, unit time Per video detector 108, stream file buffer 109 and video / audio buffer 110.

The video / audio encoders 11 and 12 of the encoding unit 10 may provide additional information for generating video / audio data and files, time stamps, FPSs, sampling rates, etc. to generate synchronized three-layer streaming files. Provided to the stream file generation unit 100. The user input is then passed to the sync unit user input processing module 101 and used to determine the sync unit.

The synchronization unit user input and network protocol processing modules 101 and 102 of the stream file generation unit 100 determine the size of the video / audio unit to be included in the multimedia portion of the file format according to the characteristics of the user or network protocol, thereby generating the file header module. Provided at 103.

The user who will provide the stream service may determine, via the synchronization unit user input processing module 101, a unit synchronization unit that should be synchronized per hour for his system.

The size of the video / audio unit, which can be determined by the user's request, can be one video / audio frame from the entire video / audio stream.

The synchronization value used in the synchronization unit user input processing module 101 is used in the unit duration processing module 104, and this magnitude is actually set by the time value input in the user input. That is, when the sync unit user input is 1000 ms in a video frame having an FPS of 3, three video frames are determined as one video unit per second.

In addition, the network protocol processing module 102 is intended to improve network transmission efficiency, so that the segmentation (segmentation) or combination (assembly) does not occur during network transmission by varying the size of the video / audio unit constituting the file according to the network characteristics. do. If there is no sync unit user input, the network protocol processing module 102 is driven. If the network protocol processing module 102 is not driven, a value set as default is set in synchronization units.

The file header generation module 103 generates a header of the synchronized three-layer streaming file through the information provided from the video / audio encoders 11 and 12, and then converts the header information of the generated streaming file into a unit duration processing module ( 104).

The video / audio data is stored in the video / audio buffer 110 until the file header is generated through the file header generation module 103. When the file header is generated, substantial multimedia data is generated.

The unit duration processing module 104 drives the video / audio unit header generation module 105, and the unit duration generated by the synchronization unit user input processing module 101 and the network protocol processing module 102 as described above. The value is the top level module that creates the multimedia data by creating a video / audio unit.

In addition to the unit duration processing module 104, all the modules described below are repeated until the video / audio data generated from the video / audio encoders 11 and 12 is finished.

The video / audio unit header generation module 105 generates the video / audio unit header information shown in FIG. 2, and includes a video / audio buffer 110, a video / audio frame generation module 106, and a video / audio detector per unit time. 107 and 108 are driven.

The video / audio frame header generation module 106 generates header information for the video / audio frame shown in FIG. 2, and the video / audio detectors 107 and 108 per unit time are synchronized with the synchronous unit user input processing module 101. The video / audio unit is received from the video / audio buffer 110 by applying the video / audio data size that can be efficiently applied to the unit time generated by the network protocol processing module 102 or the network protocol. It is generated and stored in the stream file buffer 109.

The stream file buffer 109 is a buffer for creating a generated file header and a video / audio unit, and the video / audio buffer 110 is a buffer for storing video / audio data generated from the encoder 10. Is used to store video / audio data when there is a time mismatch between data processing and multimedia data processing.

With reference to FIG. 4 attached to the formatting method using a file formatting apparatus for a multimedia streaming service according to the present invention configured as described above, it will be described in detail step by step.

4 is a flowchart illustrating an operation of a file formatting method for a multimedia streaming service according to the present invention.

First, video / audio information and data are received from the encoder 10 (S101).

Subsequently, in order to determine a synchronous user input, that is, a size of a video / audio unit, a synchronous unit user input process is performed from the user (S103).

That is, it is determined whether the user's input exists (S104), and if the user's input for determining the size of the video / audio unit exists, the file header generation module 103 generates a file header according to the user's input. It is made (S108).

However, if there is no user input for determining the size of the video / audio unit, the network protocol processing module 102 processes the video / audio data in units of network packets in consideration of network characteristics (S105).

At this time, it is determined whether to process the video / audio data in the network packet unit (S106), and as a result of determining the size of the video / audio unit according to the network protocol characteristics when processing the video / audio data in the network packet unit will be. The size of the video / audio unit determined as described above is calculated by comparing the video / audio bit rate with the actual video frame time value corresponding to one network frame. The file header is generated through the illustrated file header generation module 103 (S108).

If the video / audio data is not processed in units of network packets in step S106, the size of the video / audio unit is determined by applying a unit duration set as a default (S107).

Subsequently, it is determined whether the file header is generated (S109). If the file header is not generated, buffering of the video / audio data is performed until the file header is generated (S110). The generation of the file header uses information provided from the video / audio encoder 10, and continuously buffers the video / audio data until the file header is generated.

When all files are generated in step S109, the unit duration value generated by a user input or a network protocol is checked (S211).

As a result of the check, if the unit duration has not elapsed, it continuously checks whether the unit duration has elapsed while continuously performing the buffering operation for the video / audio (S113). On the contrary, if the unit duration has elapsed, the video / audio unit A header portion is generated (S114).

Subsequently, when the header of the video / audio unit is generated, the video / audio data is detected and the header is generated (S115), and the video / audio data including the generated header is stored in the stream buffer to generate a stream file ( S116, S117).

(Second embodiment)

A second embodiment of the present invention relates to an apparatus and a method for synchronizing audio and video data at a receiving end when a stream file generated in the first embodiment is transmitted, which will be described in detail with reference to the accompanying drawings. Let's do it.

FIG. 5 is a block diagram illustrating a detailed block configuration of a file synchronization device for a multimedia streaming service according to the present invention. In view of the configuration, a streaming file synchronization unit 300 for performing synchronization of a received multimedia stream file and streaming is shown. The decoder 20 is configured to decode the video / audio signal synchronized with the file synchronizer 300. Here, the decoding unit 20 may be composed of a video decoder 21 and an audio decoder 22.

The streaming file synchronization unit 300 may include a file header processing module 301, an FPS processing module 302, an audio sampling processing module 303, a video / audio unit processing module 304, and a video / audio unit header processing module. 305, video / audio frame header processing module 306, video / audio data extraction module 307, sync layer processing module 308, system resource monitoring module 309, synchronization correction module 310, video / Audio synchronization processing module 311, video / audio synchronization threshold range processing module 312, video / audio time stamp processing module 313, and video / audio decoder transmission module 314.

The file header processing module 301 performs separation of the synchronized three-layer streaming file header, and from the separated file header, the FPS processing module 302 determines the presentation time values at which each video frame should be played. Then, the start time and the end time for which one frame must be expressed are calculated in consideration of the presentation time between the frames. That is, the presentation time threshold region for each video frame is determined.

The audio sampling processing module 303 also calculates the audio sampling value to calculate the time at which each audio frame should be played and to determine its threshold region.

When the analysis of the multimedia stream file header is completed by the file header processing module 301, processing of the multimedia data is started by the video / audio unit processing module 304, and other than the video / audio unit processing module 308. The other modules are called repeatedly by the video / audio unit processing module 304.

The video / audio unit processing module 304 is responsible for extracting the multimedia data portion of the synchronized three-layer streaming file, and is a video / audio unit header processing module 305 which is a substantial multimedia data processing process, video / audio The frame header processing module 306 and the video / audio data extraction module 307 are called and the sync layer processing module 308 is called to select a synchronization layer based on the extracted data.

The video / audio unit header processing module 305 and the video / audio frame header processing module 306 extract, analyze, and process header information related to the video / audio unit and the frame.

The video / audio data extraction module 307 extracts video / audio data and header information based on the information provided from the video / audio frame header processing module 306 to provide data to the video / audio decoder transmission module 314. send.

The system resource monitoring module 309 is a module for measuring system resources. The system resource monitoring module 309 determines a synchronization layer in consideration of a CPU, a storage device, and a network bandwidth. The sync layer dynamically determines the sync layer whenever there is a change in the layer within the range allowed by the system. The sync layer processing module 308 determines the sync layer based on the information determined by the system resource monitoring module 309.

The sync layer is divided into three types.

In the first layer, only video / audio units synchronize. Since the synchronization is performed using only the unit duration value of the file header, the specific unit duration is checked and only the corresponding synchronized units are transmitted to the decoder 20. Therefore, the synchronization process is simple, but accurate synchronization is achieved according to system characteristics. Has the disadvantage of being difficult.

The second layer performs synchronization by calculating an expected play time and an estimated play time threshold for each frame of the video / audio included in the video / audio unit through the video FPS and the audio sampling layer. will be. For each frame included in the video / audio unit, the video / audio synchronization threshold range processing module 312 shown in FIG. 5 compares the expected playback time and the expected playback time threshold range of each frame by its scheduling method. When each video / audio frame is included in the threshold range, the video / audio frames are played.

Such a second-layer can perform synchronization for each frame with only the FPS and sampling layer while saving system resources relatively without time stamp value for each frame.

The third layer does not immediately play the video / audio frames included in the critical area by the video / audio synchronization threshold range processing module 312 shown in FIG. 5, that is, the time stamp value included in each frame header, that is, in FIG. 3. The video / audio frame is reproduced at the actual reproduction time by comparing the video / audio representation time values output from the illustrated encoder 10.

The video / audio synchronization processing module 311 shown in FIG. 5 processes the first layer described above, the video / audio synchronization threshold range processing module 312 processes the second layer described above, and a video / audio time stamp. The processing module 312 is to perform the third layer synchronization described above.

In addition, the synchronization correction module 310 is driven by the synchronization layer processing module 308, and the video / audio synchronization processing module 311 and the video / audio synchronization threshold processing module (3) used for synchronization to the third layer. 312), the video / audio time stamp processing module 313 is called, and the first, second, and third layers are sleeped or woken by system scheduling to enhance process processing efficiency.

The synchronization correction module 310 performs synchronization on the data transmitted to the video / audio decoder transmission module 314 based on the synchronization layer generated by the synchronization layer processing module 308. The video / audio decoder transmission module 314 is a video / audio decoder 21, 22 based on the synchronization information provided by the synchronization correction module 310 to the video / audio data extracted by the video / audio data extraction module 307. To send).

A synchronization method using a file synchronization device for a multimedia streaming service having the above configuration will be described step by step and in detail with reference to FIG. 6.

First, when a multimedia stream file is input in the form of a network or a local file (S201), stream file header information included in the input stream file and a video FPS are extracted to determine a playback time of each frame in the video unit (S202). , 203).

In addition, the audio sampling process is performed on the input stream file to determine the reproduction time of each frame in the audio unit (S204).

When the header is analyzed as described above, the system resource is checked (S205) to detect the synchronization layer (S206). That is, after checking system resources, the synchronization layer can determine the synchronization layer.

The system resource check process is involved in all the processes in which synchronization is performed, and adjusts the synchronization layer at regular time intervals.

If it is determined that the inspection of the first layer is possible as a result of the synchronization detection (S207), and if the inspection of the first layer is not possible as a result of the determination, the video / audio data is detected from the input multimedia stream file and transmitted to the video / audio decoder. (S208).

However, in step S207, when the first layer inspection is possible, the unit duration is checked (S209), and it is determined whether the unit duration has elapsed (S210).

As a result, if the unit duration has elapsed, the video / audio unit is detected (S211). If the unit duration has not elapsed, the unit duration is continuously checked until the unit duration has elapsed.

If the video / audio unit is detected in step S211, it is checked whether a second layer inspection is possible through system resources (S212).

As a result of the check, if the second layer inspection is not possible, video / audio data is detected and the detected video / audio data is transmitted to the decoder 20. If the second layer inspection is possible, the video FPS processing and the audio sampling processing are performed. The video / audio expression calculation value per second, that is, the video / audio playback time value, is determined based on the value calculated from S213.

The synchronization threshold for each frame in the video / audio unit is determined (S214).

Subsequently, the synchronization correction module 310 illustrated in FIG. 5 compares the play time of each video / audio frame (S215) and checks whether the play time of each video / audio frame is included in the determined threshold area (S216). .

As a result of the inspection, when the video / audio frame reproduction time is not included in the critical region, synchronization is corrected (S217). That is, after slipping the second layer inspection process through the synchronization correction module 310 illustrated in FIG. 5, the availability of the third layer is checked.

However, if the playback time of each video / audio frame is included in the critical region in step S216, it is determined whether the inspection of the third layer is possible (S218).

If it is determined that the third layer inspection is impossible, the video / audio data detected in the critical region is transmitted to the decoder 20 (S222). On the contrary, when the inspection of the third layer is possible, the time stamp value of each video / audio frame is determined. In operation S219, it is determined whether the time stamp values of the video / audio match each other (S220).

If the time stamp values of the video / audio frames do not coincide with each other, after synchronization is corrected (S221), the time stamp values of the video / audio frames are compared. That is, synchronization correction is continuously performed until the time stamp values of the video / audio frames coincide with each other.

On the contrary, in step S220, when the time stamp values of the video / audio frames coincide with each other, the detected video / audio data is transmitted to the decoder 20 (S222). Here, the comparison of the time stamps with respect to the frames of video / audio is performed by the synchronization correction module 310 shown in FIG. 5, and when the time stamps of the video / audio frames do not coincide with each other, the inspection of the third layer is performed. Sleep and wake up at any time by applying a new schedule.

Subsequently, when the one-cycle data synchronization process is completed, it is determined whether the video / audio data is over (S223). When the video / audio data is over, the synchronization operation is terminated, and another video / audio data is synchronized. In this case, it is fed back to step S206 to determine a new synchronization layer to perform a synchronization process of the multimedia stream file.

The method and apparatus for file formatting and synchronization for a multimedia streaming service according to the present invention as described above are related to an apparatus and method for generating a synchronized multimedia stream file suitable for an internet streaming service and a local player and synchronizing the generated stream file. By multiplexing and storing multimedia data, it creates files suitable for streaming service and local player to minimize the technology of synchronization from the server or player providing the stream service and control the synchronization elements according to user demand and system performance. It is. Also available in Internet Protocol and RTP / RTCP, the new synchronization method allows for increased synchronization and provides a synchronized file format, making it easy to implement synchronization in many applications that need to provide multimedia streaming services. There is an advantage to that. And by providing synchronization in consideration of system performance and resources, there is an advantage that can be used in many other systems.

Claims (29)

In the stream file formatting method for a multimedia streaming service, If encoded video / audio data is input, determining a size of a video / audio unit for the input video / audio data; Generating a header of a stream file according to the size of the determined video / audio unit; After generating the file header, checking a video / audio unit duration to determine whether a video unit duration has elapsed; When the unit duration has elapsed, generating a header of the video / audio unit; Generating a multimedia stream file by combining the header of the generated stream file with the multimedia video / audio data including the generated video / audio unit header. The method of claim 1, Determining the size of the video / audio unit, Determining whether there is a synchronization unit user input for determining the size of the video / audio unit from the user; a) if the synchronization unit user input exists, determine the size of the video / audio unit according to the user input to generate a file header, b) if there is no sync unit user input, determining whether to determine the size of the video / audio unit on a network packet basis; And determining the size of the video / audio unit in accordance with network protocol characteristics when determining the size of the video / audio unit in units of network packets. The method of claim 2, If the size of the video / audio unit is not determined in the network packet unit as a result of the determination, the size of the video / audio unit is determined by applying a unit duration set as a default. The method according to claim 1 or 2, Determining the size of the video / audio unit comprises: Calculating a video frame time value corresponding to at least one network frame by comparing the determined size of the video / audio unit with a bit rate of the input video / audio data; And determining a video / audio unit duration value using the calculated time value of the video frame, and then generating a file header according to the determined file duration value. The method of claim 1, In the generating of the header of the stream file, if the stream file is not generated, buffering the video / audio data until the file header is generated. . The method of claim 1, In the step of determining whether the unit duration has elapsed, if the video unit duration has not elapsed, performing a buffering for the video / audio until the unit duration has elapsed, the method for formatting a stream file for a multimedia streaming service . The method of claim 1, The multimedia stream file, Stream file formatting method for a multimedia streaming service comprising a file header and multimedia data. The method of claim 7, wherein The multimedia data, A stream file formatting method for a multimedia streaming service comprising a plurality of video / audio units including video / audio data necessary for a playback process and including synchronization information and order information. The method of claim 7, wherein The file header, A first field indicating a file header full length; A second field for determining a size of a video / audio unit constituting a multimedia data portion with a value determined by a user request or a characteristic of a network protocol; A third field indicating descriptive information about the video format as information about the video format of the multimedia data portion; A fourth field indicating a version of the video player; A fifth field indicating the number of video frames to be processed per second in Frame / Second; A sixth field representing a bit rate occurring per second of the actual video stream; A seventh field indicating descriptive information about the audio format as information on the audio format of the multimedia data portion; An eighth field indicating a player version to process audio; A ninth field indicating a number of samples of audio frames to be processed per second; A stream file formatting method for a multimedia streaming service comprising a tenth field indicating a bit rate occurring per second of the actual audio stream. The method of claim 8, The video / audio unit, Type field to distinguish the video / audio unit, A Unit Length field indicating the total size of each video / audio unit, Stream file formatting method for a multimedia streaming service comprising a plurality of video / audio frames. The method of claim 10, The plurality of video / audio frames, A first field indicating an order of video / audio frames and including order information related to network transmission; A first field indicating a value generated by the video / audio frame encoder in the actual time at which each video / audio frame should be represented; A stream file formatting method for a multimedia streaming service comprising a real length of video / audio data and a third field containing real video data and audio data. In the stream file synchronization method for a multimedia streaming service, If a formatted multimedia stream file is received, extracting file header information and FPS value of the received stream file and determining the playback time of each frame in the video / audio unit through an audio sampling process; Detecting video / audio data using the extracted file header, and then identifying system resources to determine at least one video / audio data synchronization check layer; a) Once the synchronization check layer is determined, it is checked whether the layer can be inspected according to the determined synchronization layer, and according to the check result, the unit duration check and the video / audio unit are detected. b) determine a video / audio playback time value from the extracted video / audio FPS to determine a video, FPS, audio synchronization threshold and compare whether the playback time is included in the threshold region; c) comparing the time stamp values of each frame of the video / audio with each other; As a result of the comparison, when the playback time of each video / audio frame is not included in the critical region and the video / audio frames coincide with each other, it is determined that video and audio data are synchronized and decode the detected video / audio data. Stream file synchronization method for a multimedia streaming service comprising the step of providing to the device. The method of claim 12, The synchronization check layer, A first layer that detects a video / audio unit by checking a duration value of the video / audio unit; When the video audio / unit is detected through the first layer inspection, a video / audio playback time value is calculated per second from the video / audio FPS in the detected video / audio unit, and the video / audio synchronization threshold is calculated according to the calculated playback time value. A second layer that determines whether the video data is included in the threshold region by comparing the playback time of each video frame and the audio frame; Stream file synchronization for a multimedia streaming service including a third layer that compares the time stamp values of each frame of video / audio by comparing the time stamp values of each frame of video / audio when the playback time is included in the critical region through the second layer check Way. The method according to claim 12 or 13, Providing the detected video / audio data to a decoding device, Checking a video / audio unit duration when a first layer inspection is possible as a result of checking whether the layer can be inspected; Detecting a video / audio unit when the video / audio unit duration has elapsed as a result of the inspection; If a video / audio unit is detected, determining whether a second layer inspection is possible; If possible, determining a video / audio playback time value per second from the extracted video / audio FPS, and determining a synchronization threshold for each frame in the video / audio unit; If the synchronization threshold is determined, comparing the video / audio frame playback time to determine whether a playback time is included in the determined threshold region; Determining whether the third layer inspection is possible when the video / audio playback time is included in the threshold region as a result of the determination; Comparing a time stamp value of each frame of video / audio when the third layer inspection is possible; As a result of the comparison, when the video / audio time stamp values match, determining that the video data and the audio data are synchronized, and transmitting the detected video and audio data to the decoding apparatus. . The method of claim 14, When the inspection of the first layer is impossible, the video / audio data is detected. When the inspection of the first layer is possible but the inspection of the second, layer, and third layer is impossible, respectively, the detected video audio data is transmitted to the decoding apparatus. Stream file synchronization method for multimedia streaming service. The method of claim 14, In the determining of whether the video data is included in the critical region, when the video / audio playback time is not included in the critical region, the second layer inspection process is slipped, and after a predetermined time, the playback time of each video / audio frame is compared again. Stream file synchronization method for a multimedia streaming service. The method of claim 14, In the step of comparing the time stamp value of each video / audio frame, When the time stamp value of each video / audio frame is inconsistent, a third layer check is performed, and a new schedule is applied to the multimedia streaming service for comparing the time stamp value of each video / audio frame again after a predetermined time. How to sync stream files. An apparatus for formatting video / audio data provided from an encoder into a stream file for a multimedia streaming service, the apparatus comprising: Unit size determining means for determining a size of a video / audio unit for input video / audio data when encoded video / audio data is input from an encoder; File header generating means for generating a header of the stream file according to the size of the video / audio unit determined by the unit size determining means; Unit duration processing means for checking a video / audio unit duration after generating the file header to determine whether a video unit duration has elapsed; Video / audio unit header generating means for generating a header of the video / audio unit when the unit duration has elapsed; A stream for multimedia streaming service comprising means for generating a multimedia stream file by combining the multimedia video / audio data including the generated video / audio unit header and the generated stream file header File Formatting Device. The method of claim 18, The unit size determining means, A synchronization unit user input processing unit for determining the size of the video / audio unit according to the user input when there is a synchronization unit user input for determining the size of the video / audio unit from the user; And a network protocol processing unit configured to determine the size of a video / audio unit on a packet basis according to network protocol characteristics when there is no sync unit user input. The method of claim 19, If the network protocol processor fails to determine the size of the video / audio unit in units of network packets, the stream file formatting apparatus for determining a size of the video / audio unit by applying a unit duration set as a default. The method of claim 18 or 19, The unit size determining means, A video / audio playback time calculator for comparing a size of the video / audio unit with a bit rate of input video / audio data to calculate a video frame playback time value corresponding to at least one network frame; And a duration determiner configured to determine a video / audio unit duration value by using the calculated reproduction time value of the video frame, and to provide the determined file duration value to the file header generation means. The method of claim 18, A first buffer which performs buffering on video / audio data until a file header is generated when the stream file is not generated by the file header generating means; And a second buffer for performing video / audio buffering until the unit duration has elapsed when the unit duration processor has not elapsed. An apparatus for synchronizing video / audio data of a stream file formatted for a multimedia streaming service, Input processing means for extracting the file header information and the FPS value of the received stream file when the formatted multimedia stream file is received, and determining the reproduction time of each frame in the video / audio unit through audio sampling processing; First extracting means for extracting multimedia data using the file header extracted from the input processing means; Second extraction means for extracting unit header, frame header and video / audio data in the extracted multimedia data; Layer selection means for identifying the extracted video / audio data and system resources and selecting at least one video / audio data synchronization check layer; When the synchronization check layer is selected, the layer is checked according to the determined synchronization layer and then the video / audio playback time value is determined from the extracted video / audio FPS to determine the video, FPS and audio synchronization thresholds. Synchronization threshold range processing means for determining whether time is included in the threshold region; Time stamp processing means for detecting and comparing time stamp values of each frame of the video / audio; As a result of the comparison, when the playback time of each video / audio frame is not included in the critical region and the video / audio frames coincide with each other, it is determined that video and audio data are synchronized and decoded the detected video / audio data. Stream file synchronization apparatus for a multimedia streaming service comprising a synchronization processing means provided to the device. The method of claim 23, wherein The synchronization check layer in the layer selection means, A first layer that detects a video / audio unit by checking a duration value of the video / audio unit; When the video audio / unit is detected through the first layer inspection, a video / audio playback time value is calculated per second from the video / audio FPS in the detected video / audio unit, and the video / audio synchronization threshold is calculated according to the calculated playback time value. A second layer which determines whether the video data is included in the critical area by comparing the playback time of each video frame and the audio frame; Stream file synchronization for a multimedia streaming service including a third layer that compares the time stamp values of each frame of video / audio by comparing the time stamp values of each frame of video / audio when the playback time is included in the critical region through the second layer check Device. The method of claim 23 or 24, The synchronization processing module, If it is determined that the inspection of the layer is possible, if the first layer inspection is possible, the multimedia streaming service includes a detector for detecting the video / audio unit duration and detecting the video / audio unit when the video / audio unit duration has elapsed. Stream file synchronization device for. The method of claim 25, The synchronization processing module, When the video / audio unit is detected by the detection unit, it is determined whether the second layer inspection is possible, and when the second layer inspection is possible, the video / audio playback time value is determined from the extracted video / audio FPS per second, and the video / audio unit is determined. A synchronization threshold range processor for determining a synchronization threshold for each frame in the audio unit; When the synchronization threshold is determined, it is determined whether the playback time is included in the determined threshold region by comparing the video / audio frame playback time, and if the third layer check is possible when the video / audio playback time is included in the threshold region. Stream file synchronization apparatus for a multimedia streaming service further comprising a comparison determination unit. The method of claim 26, Synchronization processing means, A time stamp comparison unit comparing the time stamp values of each video / audio frame when the third layer inspection is possible in the comparison determination unit; As a result of the comparison, when the video / audio time stamp values match, the stream file synchronization for the multimedia streaming service further includes a transmission unit for determining that the video data and the audio data are synchronized and transmitting the detected video audio data to the decoding apparatus. Device. In the recording medium which can be read by the digital processing apparatus, there is tangibly implemented a program of instructions that can be executed by the digital processing apparatus to perform the stream file formatting method for the multimedia streaming service. If encoded video / audio data is input, determining a size of a video / audio unit for the input video / audio data; Generating a header of a stream file according to the size of the determined video / audio unit; After generating the file header, checking a video / audio unit duration to determine whether a video unit duration has elapsed; When the unit duration has elapsed, generating a header of the video / audio unit; A recording medium for generating a multimedia stream file by combining the multimedia video / audio data including the generated video / audio unit header with the generated stream file header In order to perform a stream file synchronization method for a multimedia streaming service, a program of instructions executable by a digital processing apparatus is tangibly embodied, and in a recording medium that can be read by a digital processing apparatus, If a formatted multimedia stream file is received, extracting file header information and FPS value of the received stream file and determining the playback time of each frame in the video / audio unit through an audio sampling process; Detecting video / audio data using the extracted file header, and then identifying system resources to determine at least one video / audio data synchronization check layer; a) Once the synchronization check layer is determined, it is checked whether the layer can be inspected according to the determined synchronization layer, and according to the check result, the unit duration check and the video / audio unit are detected. b) determine a video / audio playback time value from the extracted video / audio FPS to determine a video, FPS, audio synchronization threshold and compare whether the playback time is included in the threshold region; c) comparing the time stamp values of each frame of the video / audio with each other; As a result of the comparison, when the playback time of each video / audio frame is not included in the critical region and the video / audio frames coincide with each other, it is determined that video and audio data are synchronized and decoded the detected video / audio data. A recording medium performing the step of providing to the device.