KR20100127896A - Multimedia supporting system and method thereof - Google Patents

Abstract

PURPOSE: A multimedia supporting system for controlling the locking time of video and audio is provided to remove stream in the memory space and control the locking time of video and audio. CONSTITUTION: A multimedia support system(100) includes an encoder(10), a memory unit(20), a bit control unit(30), and a transmission unit(40). The encoder encodes input media. The memory unit stores the stream outputted from the encoder. The bit control unit controls output bit rate about the stored stream in the memory unit. The transmission unit controls the size about a data packet controlled by the bit control unit.

Description

Multimedia support system and method thereof

The present invention relates to a multimedia support system and a method for supporting the same, and more particularly, to a multimedia support system capable of quickly consuming or removing a stream in a memory space and adjusting a synchronization time between audio and video, and a support method thereof. It is about.

Recently, as the subject of the production and processing of media has been changed to individuals, devices for transmitting media such as video in real time or corresponding software have become important. For example, a user may provide news by using Internet contents such as User Created Contents (UCC) or provide a video service using a Voice over Internet Protocol (VoIP) communication network. In addition, the spread of various communication devices has made it possible for users to easily produce and distribute multimedia.

1 is a diagram schematically illustrating a general multimedia support system. Referring to the drawings, a video source input from an image source such as a video camera or a digital broadcast is processed by an image signal processor 1 to be transmitted to the video encoder 2 through a video interface, and also an uncompressed sound source. The audio signal is processed by the audio signal processing unit 3 and transmitted to the audio encoder 4 through an audio interface. The signal synthesizing unit 5 mixes the encoded video signal and the encoded audio signal to transmit the transmission unit 6. Through IP-based protocol, it is delivered to the desired terminal. At this time, if there is a file or a plurality of contents incidentally, a streaming server managing the same may be provided to respond to an external request.

On the other hand, in the case of wirelessly transmitting streaming based on IP as described above, the media stream generated from each encoder is temporarily stored in the memory space and then transmitted in an available size according to the network condition, and the network condition is not good. The size of the output stream should be limited so that there is not enough space in the memory so that there is no data that cannot be transmitted or lost. At this time, the stream before the size change is left in the memory, so the stream of the changed size is transmitted after being transmitted and cannot be quickly resized. Therefore, the stream in the memory space must be exhausted or removed quickly.

In addition, even when the external image source is changed from the terminal, since the previous data still remains in the memory, it is necessary to consume it and adjust the synchronization time of the audio and the video.

The present invention was devised to meet the above requirements, and provides a multimedia support system and a method for supporting the same, which can quickly consume or remove a stream in a memory space and control the synchronization time of audio and video. For the purpose of

A first embodiment of a multimedia support system according to the present invention for achieving the above object comprises: an encoder for encoding input media; A memory unit for temporarily storing a stream input from the encoder; A bit control unit controlling an output bit rate of the stream stored in the memory unit; And a transmitter for controlling and transmitting a size or a transmission interval for the data packet controlled by the bit controller.

The multimedia support system may further include an application unit configured to determine a stream type of the media based on an output bit rate controlled by the bit controller.

In addition, the multimedia support system may further include a parser for controlling the size of the bit string controlled by the bit control unit to be used by the application unit.

Here, in the case of a key frame, the parser may adjust the length of the bit string by using an identifier indicating the end of the block used by the encoder.

In addition, in the case of a bidirectional predictive frame, the parser preferably uses a special bit string including a copy mode.

In the case of a prediction frame, it is preferable that the parser uses a key frame and a bidirectional prediction frame.

Advantageously, said bit control section controls the output bit rate based on a network state in a transport layer and a network state in an IP layer.

Here, the control of the output bit rate by the bit control unit can be expressed as follows.

Here, ω ₁ ω ₂ is a weighting factor, Coeff _tr is a network state related expression coefficient at the transport layer, and Coeff _ip is a network state related expression coefficient at the IP layer.

According to a second embodiment of a multimedia support system according to the present invention for achieving the above object, when an image source change request signal is input from the outside, a change of a predetermined value or more is detected based on a motion vector value and a prediction error value. A scene detector; And a stream mixer for smoothly transitioning an image by mixing the stream remaining in the memory unit with a new stream when a change is detected by the scene detector.

Here, when the change is detected by the scene detection unit, the multimedia support system adjusts the time stamp during the transmission time based on the time stamp of the bit string remaining in the memory unit and the time stamp of the newly created bit string. It may further include a stamp adjusting unit.

On the other hand, the multimedia support system, the step of encoding the input media; Temporarily storing the stream output from the encoding step; Controlling an output bit rate for the stored stream; And controlling the size or transmission interval of the data packet whose bit rate is controlled by the bit rate control step and transmitting the same.

Preferably, the multimedia support method may further include determining a stream type for the media based on the output bit rate controlled by the bit rate control step.

In this case, the bit rate control step of controlling the output bit rate based on the network state in the transport layer and the network state in the IP layer.

In addition, the control of the output bit rate by the bit rate control step can be expressed as follows.

Here, ω ₁ ω ₂ is a weighting factor, Coeff _tr is a network state related expression coefficient at the transport layer, and Coeff _ip is a network state related expression coefficient at the IP layer.

The multimedia support method may further include detecting a change of a predetermined value or more based on a motion vector value and a prediction error value when an image source change request signal is input from the outside; And when the change is detected by the scene detector, smoothly transitioning the image by mixing the stream remaining in the memory unit with a new stream.

In addition, the multimedia support method, if a change is detected by the sensing step, further comprising the step of adjusting the time stamp during the transmission time based on the time stamp of the bit string remaining in the memory and the time stamp of the newly created bit string. It may also include.

Thus, the multimedia support system and its support method according to the present invention can quickly consume or remove the stream in the memory space, it is possible to adjust the synchronization time of the audio and video.

Hereinafter, with reference to the accompanying drawings will be described a preferred embodiment of the present invention. However, the embodiments of the present invention described below are provided to enable those skilled in the art to more easily understand the present invention, and the scope of the present invention is not limited to the described embodiments.

2 is a diagram schematically illustrating a first embodiment of a multimedia support system according to the present invention. Referring to the drawings, the multimedia support system 100 according to the first embodiment of the present invention includes the encoder 10, the memory unit 20, the bit manufacturing 30, the transmission unit 40, the application unit 50, The parser 60, the state monitoring part 70, and the IP management part 80 are provided. Here, FIG. 2 is a diagram illustrating a multimedia support system serving as a transmitter, and in the case of acting as a receiver, the reverse order of FIG. 2 is performed. Elements to add may be added.

The encoder 10 encodes the input multimedia. In this case, the encoder 10 includes a video signal encoder for encoding a video signal and a voice signal encoder for encoding a voice signal. The encoder 10 may be in the form of a hardware or software real-time encoder or a stored video container file format interpreter. In the case of the real time encoder, the parameter about the size of the output bit string or the compression ratio of the encoder may be changed and input from the application unit 50 in real time. The compressed media data generated by the encoder 10 is stored in the buffer 20 before being transmitted through the network, and the data in the buffer (memory space) 20 is read and transmitted by the application unit 50 by a predetermined portion. However, in this case, when the bit rate is lowered according to the network condition, the large data still needs to be sent, which immediately affects playback. To this end, the bit control unit 30 adjusts the size of the bit string usable by the application unit 50.

In this case, the size of the input may use a state coefficient obtained from the IP manager 70 and a state coefficient in the transport layer, and may be expressed as in Equation 1 below.

Here, ω ₁ ω ₂ are weight coefficients, respectively, and are determined at the time of system design. In addition, Coeff _tr is a network state related expression coefficient in the transport layer, and has a value between 0 and 1. In addition, Coeff _ip is a network state related expression coefficient in the IP layer and has a value between 0 and 1. Here, both coefficients have a value of 1 when they are not used.

The memory unit 20 is formed as a buffer for temporarily storing the stream output from the encoder 10.

The bit control unit 30 controls the output bit rate for the stream stored in the memory unit 20 and transmits it to the transmission unit 40. At this time, the bit controller 30 adjusts the size through the parser 60.

The application unit 50 determines the stream type for the multimedia based on the output bit rate controlled by the bit control unit 30. In addition, the application unit 50 transmits data from the encoder 10 to the transmission unit 40 with respect to the availability of the media or an external video in real time according to an external request. In addition, the size of the output bit stream of the bit control unit 30 and the encoder 10 is controlled in accordance with the transmission state of the transmission unit 40 or the network state from the IP manager 70.

The parser 60 interprets the stream controlled by the bit control unit 30 and controls the size of the bit string controlled by the bit control unit 30 so as to be usable by the application unit 50. The parser 60 is a module of a decoder corresponding to the encoder 10 supported by the terminal. The parser 60 can be used to reduce the size in the following manner. In the case of keyframes, the length of the bit string is adjusted by using an identifier indicating the end of the block used in the block unit encoder as in MPEG or JPEG. do. In the case of B-Frame, since the motion vector is used, a special bit string, which is one of short bit strings such as Copy Mode, can be used. Predictive frame method can be used. In addition, the key frame having the largest output size may be reduced by changing the output of the key frame to a predetermined portion of the P and B frames and changing the reference frame position. In addition, since the end of every block is known, the size of the transmission packet can be easily determined as compared with the size of the entire frame. The error correction code unit (not shown) that detects and corrects data lost during network transmission can use the parser to know the size of each block or the size of the frame. Since it is more efficient to control the buffer of the network buffer or the decoder, the effective packet size and the error code size may be determined by reflecting this. In addition, correction codes may be added by omitting or varying lengths according to network conditions. The output bit string of a predetermined size is transmitted to the transmitter and may be divided in size according to the state of the transmitter.

The state monitoring unit 70 monitors the state of the transmission unit 40, and the IP manager 80 converts the form of the network packet of the stream transmitted according to the network state. The transmitter 40 may know the network state by using the transmission protocol used. In the case of TCP, the network status can be checked by checking the remaining buffer size or checking the data request cycle from the transmission buffer control (sliding) method. The time at which each packet is transmitted, the average time, etc. may be exchanged between the terminals in the form of a message, or a request interval may be used. Use this when you know the delay or jitter in the protocol directly or indirectly, such as RTP. By using this network state, it is possible to control the output size of the bit controller and the encoder, and to control the size and transmission interval of the packet to be transmitted. The packet transmitted to the IP manager 80 changes the TCP / UDP type according to the network status using the IP manager. The IP layer classifies each packet by port number. If the network condition is good or the transmission delay is low, the packet is transmitted using TCP. Otherwise, the packet is transmitted using UDP.

The data used for UDP and TCP are the same data received by the application. In case of UDP, the error detection code is optional. The terminal receiving this can also distinguish the final destination of the input packet and the required application by using the port number in use by the application. TCP or UDP can be identified by using the protocol field in the IP header. The network state may adjust the size, transmission, and reception interval of a packet transmitted using the state input to the IP manager 80 or the transmitter 40.

3 is a diagram schematically illustrating a second embodiment of a multimedia support system according to the present invention. Referring to the drawings, the multimedia support system according to the second embodiment includes an encoder 10, a memory unit 20, a bit controller 30, an application unit 50, a parser 60, a scene detector 110, A stream mixer 120 and a timestamp adjuster 130. Here, the same reference numerals are assigned to the same components as the components of FIG. 2, and their detailed functions and operations will be omitted.

When the image source change request signal is input from the outside, the scene detector 110 detects a change of a predetermined value or more based on a motion vector value and a prediction error value.

When a scene change is detected by the scene detector 110, the stream mixer 120 mixes the stream remaining in the memory 20 and a newly input stream to smoothly transition the image.

When a scene change is detected by the scene detector 110, the timestamp adjuster 130 based on the timestamp of the bit string remaining in the memory unit 20 and the timestamp of the newly created bit string based on the timestamp during the transmission time. Adjust

The terminal may receive a change request for a channel or an external video source, and in this case, the data loaded in the memory makes it difficult to immediately play the changed data in the receiving unit. It belongs to B and P frames, which degrades the motion vector detection performance of the encoder. At this time, the multimedia support system 100 according to the present invention enables the playback of the changed media in a short time using the scene detector 110, the stream mixer 120, and the timestamp adjuster 130.

When a change request is received from the outside, the application unit 50 blocks the input image of the encoder 10 for a predetermined time, mixes the last image before the change and the newly incoming image with a predetermined portion of fade in and out, and delivers it to the encoder 10. As a result, the performance of the motion vector detector of the encoder 10 is maintained.

The scene detection unit 110 is used to minimize the scene change, and P and B frames having a relatively large size are generated according to the heterogeneity of the change in the content and the many motion vector values and the prediction error values generated at the scene change. By not using it, it is possible to complete the transmission of the coded bit stream of the rapidly changed content. Frames obtained from the scene detector from the first keyframe or, if not detected, are transmitted to the last frame. By using the same stream as the bit stream type of the newly used content, the two streams are mixed through the parser and the stream mixer to smoothly transition the image and quickly transfer to the new content. At this time, the time stamp used to synchronize the lip sync with the audio is not matched. Therefore, the time stamp is adjusted during the transmission time as shown in Equation 2 below.

Time_stampm is the timestamp value of the bit string remaining in memory, time_stampn is the time stamp value of the newly created bit string, and the difference between these two values is multiplied by a constant value η. multiply by τ. τ is increased to be greater than or equal to the timestamp of new content after a certain amount of transmission, and if larger or equal, then new data is sent without mixing even if the previous bit string remains.

4 is a flowchart illustrating a multimedia supporting method according to FIGS. 2 and 3. Referring to the figure, the encoder 10 encodes the input multimedia (S101). The stream encoded by the encoder 10 is stored in the memory unit 20 (S103).

The bit control unit 30 controls the output bit rate for the stream stored in the memory unit 20 (S105). At this time, the application unit 50 determines the stream type for the multimedia based on the output bit rate controlled by the bit control unit 30 (S107).

The transmitter 40 controls the size or transmission interval for the data packet whose bit rate is controlled by the bit controller 30 and transmits it to the network through the IP manager 70 (S109).

When the image source change request signal is input from the outside, the scene detector 110 detects a change of the predetermined value or more based on the motion vector value and the prediction error value (S111). When the change is detected by the scene detector 110, the stream mixer 120 mixes the stream remaining in the memory unit 20 with a new stream to smoothly transition the image (S113).

In addition, when a change is detected by the scene detector 110, the timestamp adjuster 130 based on the timestamp of the bit string remaining in the memory 20 and the timestamp of the newly created bit string based on the time during the transmission time. Adjust the stamp (S115). At this time, the adjustment of the time stamp may be made as in Equation 2.

As a result, it is possible to actively cope with network transmission in free space by changing TCP or UDP network according to the network condition, and it is faster than changing the compression rate of the existing encoder by controlling the size of the bit string using the parser. It is possible to vary the bitstream and to determine the effective packet size.

Although the preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the present invention is not limited to the specific embodiments of the present invention without departing from the spirit of the present invention as claimed in the claims. Anyone skilled in the art can make various modifications, as well as such modifications are within the scope of the claims.

1 is a diagram schematically illustrating a general multimedia support system.

2 is a diagram schematically illustrating a first embodiment of a multimedia support system according to the present invention.

3 is a diagram schematically illustrating a second embodiment of a multimedia support system according to the present invention.

4 is a flowchart illustrating a multimedia support method by the multimedia support system of FIGS. 2 and 3.

Description of the Related Art

10: encoder 20: memory unit

30: bit control unit 40: transmission unit

50: application unit 60: parser

70: status monitoring unit 80: IP management unit

110: scene detection unit 120: stream mixer

130: timestamp adjustment unit

Claims (18)

An encoder for encoding the input media; A memory unit for temporarily storing a stream output from the encoder; A bit control unit controlling an output bit rate of the stream stored in the memory unit; And And a transmitter for controlling and transmitting a size or a transmission interval for the data packet controlled by the bit controller. The method of claim 1, And an application unit which determines a stream type for the media based on an output bit rate controlled by the bit control unit. 3. The method of claim 2, And a parser for controlling the size of the bit string controlled by the bit control unit to be used by the application unit. The method of claim 3, wherein In the case of a key frame, the parser adjusts the length of the bit string using an identifier indicating the end of a block used by the encoder. The method of claim 3, wherein In the case of an interactive prediction frame, the parser uses a special bit string including a copy mode. The method of claim 3, wherein In the case of a predictive frame, the parser uses a key frame and a bidirectional predictive frame in a mixed manner. The method according to any one of claims 1 to 6, And the bit controller controls the output bit rate based on the network state in the transport layer and the network state in the IP layer. The method of claim 7, wherein The multimedia support system, characterized in that the control of the output bit rate by the bit control unit can be expressed as follows:

Here, ω ₁ ω ₂ is a weighting factor, Coeff _tr is a network state related expression coefficient at the transport layer, and Coeff _ip is a network state related expression coefficient at the IP layer. The method according to any one of claims 1 to 6, A scene detector for detecting a change of a predetermined value or more based on a motion vector value and a prediction error value when an image source change request signal is input from the outside; And And a stream mixer for smoothly transitioning an image by mixing the stream remaining in the memory unit with a new stream when a change is detected by the scene detector. The method of claim 9, If the change is detected by the scene detection unit, further comprising a time stamp adjusting unit for adjusting the time stamp during the transmission time based on the time stamp of the bit string remaining in the memory unit and the time stamp of the newly created bit string Multimedia support system. A scene detector for detecting a change of a predetermined value or more based on a motion vector value and a prediction error value when an image source change request signal is input from the outside; And And a stream mixer for smoothly transitioning an image by mixing the stream remaining in the memory unit with a new stream when a change is detected by the scene detector. The method of claim 11, If the change is detected by the scene detection unit, further comprising a time stamp adjustment unit for adjusting the time stamp during the transmission time based on the time stamp of the bit string remaining in the memory unit and the time stamp of the newly created bit string Multimedia support system. Encoding the input media; Temporarily storing the stream output from the encoding step; Controlling an output bit rate for the stored stream; And And controlling the size or transmission interval for the data packet whose bit rate is controlled by the bit rate control step. The method of claim 13, And determining the stream type for the media based on the output bit rate controlled by the bit rate control step. The method according to claim 13 or 14, The bit rate control step of controlling the output bit rate based on the network state in the transport layer and the network state in the IP layer. The method of claim 15, The multimedia support method, characterized in that the control of the output bit rate by the bit rate control step can be represented as follows:

Here, ω ₁ ω ₂ is a weighting factor, Coeff _tr is a network state related expression coefficient at the transport layer, and Coeff _ip is a network state related expression coefficient at the IP layer. The method according to any one of claims 13 to 16, Detecting a change of a predetermined value or more based on a motion vector value and a prediction error value when an image source change request signal is input from the outside; And And if the change is detected by the sensing step, smoothly shifting the image by mixing the stream remaining in the memory unit with a new stream. The method of claim 17, And if the change is detected by the detecting step, adjusting the time stamp during the transmission time based on the time stamp of the bit string remaining in the memory unit and the time stamp of the newly created bit string. Way.

Images