KR101074073B1 - A buffer structure for storing multimedia data, and a buffering method - Google Patents

Abstract

The present invention relates to a buffer structure and a buffering method for storing multimedia data, which are temporarily stored in a memory before decoding an encoded picture and audio stream in an application program when exchanging pictures and audio through a network. A multimedia data storage buffer according to the present invention comprises: a multimedia data storage buffer for temporarily storing image or audio data, comprising: a data list comprising nodes for storing the image or audio data stream; A main list consisting of nodes having information about the data list; A main list management module for adding, deleting, or searching a predetermined node to the main list; And a data list management module for adding, deleting, or searching a predetermined node to the data list. According to the present invention, by efficiently implementing a buffer for temporarily storing the encoded picture and audio stream in memory before decoding, it is possible to increase the memory efficiency of the data received from any port, and to process a large number of data quickly. Image and audio can be easily synchronized.

Buffer, main list, data list, multimedia data, video conferencing

Description

A buffer structure for storing multimedia data, and a buffering method}

1 is a block diagram of a video conferencing system having a data storage buffer according to an embodiment of the present invention.

2 is a diagram illustrating a data structure stored in a data storage buffer according to an embodiment of the present invention.

3 is a diagram illustrating a structure of a data storage buffer according to an embodiment of the present invention.

4 is a flowchart illustrating a data transmission / reception method using a data storage buffer according to an embodiment of the present invention.

5 is an operation flowchart of a server provided with a data storage buffer according to an embodiment of the present invention.

6 is a diagram illustrating the operation of the main list of the data storage buffer according to the embodiment of the present invention.

7 is a diagram illustrating an operation of a data list of a data storage buffer according to an embodiment of the present invention.

The present invention relates to a buffer structure and a buffering method for storing multimedia data. More particularly, when an image and a voice are exchanged through a network, the multimedia is temporarily stored in a memory before the encoded image and audio stream is decoded by an application program. The present invention relates to a buffer structure and a buffering method for data storage.

Video conferencing system is a system that allows people who are far apart from each other to conduct meetings through video and sound reflected on the monitor screen installed in their room. An electronic blackboard, a facsimile, or the like, in which one piece of information is displayed on the other party as it is, may be mounted.

In such videoconferencing, a plurality of video and audio data are exchanged between conference participant terminals, each video and audio data encoded and transmitted at the transmitter of the conference participant terminal, decoded at the receiver and displayed to the user.

When the image and audio data received by the receiver are input to a designated decoder for data interpretation without using a buffer in a network where no delay occurs, on the other hand, when the data is received or delayed due to the order of the data being changed according to the state of the network. The receiver using the network in which the data is generated has a buffer to align and store the image and audio data for a predetermined time. When synchronizing for the data alignment, the user may display data in which the generated time is included in a specific time region with reference to the time when the image and audio data are generated. In addition, when video and audio data are processed by a server in a videoconferencing system, the videoconferencing server can use an efficient data processing buffer to process a large number of data.

For example, when a user transmitter encodes and transmits image data, each frame is transmitted in one data packet at a rate of 30 frames / second, and when buffered for 3 seconds, 90000 (30) is required to process 1000 image streams. Frame / sec x 3 sec x 1000), and 30000 inputs and outputs per second occur respectively. Processing 10 picture streams produces 900 nodes and 300 inputs and outputs per second under the same conditions. At this time, the size of each image data is not constant.

On the other hand, in application programs where frequent data input and output occur and the data size is not constant, a linked list, a skip list, or a hybrid list is used for data buffering. do. For example, when using a network without delay or processing one video and audio stream, one buffer is used, and the configuration of the buffer may use a linked list, a skip list, or a hybrid list. In addition, when processing a plurality of image streams, a plurality of buffers are used, and the buffer structure utilizes a data structure having a short time for adding, deleting and extracting nodes.

Specifically, the linked list must manage the start and end of the list, and when the number of nodes increases, the time for sorting and adding nodes or extracting nodes increases due to an increase in search time. For example, there is no network delay, or used in applications that process the simplest video and audio. In addition, in an application program that stores a plurality of image and audio data in one linked list, when an output request is provided, the front node is provided, and when the image and audio are synchronized, the delay of specific image data becomes large and the data is processed. If the first node is not deleted from the buffer, another node may not be output and the entire data processing may be delayed.

In addition, the skip list and the hybrid list have a higher node processing efficiency than a binary tree and may process a large amount of data. In this case, in order to process a plurality of image and audio data, a plurality of lists may be created and a module for managing the same may be used.

On the other hand, as a prior art, Korean Patent Application No. 2001-64172 (filed October 18, 2001) discloses the invention entitled "Buffer Structure and Buffering Method for Multimedia Data". Circular List) provides a method of configuring a buffer for storing images and audio.

Specifically, the prior invention is a structure in which a part of the prototype list is implemented as a skip list, the necessary memory is pre-allocated, and the data is sorted by using the time coefficient of each data packet, thereby making the input and output of data more efficient. In addition, in the correlation between data processing efficiency and memory usage, the memory usage efficiency is reduced to increase data processing efficiency.                         

In order to process multiple streams in a multimedia conference using this structure, a dedicated buffer may be allocated to each stream to process streams, or time streams of multiple streams may be synchronized to store all streams in one buffer for processing. have.

However, in the prior invention, when a buffer is allocated to each stream, the memory for the buffer is allocated in advance, and thus the use efficiency of the memory is lowered, and when the buffer size is larger than the expected size due to an increase in delay due to the change of network environment. There is a problem with updating the buffer. In addition, in the present invention, when a plurality of video and audio data are received through a single port, there is a need for a method of finding a buffer allocated to each stream to store the received stream. In case of storing, the length of the list increases linearly, which increases the input and output time.

An object of the present invention for solving the above problems is to provide a buffer structure and buffering method for storing multimedia data that can increase the memory efficiency of data received from any port by utilizing a buffer divided into a main list and a data list It is for.

Another object of the present invention is to provide a buffer structure and a buffering method for storing multimedia data which are easy to process a plurality of data quickly and to synchronize images and voices.

As a means for achieving the above object, a buffer for storing multimedia data according to the present invention,

In the multimedia data storage buffer for temporarily storing the image or audio data,

A data list consisting of nodes in which the video or audio data stream is stored;

A main list consisting of nodes having information about the data list;

A main list management module for adding, deleting, or searching a predetermined node to the main list; And

A data list management module for adding, deleting, or searching a predetermined node to the data list

There is a feature configured to include.

Here, the node of the main list is characterized by consisting of header information of the data list.

Here, the main list management module extracts a location of the data list to store the stream from the main list by referring to the stream ID of the received data packet.

The data list management module may input data to the data list in ascending order using a time stamp (TS) of a received data packet.                     

Herein, when the connection between the user client and the server is terminated, the data list management module deletes the data list related to the terminated user client.

Here, the main list or data list may be formed in any one of a linked list, a skip list, or a hybrid list method.

On the other hand, as another means for achieving the above object, the buffering method of the buffer for storing multimedia data according to the present invention,

In the buffering method of the buffer for storing multimedia data between the user client and the server,

a) dividing a buffer in which data is temporarily stored into a main list comprising a data list in which image and audio data streams are stored and a node having information on the data list;

b) when the user is connected to a server, the server generating and assigning at least one stream ID to the user;

c) searching for a location to store the user's data stream with reference to the stream ID in the main list; And

d) storing the received data stream in the retrieved data list

There is a feature consisting of.

In the step c), if the data list retrieval fails, a new data list is generated and added to the main list.                     

In step d), the data stream may be stored in the data list in an ascending order using a time stamp (TS) of a received data packet.

Here, when the connection between the user client and the server is terminated, the method may further include deleting a data list related to the terminated user client.

Here, the stream ID of step b) is one stream ID for distinguishing the user for each data stream type.

Therefore, according to the present invention, by efficiently implementing a buffer for temporarily storing the encoded image and audio streams in the memory before decoding, it is possible to increase the memory efficiency of the data received from any port, and to quickly It can process and synchronize the image and sound easily.

Hereinafter, a buffer structure and a buffering method for storing multimedia data according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

In the buffer structure for storing multimedia data according to an embodiment of the present invention, data received from an application program processing a plurality of data or a network that may cause delays may be classified and sorted based on an image, a voice, a transmitter, and the like. Can be used for storage.

Therefore, when using a buffer for storing multimedia data according to an embodiment of the present invention, it is necessary to process delays for each stream and to synchronize multimedia data such as video and audio without affecting the entire application program. do. In addition, in order to synchronize multimedia data such as a plurality of images and voices, data generated at a similar time should be displayed to the user with reference to the time when the stored data are generated.

1 is a configuration diagram of a video conferencing system having a data storage buffer according to an exemplary embodiment of the present invention, and illustrates an environment including a transmitter 200, a receiver 300, a server 100, and a network 400.

Referring to FIG. 1, the transmitter 200 includes a microphone 210 for receiving voice, a camera 220 for receiving an image, a capture module 230 for storing voice and image data, and a time coefficient for the captured data. A TS / ID setting module 240 for specifying a Time Stamp (TS) and a Stream Identification Number (ID), an encoding module 250 for compressing voice and image data, and data in units of data packets to a network. Outgoing transmission module 260.

The receiver 300 receives a data packet from a network, a receiving module 310, a TS / ID analysis module 320 for processing a time coefficient and a stream ID attached to the received data packet, and a buffer for temporarily storing the received data. Module 330, a decoding module 340 to decompress compressed data, a playback module 350 to play back voice and image data, a speaker 360 as a physical device to play back voice data, and a physical device to play back an image stream. It consists of a screen 370.                     

The server 100 may include a server receiving module 110, a server transmitting module 120, a server TS / ID analysis module 130, a buffer module 140, and an ID management module 150. Here, the server receiving module 110, the server transmitting module 120, the server TS / ID analysis module 130 and the buffer module 140, the image and audio data is distributed to the receiver via the server, the server The modules 100 are provided for data packet processing.

Here, the server receiving module 110 receives a data packet from a sender, the server sending module 120 transmits a data packet to a receiver, and the server TS / ID analysis module 130 transmits a data packet related to transmission and reception. Extract the time coefficient and stream ID of the data packet is stored in the buffer module 140. In addition, the ID management module 150 generates and issues IDs of video and audio streams to be transmitted by the transmitter 200 when a request is made by the transmitter 200. At this time, the stream ID is a positive integer. If the transmitter 200 transmits multiple streams, multiple stream IDs may be required. In this case, if the stream ID distinguishes users from each stream type, several streams may be divided into one stream ID.

In addition, the network 400 is any network that transmits voice and image data, and is, for example, an Internet Protocol (IP), a Transmission Control Protocol (TCP), a User Datagram Protocol (UDP), or a Real Time Protocol (RTP). May be any network capable of transmitting voice or image data.

Meanwhile, the buffer module 140 adds, deletes, and retrieves a main list 142 having data list information, a data list 144 in which image and audio data are actually stored, and a node in the main list 142. A main list management module 141 having a function, and a data list management module 143 having a function of adding, deleting, and searching nodes to the data list 144. That is, the buffer for storing multimedia data according to an embodiment of the present invention provides a space for storing each image and audio stream, and also provides an image and audio for providing a high input / output speed for storing a plurality of images and audio. It consists of a data list for storing data and a main list having information about the data list.

Meanwhile, when each image and audio data is captured by the TS / ID setting module 240 of the transmitter 200, the time coefficient of the captured data is read by reading the time coefficient from a time counter managed by the transmitter 200. Set. At this time, the time coefficient for the captured voice data packet indicates the time spoken by the user, and the time coefficient for the image data packet indicates the time when the user's image was taken.

Since this time coefficient is information managed by the transmitter 200 irrespective of standard time, it is not synchronized with the time coefficient managed by the receiver 300. At this time, the captured data is compressed in the encoding module 250.

In addition, the transmitting module 260 configures the compressed data in units of one data packet or in units of several data packets, sets time count information designated by the TS / ID setting module 240 in each data packet. Then, the transmitter 200 requests the server 100 to set the issued stream ID and transmits it to the network 400.

In addition, the receiver 300, the receiving module 310 receives the data packet from the network 400. At this time, the TS / ID analysis module 320 of the receiver 300 extracts the stream ID and time coefficient information included in the received data packet, transfers it to the buffer module 330, and the buffer module 330. Stores data in ascending order of time coefficients for each stream ID.

2 is a diagram illustrating a data structure stored in a data storage buffer according to an exemplary embodiment of the present invention, which shows a format of a data packet transmitted from a network, and includes a plurality of headers 510, 520, and ID 530. , TS 540 and data 550.

Referring to FIG. 2, a data structure stored in a data storage buffer according to an embodiment of the present invention includes various headers 510 and 520 necessary for a network to transmit data packets, and includes actual image and audio data ( Time coefficient (TS) information 540 is preceded by 550.

Here, the headers 510 and 520 necessary for transmitting data packets in the network may include an IP header, a TCP header, a UDP header, an RTP header, and the like. Although the time coefficient information may be included in these headers, the time coefficients included in these headers are time coefficients indicating the time when the header is added to the data packet, and thus it is not accurate to use them. Therefore, the time coefficient information is preferably stored separately from the header.

3 is a diagram illustrating a structure of a data storage buffer according to an embodiment of the present invention.                     

Referring to FIG. 3, a data storage buffer according to an embodiment of the present invention includes a main list 142 composed of nodes having information about a data list 144, and a plurality of nodes configured to store actual image and audio streams. It consists of a data list 144.

The main list 142 is processed by the main list management module 141 shown in FIG. The main list management module 141 extracts the location of the data list 144 to store the stream from the main list 142 by referring to the stream ID of the received data packet.

In addition, the data list management module 143 shown in FIG. 1 inputs the data list 144 in ascending order using the time coefficient of the received data packet, and the transmitter 200 and the server 100 shown in FIG. ) Is terminated, the data list 144 associated with the terminated transmitter 200 is deleted.

In addition, in an embodiment of the present invention, to increase the use efficiency of the memory, the memory is dynamically allocated and stored when adding data. If data processing efficiency is more important than memory usage efficiency, the required memory may be allocated in advance, but the dynamic memory may be used in a relatively stable network environment.

In addition, the main list 142 and the data list 144 may use a linked list, a skip list, or a hybrid list.

4 is an operation flowchart of a data transmission / reception method using a data storage buffer according to an embodiment of the present invention.                     

Referring to FIG. 4, first, a user client is connected to a video conferencing server (S401). That is, after the user client makes a connection request to the server and waits for a response from the server, the user client is connected to the server according to the response of the server. Here, the user client refers to a plurality of clients that can participate in the videoconferencing, and since the same data transmission and reception are performed separately, the data transmission / reception method performed between the individual user client and the videoconferencing server will be described.

Thereafter, the user client checks whether a stream ID is received from the server (S402). That is, when the server transmits its information, it is checked whether the user ID has assigned a stream ID to use.

Thereafter, the user stores the allocated stream ID (S403), and uses the multimedia information in the future.

Next, the user determines whether to participate in the video conference (S404), and then, the user participates in the video conference to check whether the video / audio data is captured (S405).

If the user captures the video / audio data, the user client attaches the time coefficient and the stream ID to the data packet to the current conference participant (S406). At this time, if there is no conference participant, the time coefficient and stream ID are not transmitted.

Next, in the state where the user does not capture the video / audio data, the user client checks whether the videoconferencing server has sent the meeting participant information (S407), and if the user client receives the meeting participant information, In one case, the meeting participant information is stored or updated (S408), and then referred to when the user client transmits video and audio data.

Subsequently, the user client checks whether video and audio data has been received from another conference participant (S409). If the user client receives video and audio data, the user client refers to the stream ID of the received data packet according to an embodiment of the present invention. In the main list 142 of the buffer, a data list 144 to store data is searched for (S410).

If retrieval of the data list 144 to store data from the main list of the buffer fails, a new data list 144 is generated (S411), and if the data list to store data from the main list 142 of the buffer, If the search is successful or the new data list is generated, a location to store data in the data list is searched and stored with reference to a time counter (S412).

Next, the user client determines whether to join / end the video conference (S413). If the user client continues to participate in the video conference, the process returns to step S404 and the above-described steps S404 to S412 are repeated.

5 is an operation flowchart of a server having a data storage buffer according to an exemplary embodiment of the present invention, which will be described as an example of the case where the server is a video conference server conducting video conference.

Referring to FIG. 5, first, the videoconferencing server checks whether there is a connection request from a user client (S501), and if there is the connection request, accepts the connection request and is connected to the user, and streams for a new user. The ID is generated and the stream ID is transmitted to the new user (S502).

In this case, the stream ID may use any positive integer, and the stream ID may be generated using a random number, or may be used while sequentially increasing an integer value in a predetermined range.

If a user transmits multiple streams, multiple stream IDs may be required.

In addition, if the network port or stream processing module is different according to the stream format of the user's image, the presentation material image, the user's voice, etc., the stream ID may distinguish each user by the stream format. In this case, several streams may be divided into one stream ID. Therefore, when not using such a stream format, a plurality of stream IDs should be assigned to the user.

Next, when the user requests to join or terminate the meeting (S503), the server adds or deletes the meeting participant and transmits the participant information or the changed information to the meeting participant (S504).

Next, by checking whether the user wants to terminate the connection with the server (S505), if the user wants to terminate the connection with the server, the server deletes the stream ID used by the user (S506). ).                     

6 is a diagram illustrating an operation of a main list of a data storage buffer according to an exemplary embodiment of the present invention, and illustrates an example in which the main list 142 is configured using a skip list method.

Referring to FIG. 6, for example, when a data packet having a stream ID of 56 is received, it is input between a node having a stream ID of 55 and a node having 67. In this case, the node of the main list 142 may be composed of header information of the data list 144.

7 is a diagram illustrating an operation of a data list of a data storage buffer according to an exemplary embodiment of the present invention, and illustrates an example in which the data list 144 is configured using a skip list method.

Referring to FIG. 7, for example, when data having a stream ID of 33 is received, it is input between a node having a stream ID of 32 and a node having 35. In this case, linked lists, skip lists, and hybrid lists may be used to construct the data list 144.

As a result, embodiments of the present invention can easily adapt to changes in network conditions when users connected to various networks compress and transmit video and audio using hardware having different performances, and receive the received data on a stream-by-stream basis. It provides a structure of a buffer to be stored and a method of using the same.

The buffer structure according to the embodiment of the present invention is applied to an environment in which various computers transmit and receive multimedia data. In the process of transmitting and receiving multimedia data, multimedia data is efficiently stored temporarily in a memory before the application program processes the data. To be displayed as it was before it was compressed.

Therefore, the buffer structure according to the embodiment of the present invention can be used in a server for relaying multimedia data, a personal multimedia data storage program, and a video conference application program. When used in a server processing multimedia data, a plurality of video and audio streams can be classified and distributed by utilizing a data structure having high input / output efficiency of the main list and the data list. In addition, when utilized in a terminal for processing image data, image and audio data are classified, sorted, stored, and outputted to facilitate synchronization of each stream unit such as a lip sync.

While the invention has been shown and described in connection with specific embodiments thereof, it will be appreciated that various modifications and changes can be made without departing from the spirit and scope of the invention as indicated by the claims. Anyone who owns it can easily find out.

According to the present invention, by efficiently implementing a buffer for temporarily storing the encoded picture and audio stream in memory before decoding, it is possible to increase the memory efficiency of the data received from any port, and to process a large number of data quickly. Image and audio can be easily synchronized.

Claims (11)

In the multimedia data storage buffer for temporarily storing the image or audio data, A data list consisting of nodes in which the video or audio data stream is stored; A main list consisting of nodes having information about the data list; A main list management module for adding, deleting, or retrieving a predetermined node to the main list based on the stream ID of the received data packet; And A data list management module for adding, deleting, or searching a predetermined node to the data list A buffer for storing multimedia data comprising a. The method of claim 1, And the node of the main list comprises header information of the data list. delete The method of claim 1, And the data list management module inputs data into the data list in ascending order using a time stamp (TS) of a received data packet. The method of claim 1, And the data list management module deletes the data list related to the terminated user client when the connection between the user client and the server is terminated. The method of claim 1, The main list or the data list is a buffer for multimedia data storage, characterized in that formed in any one of a linked list, skip list, or a hybrid list method. In the buffering method of the buffer for storing multimedia data between the user client and the server, a) dividing a buffer in which data is temporarily stored into a main list comprising a data list in which image and audio data streams are stored and a node having information on the data list; b) connecting the user client and the server to receive the transmitted data stream; c) searching for a location in the main list to store the data stream based on the stream ID of the data stream; And d) storing the received data stream in the retrieved data list The buffering method of the buffer for storing multimedia data comprising a. The method of claim 7, wherein In the step c), if the data list retrieval fails, a new data list is generated and added to the main list. The method of claim 7, wherein And d) storing the data stream in ascending order in the data list using a time stamp (TS) of a received data packet. The method of claim 7, wherein And after the step d), when the connection between the user client and the server is terminated, deleting the data list related to the terminated user client. The method of claim 7, wherein The stream ID is a buffering method of a buffer for storing multimedia data, characterized in that one stream ID for distinguishing the user client for each data stream format.

Images