KR20100061265A - Apparatus for mixing video stream and method thereof - Google Patents

Abstract

PURPOSE: An apparatus for mixing a video stream and a method thereof are provided to guarantee real time characteristic in the mixture of a plurality of video streams by mixing compacted video streams as one of compacted video of the desired screen configuration in the compressed domain. CONSTITUTION: An input buffer unit(10) receives the input of a plurality of compacted video streams. A stream analyzing unit(20) analyzes the bit string of the inputted video stream. A stream mixing unit(30) selectively varies slider header area of a plurality of compacted video stream bit strings according to the screen configuration based on the analyzed result. An output buffer unit(40) outputs the mixed compacted video stream according to the slice header zone change.

Description

Apparatus for mixing video stream and method

One aspect of the present invention relates to an image processing technique, and more particularly, to a video signal mixing technique.

This study was derived from the research conducted as part of the experimental project for the development of optical subscriber network (FTTH) services by the Ministry of Knowledge Economy and the Ministry of Information and Communication Research.

Digital video data can be used in a variety of applications including video conferencing, high-definition television, video-on-demand (VOD) receivers, personal computers that support moving picture experts group (MPEG) video, game consoles, terrestrial digital broadcast receivers, digital satellite broadcast receivers, and cable television (CATV). Used in the field However, the digital image data is compressed by an efficient image compression method, rather than being used as it is because the amount of data greatly increases in the process of digitizing the characteristics of the image itself and the analog signal.

Video compression techniques include compression standards such as MPEG and H.26x. H.264 / AVC (Advanced Video Coding) is a video compression technology standardized by ITU-T Video Coding Experts Group (VCEG) and Joint Video Team (JVT) of ISO / IEC MPEG. The H.264 / AVC standard shows excellent rate-distortion performance while maintaining a block-based coding scheme like other conventional video compression standards.

According to an aspect, a video stream mixing apparatus and a method for mixing compressed video streams into a single compressed video stream having a user desired screen configuration in a compression domain are proposed.

In one embodiment, a video stream mixing apparatus includes an input buffer unit for receiving a plurality of compressed video streams, a stream analyzer for analyzing a bit string of an input video stream, and a plurality of final screens based on an analysis result. A stream mixing unit for selectively changing the slice header region of the compressed video stream bit string according to the screen configuration, and an output buffer unit for outputting the compressed compressed video stream according to the slice header region change.

Meanwhile, a video stream mixing method according to another aspect includes receiving a plurality of compressed video streams, analyzing a bit string of the input video stream, and configuring a plurality of compressed video streams to form a final mixed screen based on the analysis result. Selectively changing the slice header area of the bit string according to the screen configuration, and outputting a mixed compressed video stream according to the slice header area change.

According to an embodiment of the present invention, the compressed video streams may be mixed into a single compressed video stream having a screen configuration desired by the user in the compression domain without reconstructing the pixel region. Therefore, when mixing a plurality of video streams, it is possible to ensure real-time and to prevent image degradation.

Hereinafter, with reference to the accompanying drawings will be described embodiments of the present invention; In the following description of the present invention, if it is determined that detailed descriptions of related well-known functions or configurations may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted. In addition, terms to be described below are terms defined in consideration of functions in the present invention, which may vary according to intention or custom of a user or an operator. Therefore, the definition should be made based on the contents throughout the specification.

1A and 1B are exemplary views illustrating a service system to which the present invention is applied.

1A is an exemplary diagram illustrating an Internet-based participatory broadcast service system according to an embodiment of the present invention. Referring to FIG. 1A, a video stream mixing apparatus according to an embodiment may be applied to an Internet-based participant broadcasting service system. In this case, the broadcast service may be a participatory IPTV broadcast service.

The Internet-based participant broadcasting service system according to an exemplary embodiment includes a broadcast service providing apparatus 100 for providing a broadcast service, a viewer terminal 120 for watching a broadcast service, and a participant terminal 110 for simultaneously watching a broadcast service and participating in a broadcast. It consists of

At this time, according to an embodiment, the broadcast service providing apparatus 100 receives a compressed video stream from the participant terminal 110 when the viewer wants to participate in a service (for example, a program) that is currently being broadcast through the Internet. . Then, the video stream received from the participant terminal 110 is mixed in a specific area of the broadcast video stream and retransmitted to the participant terminal 110 in real time.

The media server of the broadcast service providing system mixes the compressed video stream transmitted from the participant terminal in the broadcast and the compressed video stream broadcasted by the broadcast program providing apparatus to the screen for composing the broadcast program, respectively, and then converts it into a video stream for transmission. It can be compressed and transmitted to the viewer terminal and the participant terminal.

However, in the video stream mixing process, the media server decodes the compressed video streams by the decoder in the pixel unit, mixes the pixel data, and compresses the compressed video streams again through an encoder having lossy coding characteristics. As a result, the screen quality may deteriorate. In addition, as the process of decoding the plurality of video streams pixel by pixel and the process of compressing the mixed video data through the media server is performed, the time required for simultaneously processing a plurality of video streams of high quality may be delayed. Therefore, it is difficult to produce a broadcast video stream requiring real time.

However, according to an embodiment of the present invention, the broadcast service providing apparatus 100, not the media server, mixes a plurality of compressed video streams. In addition, the broadcast service providing apparatus 100 mixes the compressed video streams in the compressed domain without decoding the compressed video streams to the pixel domain or the DCT (Discreate Cosine Transform) domain. Therefore, video streams can be mixed in real time without deterioration of image quality.

1B is an exemplary diagram illustrating a multi-party video conferencing service system according to another embodiment of the present invention. Referring to FIG. 1B, a video stream mixing apparatus according to an embodiment may be applied to a multiparty video conferencing service system. The multi-party video conferencing service system includes a conference service providing apparatus 130 for providing a multi-party conference broadcast and a participant terminal 140 participating in a conference.

At this time, according to an embodiment, the multi-party video conferencing service providing apparatus 130 receives a compressed video stream from the participant terminal 140 participating in the current video conferencing service, and mixes the received video stream in a specific area in real time. It may be retransmitted to the participant terminal 140.

The MCU (Multipoint Control Unit) of the multi-party video conferencing system may perform a function similar to the media server described above with reference to FIG. 1A. However, in this case, too, there is a problem of time delay and quality deterioration in processing multiple video streams simultaneously.

However, according to an embodiment of the present invention, the video streams may be mixed in real time without deterioration of image quality by mixing a plurality of compressed video streams in the compression domain without performing decoding to the pixel domain or the DCT domain.

Meanwhile, although the broadcast service providing system and the multi-party video conferencing system are shown in FIGS. 1A and 1B, application examples of the video stream mixing apparatus according to an embodiment are not limited to the above-described embodiment. For example, the present invention can be applied to video synthesis and storage using a plurality of compressed video data, a multi-channel DVR system, and the like.

On the other hand, the video stream mixing apparatus according to an embodiment of the present invention may use H.264 / AVC compression technology. H.264 / AVC is a type of H.32x video standard proposed by ITU-T and is a digital video codec standard with very high data compression rate.

2 is a block diagram of a video stream mixing apparatus 1 according to an embodiment of the present invention. Referring to FIG. 2, the video stream mixing apparatus 1 according to an embodiment includes an input buffer unit 10, a stream analyzing unit 20, a stream mixing unit 30, and an output buffer unit 40.

The input buffer unit 10 receives a plurality of compressed video streams. Then, a bit string of a plurality of input video streams is stored. In this case, the input buffer unit 10 may receive a plurality of compressed video streams from a viewer terminal or a conference participant terminal in a multi-party video conference through an Internet-based participant broadcasting service. The bit sequence of the buffer having the largest buffer fullness among the bit sequences not yet mixed may be selectively provided to the stream analyzer 20.

The stream analyzer 20 analyzes a bit string of a video stream input through the input buffer unit 10. The stream analyzing unit 20 is a bit string of the video stream through identification information indicating the type of the NAL unit for a bit string input from a network abstraction layer unit (hereinafter, referred to as NAL unit) among the input video streams. Can be recognized. The NAL unit includes a NAL header, a slice header area, and a slice data area. In this case, the NAL header includes nal-unit-type field information, which is identification information indicating the type of the NAL unit.

The stream analyzer 20 searches for a NAL header from a start prefix code consisting of 4 bytes or 3 bytes in a bit string input in units of NAL, and identifies the information included in the retrieved NAL header. The bit string identification information identified through the (nal_unit_type) field information is provided to the stream mixing unit 30 to be described later.

In addition, the stream analyzer 20 provides a slice header region and a slice data region for the bit string of the plurality of video streams to the stream mixer 30.

If the analyzed video stream is a basic bit string received from a terminal that is the subject of screen mixing, the stream analyzer 20 starts a start prefix code of the stream corresponding to the basic bit string and a NAL header area NAL. A header, a sequence parameter set (SPS) corresponding to the sequence processing information, and a picture parameter set (PPS) corresponding to the picture processing information may be provided to the stream mixer 30. The start code information, the NAL header area, the SPS, and the PPS of the remaining video streams are not provided to the stream mixer 30.

At this time, the basic bit string is a video stream provided from a broadcast service providing apparatus through the Internet-based participant broadcasting service shown in FIG. 1A or compressed video input from a terminal of a participant who has a voice in a conference participant in a multi-party video conference shown in FIG. 1B. It may be a stream.

Meanwhile, the stream mixer 30 selectively changes slice header regions of the plurality of compressed video stream bit streams that will form the final mixed screen based on the analysis result of the stream analyzer 20 according to the screen configuration.

The output buffer unit 40 outputs the compressed video stream mixed according to the change of the slice header region through the stream mixing unit 30. In this case, the output buffer unit 40 selectively receives a bit string so that one slice bit string mixed by the stream mixer 30 may constitute one frame, and outputs the final mixed bit string.

3 is a detailed block diagram of the stream mixing unit 30 of FIG. 2 according to an embodiment of the present invention. Referring to FIG. 3, the stream mixer 30 includes a bit string determination unit 300, a sequence parameter set converter 310, and a slice header converter 320, and further includes a slice data aligner 330. It may further include.

The bit string determination unit 300 determines a sequence parameter set (SPS) corresponding to the sequence processing information by using the bit string identification information analyzed by the stream analyzer 20 of FIG. 2.

The sequence parameter set converter 310 converts the resolution information according to the screen to mix the video streams with respect to the sequence parameter set (SPS) determined by the bit string determination unit 300. For example, the sequence parameter set converter 310 changes and outputs a pic_width_in_mbs_minus1 field value indicating a horizontal size and a pic_height_in_map_units_minus1 field value indicating a vertical size of a final screen according to the resolution size of a screen to configure a final mixed screen.

The slice header converter 320 changes a slice header area of each slice constituting the screen to be mixed. In this case, the slice header converter 320 may change address information corresponding to the start macro block among the macroblocks included in the slice header area of each slice. That is, the slice header converting unit 320 converts only first_mb_in_slice field information indicating the start macroblock address of the slice header constituting one slice, and does not set the remaining macroblock addresses constituting the slice.

The H.264 / AVC standard syntax has a field indicating only the first macroblock address for each slice constituting one screen, which allows flexible definition of the screen configuration when mixing multiple video streams. In addition, according to an embodiment of the present invention, it is not necessary to perform the process of encoding the inverse VLC for extracting the macroblock address information and the difference value according to the increase of the macroblock address into VLC.

On the other hand, the slice data sorter 330 sorts the last byte of the slice data area according to the change of the slice header area of the slice. A detailed description of the alignment process of the slice data alignment unit 330 will be given later with reference to FIG. 5.

4 is a reference diagram for explaining macroblock address reallocation in a stream mixing process according to an embodiment of the present invention. 4 illustrates a case in which the video stream mixing apparatus mixes three video streams for one screen. The thick dotted line 420 shown in FIG. 4 arbitrarily shows the boundaries of the respective video streams, and the solid arrow line 430 shows the increase direction of the macroblock address for changing the value of the first_mb_in_slice field of the mixed slice header. will be.

In this case, the slice header converter 320 does not convert the first_mb_in_slice field value of the first slice header on the left side from the slice headers of the input bit streams constituting the final mixed screen (410), and the first_mb_in_slice field value of the remaining slice headers is displayed on the screen. Convert according to the configuration (400).

5 is a reference diagram illustrating a slice bit string in a slice data byte alignment process according to an embodiment of the present invention.

Referring to FIG. 5, the video stream mixing apparatus aligns the last byte of the slice data area according to the change of the slice header area of the slice.

In the H.264 / AVC syntax standard, a bit string constituting a slice in one screen is arranged in byte units including a slice header and slice data. Accordingly, according to an embodiment of the present invention, the video stream mixing apparatus rearranges the last byte of slice data constituting one slice as the macroblock address changes in value.

Reference numeral 501 of FIG. 5 shows the size of a slice bit string including an input slice header and slice data, reference numeral 502 shows the size of the input slice bit string, and reference numeral 503 denotes the last byte of the input bit string. Size is shown.

On the other hand, reference numeral 504 of FIG. 5 shows the length of the macroblock address changed to increase the size of the reference numeral 502, and the last byte of the slice data is increased by the size of the reference numeral 505 due to the bit increased by the reference numeral 504. Pushed. Therefore, the video stream mixer calculates the number of bits increased by 506 and inserts an arbitrary '0' value for byte alignment to byte-align the last bits. Reference numeral 507 shows the size of a bit string constituting a slice including the last transformed slice header and slice data.

6 is a flowchart illustrating a video stream mixing method according to an embodiment of the present invention.

Referring to FIG. 6, the video stream mixing apparatus according to an embodiment of the present invention receives a plurality of compressed video streams (S600). Subsequently, the video stream mixing apparatus analyzes a bit string of the input video stream (S610).

Then, the video stream mixing apparatus changes the slice header region of the identified bit string according to the analysis (S620) and mixes the plurality of video streams to form the final mixed screen (S630). Here, the video stream mixing apparatus determines the sequence parameter set corresponding to the sequence processing information by using the identification information of the bit string recognized through the stream analysis. Then, the resolution information is converted according to the screen for mixing the video stream with respect to the determined sequence parameter set. Next, the slice header area of each slice constituting the screen to be mixed is changed. The changed slice header area may be address information corresponding to a start macroblock among macroblocks.

Subsequently, the video stream mixing apparatus outputs the mixed video stream (S640).

The embodiments of the present invention have been described above. Those skilled in the art will appreciate that the present invention can be implemented in a modified form without departing from the essential features of the present invention. Therefore, the disclosed embodiments should be considered in descriptive sense only and not for purposes of limitation. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the scope will be construed as being included in the present invention.

1A and 1B are exemplary views illustrating a service system to which the present invention is applied.

2 is a block diagram of a video stream mixing device according to an embodiment of the present invention;

3 is a detailed configuration diagram of the stream mixing unit of FIG. 2 according to an embodiment of the present invention;

4 is a reference diagram for explaining macroblock address reallocation in a stream mixing process according to an embodiment of the present invention;

5 is a reference diagram illustrating a slice bit string in a slice data byte alignment process according to an embodiment of the present invention;

6 is a flowchart illustrating a video stream mixing method according to an embodiment of the present invention.

<Description of the symbols for the main parts of the drawings>

1: video stream mixing device 10: input buffer unit

20: stream analysis unit 30: stream mixing unit

40: output buffer unit 300: bit string determination unit

310: sequence parameter set converter 320: slice header converter

330: slice data sorting unit

Claims (12)

An input buffer unit configured to receive a plurality of compressed video streams; A stream analyzer for analyzing a bit string of the input video stream; A stream mixing unit for selectively changing slice header regions of a plurality of compressed video stream bit streams to form a final mixed screen based on the analysis result according to the screen configuration; And And an output buffer unit for outputting the compressed compressed video stream according to the change of the slice header region. The method of claim 1, wherein the stream mixing unit, A bit string determination unit for determining a sequence parameter set corresponding to sequence processing information by using identification information of the bit string analyzed by the stream analyzer; A sequence parameter set converter for converting resolution information according to a screen for mixing the video stream with respect to the determined sequence parameter set; And And a slice header converter configured to change a slice header area of each slice constituting the screen to be mixed according to the screen configuration. The method of claim 2, wherein the slice header converter, And changing the address information corresponding to the start macroblock among the macroblocks included in the slice header region of each slice. The method of claim 2, And a slice data aligning unit to align the last byte of the slice data region according to a change of the slice header region of the slice. The method of claim 1, wherein the stream analysis unit, And a bit stream of the video stream through identification information indicating a type of the network abstract layer unit for a bit stream input from a network abstract layer unit among the input video streams. The method of claim 1, wherein the stream analysis unit, Providing a slice header area and a slice data area for bit streams of the plurality of video streams to the stream mixing unit, If the analyzed video stream is a basic bit string received from a terminal that is the subject of screen mixing, the sequence code set and the picture corresponding to the start code information, the network abstract layer header area, and the sequence processing information of the stream corresponding to the basic bit string. And a picture parameter set corresponding to processing information is provided to the stream mixing unit. The method of claim 6, The basic bit string is a video stream received from a broadcast service providing apparatus for providing an Internet-based participant broadcasting service or a compressed video stream received from a terminal of a participant having a voice in a conference participant in a multi-conference video conference service. Stream mixer. The method of claim 1, The input buffer unit receives the compressed plurality of video streams from a viewer terminal using an Internet-based participant broadcasting service, And the output buffer unit outputs the mixed compressed video stream to the viewer terminal. The method of claim 1, The input buffer unit receives the compressed plurality of video streams from a conference participant terminal using a multi-party video conferencing service, And the output buffer unit outputs the mixed compressed video stream to the conference participant terminal. Receiving a plurality of compressed video streams; Analyzing a bit stream of the input video stream; Selectively changing slice header regions of a plurality of compressed video stream bit streams to form a final mixed screen based on the analysis result according to a screen configuration; And And outputting a mixed compressed video stream according to the slice header region change. The method of claim 10, wherein mixing the plurality of video streams comprises: Determining a sequence parameter set corresponding to sequence processing information by using identification information of a bit string recognized by the stream analyzer; Converting resolution information according to a screen for mixing the video stream with respect to the determined sequence parameter set; And And changing a slice header area of each slice constituting the screen to be mixed. The method of claim 11, wherein changing the slice header area of each slice comprises: And changing the address information corresponding to the start macro block among the macro blocks included in the slice header area of each slice.

Images