WO2010094213A1

WO2010094213A1 - Method, apparatus and system for transmitting and receiving multiplex media data stream

Info

Publication number: WO2010094213A1
Application number: PCT/CN2010/070180
Authority: WO
Inventors: 苏红宏
Original assignee: 华为终端有限公司
Priority date: 2009-02-20
Filing date: 2010-01-14
Publication date: 2010-08-26
Also published as: CN101489090B; CN101489090A

Abstract

A method, apparatus and system for transmitting and receiving a multiplex media data stream are provided in the embodiments of the present invention, which relate to the field of network technology, and can solve the problem that the multiplex media stream can not be combined freely and output. The transmission method comprises: grouping the multiplex media data stream to be transmitted; describing said grouped multiplex media data stream; transmitting said grouped multiplex media data stream and the grouping description corresponding to the media data stream. The receiving method comprises: receiving the multiplex media data stream with the grouping description, wherein said grouping description corresponds to said media data stream; outputting the media data stream in groups according to the grouping description of the received multiplex media data stream. The embodiments of the present invention are applied to the multiplex media stream transmission and output in the multimedia communication.

Description

Method, device and system for multi-channel media stream transmission and reception

The present invention relates to the field of network technologies, and in particular, to a method, device and system for transmitting and receiving multiple media streams in multimedia communication. Background technique

In order to further enhance the clinical sense of the conference, in conference TV, it is often necessary to simultaneously transmit video, audio and data of multiple (or groups) of participants in a conference venue. The ITU-T H.239-based standard defines the transmission of multiple media streams in a conference television. Multiple channels of media streams can be transmitted between terminals and terminals and between terminals and MCUs (Multipoint Control Units). Only defines how to send an auxiliary video stream, and H.239 only defines the role of each media stream. This role can be a moving image or slide content, but it does not indicate the relationship between multiple media streams, so it receives After acquiring the multi-channel media data, the terminal needs to be processed by the receiving end before outputting.

FIG. 1 is a schematic diagram of a method for media streaming in a prior art conference television. Two conference TV terminals are placed in the conference site A and the conference site B. Each terminal is used by one (or one) group of participants. Among them, each set of terminals includes a camera, a microphone, a display device, and a speaker. In addition, two sites, eight and B, connect to the network through switches. The terminal 1 of the site A establishes a connection with the terminal 3 of the site B, and the terminal 2 of the site A establishes a connection with the terminal 4 of the site B, so that the terminal 1 of the site A will present the sound image and data collected by the terminal 3 of the site B. The terminal 2 of A will present the sound image and data collected by the terminal 4 of the venue B. Conversely, the terminal 3 of the conference site B will present the audio and video data of the terminal 1 of the conference site A, and the terminal 4 of the conference site B will present the audio image and data of the terminal 2 of the conference site A. Through such a technical solution, the transmission of two channels of video and audio streams at two sites is realized.

There is another method in the prior art for generating a corresponding audio position for a video layout, comprising: generating, by the MCU, a multi-picture video combining a plurality of endpoint images and sound position information corresponding to the multi-screen layout, and according to the speaker status at the receiving end Sending multiple audio streams, multi-picture video and sound position information to the receiving end, and the receiving end uses multiple audio streams to reproduce according to the position information, so that the position of the end point sound heard by the participant and the position of the end point in the multi-picture ( Layout) close. For the participant, the speaker at the far end of the screen is compared to the audio broadcast at the speaker at the near end of the display The audio broadcast at the location may be attenuated or delayed. This kind of program can enhance the clinical sense of the meeting. However, in this scheme, the sound position is determined by the MCU, and is executed by the terminal, which lacks flexibility, and the terminal cannot independently adjust the position of the sound. The prior art cannot realize free combination output of multi-media media streams in multimedia communication, and does not consider transmission and processing of data services. Summary of the invention

Embodiments of the present invention provide a method, apparatus, and system for transmitting and receiving multiple media streams in multimedia communication, which can solve the problem that multiple media streams cannot be freely combined and output in multimedia communication.

Embodiments of the present invention use the following technical solutions:

A method for transmitting a multi-channel media data stream in multimedia communication, comprising:

And grouping the multi-media media data streams to be transmitted; grouping the packetized multi-media media data streams; and transmitting the packetized multi-media media data streams and packet descriptions corresponding to the media data streams.

An embodiment of the present invention further provides a method for receiving and processing a multi-channel media data stream in a multimedia communication, including:

Receiving a multi-media media data stream with a packet description and a packet description corresponding to the media data stream; and outputting the media data stream according to a packet description of the received multi-media media data stream.

An embodiment of the present invention further provides a transmission device for a multi-channel media data stream in multimedia communication, including:

a grouping unit, configured to group the multi-path media data streams to be transmitted; a packet description unit, grouping and describing the grouped multi-media media data streams; and a data transmission unit, transmitting the grouped multi-path media data And a packet description corresponding to the media data stream.

An embodiment of the present invention further provides a receiving and processing apparatus for a multi-channel media data stream in a multimedia communication, including:

a receiving unit, configured to receive a multi-media media data stream with a packet description and a packet description corresponding to the media data stream; and an output unit, configured to: according to the packet description of the received multi-channel media data stream, the packet output The media data stream.

An embodiment of the present invention further provides a conference terminal, including:

a sending device, configured to acquire and group a plurality of media data streams to be transmitted, perform packet group description on the grouped multi-channel media data stream, and transmit the grouped multi-channel media data stream and the media a packet description corresponding to the data stream; and/or a receiving processing device, configured to receive a multi-media media data stream with a packet description and a packet description corresponding to the media data stream, and output the media data according to a packet description of the received multi-channel media data stream flow.

An embodiment of the present invention further provides a system for multi-channel media data transmission in multimedia communication, including:

a data acquisition device, including a video acquisition device, an audio acquisition device, or a data acquisition device, for acquiring a multi-channel media data stream;

a transmitting device, configured to group the multi-channel media data streams acquired by the data acquiring device, perform packet group description on the grouped multi-channel media data streams, and transmit the grouped multi-channel media data streams and a packet description corresponding to the media data stream;

a receiving processing device, configured to receive a multi-media media data stream with a packet description and a packet description corresponding to the media data stream, and output the media data according to a packet description of the received multi-channel media data stream Flow

And a data output device, configured to output the grouped media data stream through a video output device, an audio output device, or a data output device.

The method, device and system for transmitting and receiving multiple media streams in multimedia communication provided by the embodiments of the present invention are capable of grouping multiple media data, and grouping and describing each packet, and performing the grouped multimedia data stream and the packet description. Transmitting; and after receiving the multimedia data stream carrying the packet description and the packet description corresponding to the multimedia data stream, outputting the media data stream according to the packet description, thereby implementing free combination output of the multiple media streams . DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the description of the prior art will be briefly described below. Obviously, the drawings in the following description are only It is a certain embodiment of the present invention, and other drawings can be obtained from those skilled in the art without any inventive labor.

1 is a schematic diagram of a method for multi-channel media streaming in a conference television of the prior art;

2 is a flowchart of a method for transmitting multi-channel media data stream in multimedia communication according to an embodiment of the present invention; FIG. 3 is a flowchart of a method for outputting multi-channel media data stream in multimedia communication according to an embodiment of the present invention; Schematic diagram of a method for transmitting and outputting multiple media data streams in multimedia communication; FIG. 5 is a flowchart of a method according to Embodiment 1 of the present invention; FIG.

6 is a schematic diagram of a multi-channel media stream grouping at an input end according to an embodiment of the present invention; FIG. 7 is a schematic diagram of a multi-channel media stream grouping at an output end according to an embodiment of the present invention; FIG. 8 is a second embodiment of the present invention. Schematic diagram of a method for transmitting and outputting media streams in a multi-channel medium;

FIG. 9 is a flowchart of a method for transmitting and outputting multiple media data streams in multimedia communication according to Embodiment 3 of the present invention; FIG.

10 is a schematic diagram of an apparatus for transmitting a multi-channel media data stream in a multimedia communication according to an embodiment of the present invention; FIG. 11 is a schematic diagram of a device for receiving a multi-channel media data stream in a multimedia communication according to an embodiment of the present invention;

FIG. 12 is a schematic diagram of a system for transmitting multi-channel media data streams in multimedia communication according to an embodiment of the present invention. detailed description

The method, device and system for transmitting and receiving multiple media streams in multimedia communication according to an embodiment of the present invention are described in detail below with reference to the accompanying drawings.

It should be understood that the described embodiments are only a part of the embodiments of the invention, and not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present invention without creative efforts are within the scope of the present invention.

As shown in FIG. 2, a method for transmitting multiple media streams in multimedia communication according to an embodiment of the present invention includes:

5201. Group the multiple media data streams to be transmitted.

To group multiple media data streams, you can use any of the following methods: Grouping according to the positional relationship between the interfaces of the media data streams, that is, dividing the media data streams of the same physical interface or physical interfaces with similar positions a group; or grouped by the spatial location of the audio and/or video, that is, the spatial position of the image obtained by the camera, the spatial position of the sound obtained by the microphone, the video position, and/or the sound position of the image being consistent or similar, and / or audio as a group; or use the array microphone to pick up the multi-channel source position, form a plurality of corresponding independent audio streams, then group the multiple audio streams; or after obtaining the panoramic image through the wide-angle camera, The panoramic image is divided into multiplexed images, and the multiplexed video streams are grouped.

5202. Perform packet group description on the grouped multi-media data stream.

Specifically, it can be carried in a media data stream or a bearer channel corresponding to the media data stream. a packet description; or carrying a packet description in a control protocol related to the media data stream; or adding a control message carrying a packet description corresponding to the media data in the media data stream, thereby implementing grouping the grouped multi-media media data streams description.

S203. Transmit the packetized multi-media media data stream and a packet description corresponding to the media data stream.

The method for multi-media media stream transmission provided by the embodiment of the present invention can specify a group feature for a transmitted media data stream by grouping multiple media data streams and adding a packet description to the grouped media data stream, and receiving the terminal. Corresponding output processing can be performed according to the packet description, thereby realizing free combination transmission of multiple media streams at different terminals.

As shown in FIG. 3, a method for receiving and processing a multi-channel media stream in a multimedia communication according to an embodiment of the present invention includes:

S301. Receive a multi-media media data stream with a packet description and a packet description corresponding to the media data stream.

Specifically, the packet description may be carried in a media data stream or a bearer channel corresponding to the media data stream; or carried in a control protocol associated with the media data stream; or carried in a control message added in the media data stream.

S302. The packet outputs the media data stream and a packet description corresponding to the media data stream according to the packet description of the received multiple media data stream.

Embodiments of the present invention for receiving a multi-channel media stream receiving process in a multimedia communication by receiving a multi-channel media data stream with a packet description and a packet description corresponding to the media data stream, and outputting the packet according to the packet description packet Media data, thereby enabling multi-media media streams to be freely combined output at different terminals.

The specific embodiments of the present invention are described below by practical application of transmission and output of multiple media streams in multimedia communication in different scenarios.

Embodiment 1

As shown in FIG. 4, it is a schematic diagram of a multi-media media stream transmission and output scenario according to the embodiment. In this embodiment, the multi-media media stream is sent by the site A, and the multi-channel media stream is received by the site B. It must be noted that, in practical applications, the site A and the site B generally include both the transmitting and receiving devices, and the transmitting and receiving devices are also included in the terminal. The embodiments of the present invention have been simplified for the convenience of the description of the present invention.

In this embodiment, the site A includes the participants P1 and P2, and the terminal 1. Participant P1 has a corresponding microphone 1, camera 1 and data processing device 1 (such as a computer), participant P2 There are respectively corresponding microphone 2, camera 2 and data processing device 2 (such as a computer); site B includes terminal 2, a plurality of participants P3, display 1 and speaker 1 respectively located at position 1, located at position 3 Display 3 and speaker 3, and display 2 at position 2. The terminal 1 and the terminal 2 are connected through a network. As shown in Figure 5, the basic processing is as follows:

S501. The terminal 1 collects 6 pieces of media data and groups them. The six sets of media data include: data service media 1, data service media 2, video media 1, video media 2, audio media 1, audio media 2.

The grouping may be performed according to the positional relationship between the interfaces of the media data streams, that is, the media data streams of the physical interfaces of the same physical interface or similar locations are grouped. As shown in FIG. 6, the six media streams are divided into two groups of L1 and L2, wherein the media data of the L1 group corresponds to the participant P1, and includes: data service media 1, video media 1, audio media 1, and L2 group. The media stream corresponds to the participant P2, and includes: data service media 2, video media 2, and audio media 2.

The above is a method for determining the grouping according to the interface relationship of the terminal 1, corresponding to the data input interface of the participant P1, the audio input interface, and the data received by the data service input interface as the L1 group, corresponding to the video input interface and the audio input interface of the participant P2. And the data received by the data service input interface as the L2 group. The video interface may be a CVBS (Composite Video Broadcast Signal or Composite Video Blanking and Sync) interface, an S-Video (Separate Video) interface, a VGA (Video Graphics Array) interface, DVI ( Digital Visual Interface, digital video) interface, etc. The audio interface can be any form of MIC (Medium Interface Connector), Line IN (input signal) analog or digital interface. The data service interface can be a network port, a USB (Universal Serial BUS) interface, or a VGA interface (data content has been converted to video output on a PC), and the data service content includes file or slide show. These interfaces may also be physically merged. For example, the video interface and the audio interface may be combined into an HDMI (High Definition Multimedia Interface) interface or an IEEE 1394 interface, and even video, audio, and data services may be connected through a USB interface. In this case, video, audio, and data services can also be transmitted through logical channels.

The present invention is not limited thereto, and there may be multiple sets of video, audio, and data service streams for each packet. For example, packet L1 may also have two audio inputs, or two video inputs.

The method of determining the grouping relationship may also be by grouping the spatial locations of the audio and/or video, that is, obtaining the spatial position of the image by the camera, and/or obtaining the spatial position of the sound through the microphone, A video, and/or audio that has a spatial location of the image, and/or a sound location that is consistent or similar, is grouped. The method for determining the grouping relationship may further be: using the array microphone to pick up the multi-channel sound source positions, forming the multiple independent audio streams, grouping the multiple audio streams; or obtaining the panoramic image through the wide-angle camera, The panoramic image is sliced into multiplexed images, and the multiplexed video streams are grouped. Then, the corresponding video media and audio media are classified into a group, for example, the video media and the audio media can be grouped according to the positional relationship. That is, the media stream that is grouped may be constructed by other means, and is not necessarily obtained directly from the interface.

5502. The terminal 1 compresses and encodes the grouped media data stream.

S503. Perform packet group description on the grouped multiple media data streams.

There are three ways to group multiple media data streams:

Manner 1 carries a packet description in a media data stream or a bearer channel corresponding to the media data stream. For example, an extended header or media data stream of an RTP (real-time transport protocol) may carry a packet description. Specifically, the group feature may be populated in an extension header of the RTP or an extension field or a specific field in the media data stream.

For example, the RTP Header Extension fills the packet description field, and the RTP streams with the same packet description are the same group. Alternatively, the packet description may be identified in the corresponding bearer channel, for example, field expansion in RTP data (payload), and the packet description is added.

It is also possible to customize the extension header of the RTP and carry the packet description in the custom extension header.

The SIP/SDP (Session Initiation Protocol, Session Initiation Protocol) communication can also be carried in the RTP to carry the packet description.

Mode 2 carries packet description information in a control protocol related to the media stream. Specifically, the group feature can be populated in an extension field in the control protocol.

Taking the H.323 standard as an example, each media stream is assigned a corresponding session ID (sessionID), so that the associated session can be associated by specifying a packet description for each session ID, such as in an OpenLogicalChannel message. , extending a group description field, used to populate the group description, as defined below:

OpenLogicalChannel ::=SEQUENCE

{

forwardLogicalChannelNumber LogicalChannelNumber, forwardLogicalChannelParameters SEQUENCE

{

portNumber INTEGER (0..65535) OPTIONAL,

dataType DataType,

multiplexParameters CHOICE

{

h222LogicalChannelParameters H222LogicalChannelParameters,

h223LogicalChannelParameters H223LogicalChannelParameters,

v76LogicalChannelParameters V76LogicalChannelParameters, h225 OLogicalChannelParameters H2250LogicalChannelParameters,

None NULL ― for use with Separate Stack when

― multiplexParameters are not required

― or appropriate

}, forwardLogicalChannelDependency LogicalChannelNumber OPTIONAL,

-- also used to refer to the primary logical channel when using video redundancy coding replacementFor LogicalChannelNumber OPTIONAL,

groupID INTEGER (0..65535) OPTIONAL

},

― Used to specify the reverse channel for bi-directional open request

reverseLogicalChannelParameters SEQUENCE

{

dataType DataType,

Multiplex parameters CHOICE

{

― H.222 parameters are never present in reverse direction

h223LogicalChannelParameters H223LogicalChannelParameters,

v76LogicalChannelParameters V76LogicalChannelParameters, h225 OLogicalChannelParameters H225 OLogicalChannelParameters

} OPTIONAL, -- Not present for H.222 reverseLogicalChannelDependency LogicalChannelNumber OPTIONAL,

― also used to refer to the primary logical channel when using video redundancy coding replacementFor LogicalChannelNumber OPTIONAL

} OPTIONAL, — Not present for uni-directional channel request separateStack NetworkAccessParameters OPTIONAL,

― for Open responder to establish the stack

encryptionSync EncryptionSync OPTIONAL ― used only by Master

}

The groupID is an extended packet description field used to populate the packet description.

Mode 3 increases the control of the packet description carrying the corresponding media data in the media data stream. Interest. Specifically, a group feature is defined in each of the control messages, and at least one media stream belonging to the same group is included in a message body of the control message.

For example, add a control message that informs the peer which media data is the same group. 4 Set the session logical channel number of the video, audio and data service data of the group L1 to 1, respectively

2, 3, then send a message Grouplndication, the message contains the logical channel number 1, 2, 3, so that the receiving end judges that the media data of the logical channel number 1, 2, 3 is the same group according to the message.

When the SIP/SDP protocol is used for multimedia communication, the packet description can also be carried by using multiple sessions. E.g:

ν=0

o=mhandley 2890844526 2890842807 IN IP4 126.16.64.4

s=Pl m=audio 49170 RTP/AVP 0

m=video 51372 RTP/AVP 31

m=application 32416 udp wb v=0

o=mhandley 2890844526 2890842807 IN IP4 126.16.64.4

s=P2 m=audio 49171 RTP/AVP 0

m=video 51373 RTP/AVP 31

m=application 32417 udp wb

The session PI (S=P1) corresponds to the group LI, and the session P2 (S=P2) corresponds to the group L2, which respectively contains the audio, video and application data streams.

Of course, you can also specify the grouping by adding an attribute, for example:

v=0 m=audio 49170 RTP/AVP 0

m=video 51372 RTP/AVP 31

m=application 32416 udp wb

m=audio 49171 RTP/AVP 0

m=video 51373 RTP/AVP 31

m=application 32417 udp wb g=49170 51372 32416

g=49171 51373 32417 The corresponding media data of ports 49170, 51372, 32416 are divided into groups by the description of g=49170 51372 32416, and the corresponding media data of ports 49171, 51373, 32417 are divided into another group by _g =49171 51373 32417.

Alternatively, grouping can be achieved by directly extending the media description and adding a group description, for example: ν=0 m=audio 49170 RTP/AVP 0 group 1

m=video 51372 RTP/AVP 31 group 1

m=application 32416 udp wb group 1 m=audio 49171 RTP/AVP 0 group2

m=video 51373 RTP/AVP 31 group2

m=application 32417 udp wb group2 In this way, groupl and group2 in the media description are used to describe the group number corresponding to the media stream, and the media streams of the same group number are in the same group. Groupl and group2 can be numeric or character identifiers.

5504. The terminal 1 sends the multi-channel media data to the terminal 2.

S505: The terminal 2 receives the multi-path media data with the packet description, and performs decoding and decompression processing.

5506. Output the multi-path media data according to the group description.

As shown in FIG. 4 and FIG. 7, specifically, according to the grouping relationship, the video data 1 and the audio data 1 corresponding to the packet L1 are output to the display 1 and the speaker 1 of the position 1, and the video data 2 and the audio data of the packet L2 are determined. 2 Output to the display 3 and the speaker 3 of the position 3, and output the data service medium 1 and the data service medium 2 of the packets L1 and L2 to the display 2 of the position 2 as needed.

The embodiment of the present invention can also implement the data service data output interface to switch and output the obtained data services of multiple senders, that is, can be switched as needed. For example, when the participant P1 at the sending end explains the slide show, the data service output interface is switched to the data service output of L1, and when P2 explains the slide show, the data service output is switched to L2. The video, audio, and data service output interfaces herein can also be in various forms and combinations as the aforementioned input interfaces.

Embodiments of the present invention may further implement switching between video data, audio data, or data service data output by an output interface according to a packet description of the received media data stream. For example, the terminal 2 can further automatically switch the output contents of the video output 1 and the video output 2 interface to the video of the group L1 when the participant P1 explains the slide, according to the group description, and output the audio 1 and The audio output 2 is switched to output the audio of the packet L1, so that the participants of the site B can simultaneously see the image of P1 on their respective displays, and hear the sound of P1 on the respective speakers. Further, the receiving end can notify the transmitting end to pause the video and audio collection of the L2 group or suspend the transmission of the video and audio of the L2 group to reduce the occupation of the bandwidth.

The network connecting terminal 1 and terminal 2 may be a circuit domain network (such as E1/SDH/ISDN) or a packet network (such as an IP network), and the communication protocol may use H.320/H.323/H.324/ SIP, etc.

In the above embodiment, the video media 1, the audio media 1 and the data service media 1 are divided into one group, which is a common application situation, but the actual grouping can be very flexible, for example, the video media 2 is taken as a group, The audio medium 2 and the video medium 1 are grouped as one.

Through the above steps, the embodiment of the present invention implements transmission and output of multi-channel media data by grouping, so that the receiving end can freely combine and output the multi-channel media streams, and also solves the transmission and control of the data service corresponding to the video and audio. . The embodiment of the invention also implements the use of a single terminal in a conference site, thereby solving the problem of excessive cost caused by using multiple terminals in the prior art. Embodiment 2

On the basis of the first embodiment, the method for transmitting and outputting the multi-channel media data stream in this embodiment may also be processed by an intermediate device (for example, a conference control device, specifically, a multi-point control unit MCU).

After receiving the media data stream carrying the packet description and the corresponding packet description, the MCU obtains the packet description of each media data, and can re-determine the grouping relationship of each media data stream according to the output requirement. In this embodiment, the grouping relationship of the media data streams may be forwarded by the multipoint control unit MCU to change the grouping relationship between the media data streams, or to regenerate the grouping relationship of the multiple media data streams.

As shown in FIG. 8, the video media stream and the audio media stream sent by the terminal 1 and the data service media stream sent by the terminal 3 are combined into a new group L at the MCU and then sent to the terminal 2. The specific combination is not limited to the above embodiment.

It can be understood that in the video conference, the conference control device mainly performs the main media stream control. Therefore, in this way, the corresponding media stream (audio media stream, video media stream, and the like) can be flexibly and according to the request of some conference terminals. Data media streams), combined, can be diverse to meet the needs of users.

The embodiment of the invention implements transmitting and outputting multiple media streams by grouping, thereby receiving The terminal can freely combine and output the multiple media streams, and also solves the transmission and control of the data service media stream corresponding to the video and audio. Implementation three

This embodiment is a method for outputting a multi-channel media stream by the terminal 2 on the basis of transmitting the multi-channel media data stream in the implementation, which specifically includes the following steps:

5901. Arranging a plurality of speakers in the conference site B;

5902. The terminal 2 acquires spatial location information of each display in the site B.

5903. The video data 1 in the received packet L1 is output through the display 1 of the position 1, and the video data 2 in the packet L2 is output through the display 3 of the position 3.

5904. The audio data 1 of the received packet L1 is output through at least one speaker in the conference site B, so that the spatial position sense is the same as or similar to the display 1 of the location 1.

S905. The audio data 2 of the received packet L2 is output through at least one speaker in the conference site B, so that the spatial position sense is the same as or similar to the display 3 of the location 3.

The above steps S904 and S905 are not limited to the above sequence, and step S905 may be before step S904.

In this embodiment, the sounds heard by the participants can be the same or similar to the corresponding video positions in the multiple venues, thereby realizing the realism of the conference television.

Embodiments of the present invention also provide an apparatus for multi-channel media stream transmission, which can solve the problem that a multi-channel media stream can only be transmitted between specific terminals.

As shown in FIG. 10, an apparatus for transmitting a plurality of media data streams in a multimedia communication according to an embodiment of the present invention includes:

a grouping unit 101, configured to group the multiple media data streams to be transmitted;

The packet description unit 102 performs packet description on the packetized multi-media media data stream; the data transmission unit 103 transmits the packetized multi-channel media data stream and a packet description corresponding to the media data stream.

Further, in a preferred embodiment of the present invention, the grouping unit 101 may group the media data streams by any one of the following methods: grouping according to a positional relationship between interfaces of the media data streams, that is, the same physics The media data streams of interfaces or similar physical interfaces are grouped into one group; or grouped by spatial locations of audio and/or video, that is, the spatial position of the image is obtained by the camera, the spatial position of the sound is obtained by the microphone, and the spatial position of the image is obtained. , and / or sound position Consistent or similar video, and/or audio as a group; or using an array microphone to pick up multiple audio source locations, forming multiple independent audio streams, grouping the multiple audio streams; or acquiring through a wide-angle camera After the panoramic image is divided into the multiplexed images, the multiplexed video streams are grouped.

The packet description unit 102 may perform group description on the grouped media data stream in any of the following manners: carrying the packet description in the media data stream or the bearer channel corresponding to the media data stream; or related to the media data stream The control protocol carries the packet description; or adds a control message carrying the packet description corresponding to the media data in the media data stream.

The media data stream in this embodiment may specifically be one or more of the following: an audio stream, a video stream, and a data stream.

In the embodiment of the present invention, the apparatus for transmitting the multi-channel media data stream in the multimedia communication, the multi-channel media data is grouped by the grouping unit, and the packet description unit performs group description on the multimedia data stream, thereby realizing the multi-channel media stream to be performed through one terminal. Transmitting, and enabling the receiving terminal to output each media data according to the packet description corresponding packet, realizing free combination transmission of the multi-channel media data at different terminals.

The invention also provides a device for receiving and processing multi-channel media data streams in multimedia communication, which can solve the problem that multi-channel media streams cannot be freely combined and output.

As shown in FIG. 11, an apparatus for receiving and processing a multi-media media stream in multimedia communication according to an embodiment of the present invention includes a receiving unit 111 and an output unit 112.

The receiving unit 111 is configured to receive a multi-media media data stream with a packet description and a packet description corresponding to the media data stream. The packet description may be carried in a media data stream or a bearer channel corresponding to the media data stream; or carried in a control protocol related to the media data stream; or carried in a control message added in the media data stream.

The output unit 112 is configured to output the media data stream in a packet according to the packet description of the received multiple media data stream.

Further, the apparatus may further include: a switching unit, configured to switch, according to a packet description of the media data stream received by the receiving unit, the media data stream output by the output unit. The apparatus can also include a suspending unit for suspending the collection or transmission of media data streams of other groups than the designated group when all of the output ports are switched to the media stream of the specified group.

Embodiment of the present invention, when receiving a multi-channel media data stream with a packet description and a packet description corresponding to the media data stream, According to the packet description, the media data stream is outputted in groups, thereby realizing that the multi-media media stream can be freely outputted at different terminals. In addition, the switching unit is also capable of switching the transmission of media data of a specific group, and the suspending unit can suspend the collection or transmission of media data of certain groups, thereby reducing network pressure.

The apparatus for transmitting multiple media streams in the multimedia communication provided by the embodiment of the present invention, and the apparatus for receiving the processing, can refer to the first, second and third embodiments of the foregoing method to realize free combination output of the multi-media media streams in the multimedia communication.

Embodiments of the present invention also provide a conference terminal, including a transmitting device and a receiving processing device. The transmitting device is configured to acquire and group a multi-channel media data stream to be transmitted, perform packet group description on the grouped multi-channel media data stream, and transmit the grouped multi-channel media data stream and the device a packet description corresponding to the media data stream; and/or

The receiving processing apparatus is configured to receive a multi-channel media data stream with a packet description and a packet description corresponding to the media data stream, and output the packet according to the packet description of the received multi-channel media data stream. Media data stream.

For a conference terminal, it is preferable to have both a transmitting device and a receiving processing device.

Based on the foregoing solution, the sending device further includes:

a grouping unit, configured to group the multi-media data streams to be transmitted, including grouping according to a positional relationship between interfaces of the media data streams, or grouping by spatial locations of audio and/or video, or picking up by using an array microphone After the multi-channel audio source is formed, the multi-channel audio streams are grouped, and the multi-channel audio streams are grouped; or after the panoramic image is acquired by the wide-angle camera, the panoramic image is divided into multiple separated images. And grouping the multi-channel video stream; the packet description unit, performing packet group description on the grouped multi-path media data stream; and including carrying the packet description in the media data stream or the bearer channel corresponding to the media data stream; Or carrying a packet description in a control protocol related to the media data stream; or adding a control message carrying a packet description corresponding to the media data in the media data stream;

And a data transmission unit, configured to transmit the packetized multi-media media data stream and a packet description corresponding to the media data stream.

The conference terminal of the embodiment of the present invention can implement the grouping of the multimedia data stream to be transmitted and the carrying of the packet description, so that the multi-channel media data stream can be sent in a packet, and the multimedia data stream that receives the packet transmission and the corresponding After the packet description, the media data streams can be output according to the packet description, thereby realizing free combination output of the multi-channel media data. The embodiment of the invention also provides a system for multi-media media stream transmission in multimedia communication, which can solve the problem that multiple media streams cannot be combined and output freely.

This embodiment can be explained in conjunction with the embodiment of Fig. 4. As shown in FIG. 12, a system for multi-media streaming in multimedia communication according to an embodiment of the present invention includes: a data acquiring device 121, a transmitting device 122, a receiving processing device 123, and a data output device 124. among them,

The data acquisition device 121 includes a video acquisition device, an audio acquisition device, or a data acquisition device, for acquiring a multi-channel media data stream.

The transmitting device 122 is configured to group the multi-channel media data streams acquired by the data acquiring device, perform packet group description on the grouped multi-channel media data streams, and transmit the grouped multi-channel media data. And a packet description corresponding to the media data stream.

The receiving processing device 123 is configured to receive a multi-channel media data stream with a packet description and a packet description corresponding to the media data stream, and according to the packet description of the received multi-channel media data stream, the packet output station The media data stream.

The data output device 124 is configured to output the grouped media data stream through a video output device, an audio output device, or a data output device.

Based on the foregoing solution, the transmission device may further include:

In a preferred embodiment of the present invention, the system may further include:

The conference control device is configured to change the grouping relationship by forwarding the media data stream grouping relationship, or group the multiple media data streams of different sites. The conference control device can be an MCU.

A system for transmitting multiple media streams in multimedia communication provided by an embodiment of the present invention Grouping the multi-channel media data, grouping the packetized multi-media media data, and transmitting the media data with the packet description and the corresponding packet description, and the output device receives the multi-channel media data After the packet description, the media data is output according to the packet description packet, so that free combination output of the multi-media media stream can be realized; the preferred MCU in the system can also change or forward the packet relationship of the media data. The media data stream grouping relationship is forwarded to change the grouping relationship, thereby enabling free combination output of the multi-media media streams.

The system for transmitting multiple media streams in the multimedia communication provided by the embodiment of the present invention can refer to the first, second, and third embodiments of the foregoing method to implement multi-channel media stream transmission, reception, and output processing in multimedia communication.

A person skilled in the art can understand that all or part of the process of implementing the above embodiment method can be completed by a computer program to instruct related hardware, and the program can be stored in a computer readable storage medium. In execution, the flow of an embodiment of the methods as described above may be included. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

The above is only the specific embodiment of the present invention, but the scope of the present invention is not limited thereto, and any change or replacement that can be easily conceived by those skilled in the art within the technical scope of the present invention is All should be covered by the scope of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

Claim

A method for transmitting a multi-channel media data stream in a multimedia communication, comprising: grouping a plurality of media data streams to be transmitted;

Performing group description on the grouped multi-media data stream;

Transmitting the packetized multi-media media data stream and a packet description corresponding to the media data stream.

2. The method according to claim 1, wherein the grouping the multiple media data streams is specifically:

Grouping according to the positional relationship between interfaces of the media data stream, that is, dividing the media data streams of the same physical interface or physical interfaces with similar positions into one group; or

By spatial location of audio and/or video, that is, the spatial position of the image is acquired by the camera, the spatial position of the sound is obtained by the microphone, and the video, and/or audio of the spatial position, and/or the sound position of the image are consistent or similar. a group; or

After the multi-channel audio source is picked up by the array microphone to form a plurality of corresponding independent audio streams, the multiple audio streams are grouped; or

After the panoramic image is acquired by the wide-angle camera, the panoramic image is sliced into multiplexed images, and the multiplexed video streams are grouped.

The method according to claim 1 or 2, wherein the grouping of the grouped multi-media data streams is performed, including:

Carrying a packet description in a media data stream or a bearer channel corresponding to the media data stream; or carrying a packet description in a control protocol associated with the media data stream; or

A control message carrying a packet description corresponding to the media data is added to the media data stream.

The method according to claim 1 or 2, wherein the grouping the multi-channel media streams to be transmitted comprises:

The conference control device changes the grouping relationship by forwarding the grouping relationship of the multiple media data streams; or the conference control device groups the multiple media data streams of different sites.

The method according to claim 1 or 2, wherein the media data stream is specifically one or more of the following: an audio stream, a video stream, and a data stream.

A method for receiving and processing a multi-media media stream in a multimedia communication, comprising: receiving a multi-channel media data stream with a packet description and a packet description corresponding to the media data stream; And outputting the media data stream according to a packet description of the received multi-media media data stream.

The method according to claim 6, wherein, according to the packet description of the received multi-media media stream, the packet outputting the media data stream comprises:

Outputting the received video data to a corresponding video output interface according to a packet description of the received media data stream; and/or

Outputting the received audio data to a corresponding audio data interface; and/or

And outputting the received data service data to a corresponding data service data output interface.

8. The method of claim 7, wherein outputting the received data service data to a corresponding data service data output interface comprises:

The data service data output interface performs switching output on the received data service data of the plurality of senders.

And outputting the video data to the corresponding display device according to the packet description of the received multi-media media data stream, acquiring spatial location information of the display device, and using at least one speaker data for the audio data related to the video data. The device outputs such that the output audio spatial location is the same or similar to the spatial location of the display device.

10. The method of claim 6, further comprising:

Switching video data, audio data, or data service data output by the output interface according to the packet description of the received media data stream.

An apparatus for transmitting a plurality of media data streams in a multimedia communication, comprising: a grouping unit, configured to group a plurality of media data streams to be transmitted;

a packet description unit, grouping and describing the grouped multi-media media data stream;

12. The apparatus according to claim 11, wherein the grouping unit groups the plurality of media data streams to be transmitted by any one of the following:

Grouping by the spatial position of the audio and/or video, that is, obtaining the spatial position of the image through the camera, the spatial position of the sound obtained by the microphone, the spatial position of the image, and/or the sound position being consistent or phased Near video, and/or audio as a group; or

The device according to claim 11, wherein the packet description unit groups the grouped multi-media data streams by any one of the following:

14. Apparatus according to any one of claims 11 to 13 wherein:

The media data stream is specifically one or more of the following: an audio stream, a video stream, and a data stream.

15. A device for receiving and processing a multi-channel media data stream in multimedia communication, comprising:

a receiving unit, configured to receive a multi-media media data stream with a packet description and a packet description corresponding to the media data stream;

And an output unit, configured to output the media data stream according to the packet description of the received multi-media media data stream.

The device according to claim 15, further comprising:

And a switching unit, configured to switch the media data stream output by the output unit according to a packet description of the media data stream received by the receiving unit.

17. A conference terminal, comprising:

a sending device, configured to acquire and group a plurality of media data streams to be transmitted, perform packet group description on the grouped multi-channel media data stream, and transmit the grouped multi-channel media data stream and the media a packet description corresponding to the data stream; and/or

a receiving processing device, configured to receive a multi-media media data stream with a packet description and a packet description corresponding to the media data stream, and output the media data according to a packet description of the received multi-channel media data stream flow.

The conference terminal according to claim 17, wherein the transmitting device comprises: a grouping unit, configured to group the multiple media data streams to be transmitted, including locations between interfaces according to the media data streams Relationships are grouped or grouped by spatial location of audio and/or video, Or using the array microphone to pick up the multi-channel sound source positions to form a plurality of corresponding independent audio streams, and then grouping the multiple audio streams; or after acquiring the panoramic image through the wide-angle camera, dividing the panoramic image into multiple After the separated images, the multiple video streams are grouped;

a packet description unit, configured to perform group description on the grouped multi-media data stream; including carrying a packet description in a media data stream or a bearer channel corresponding to the media data stream; or carrying in a control protocol related to the media data stream a packet description; or adding a control message carrying a packet description corresponding to the media data in the media data stream;

A system for transmitting a plurality of media data streams in a multimedia communication, comprising: a data acquisition device, comprising: a video acquisition device, an audio acquisition device, or a data acquisition device, configured to acquire a multi-channel media data stream;

The system according to claim 19, wherein the transmitting means comprises: a grouping unit, configured to group the plurality of media data streams to be transmitted, including a positional relationship between interfaces according to the media data stream Grouping, or grouping by spatial locations of audio and/or video, or picking up multiple audio source locations using an array microphone to form a plurality of corresponding independent audio streams, grouping the multiple audio streams; or After the wide-angle camera acquires the panoramic image, the panoramic image is divided into multiple demultiplexed images, and the multiple video streams are grouped;

a data transmission unit, transmitting the packetized multi-media media data stream and the media data stream pair The description of the grouping should be.

The system according to claim 19 or 20, further comprising:

The conference control device is configured to change the grouping relationship by forwarding the media data stream grouping relationship, or group the multiple media data streams of different sites.