CN104853226A

CN104853226A - Method, device, equipment and system for processing multimedia data

Info

Publication number: CN104853226A
Application number: CN201410053254.XA
Authority: CN
Inventors: 邸佩云; 许阳坡; 艾常权
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2014-02-17
Filing date: 2014-02-17
Publication date: 2015-08-19

Abstract

The embodiment of the invention provides a method, device, equipment and system for processing multimedia data. The method comprises the steps: obtaining coding reference information, coding to-be-coded data if the coding reference information is judged to meet the preset conditions, and obtaining first-segment data and second-segment data, wherein the quality of the first-segment data is higher than the quality of the second-segment data; transmitting the first-segment data to an enhanced segment transmission queue, and transmitting the second-segment data to a real-time segment transmission queue; transmitting the first-segment data and the second-segment data to a server under the condition that the second-segment data is transmitted in advance, so that the server provides real-time data service for a user according to the second-segment data, and employs the first-segment data and the corresponding second-segment data to provide non-real-time data service for the user.

Description

The processing method of multi-medium data, device, equipment and system

Technical field

The embodiment of the present invention relates to wireless communication technology, particularly relates to a kind of processing method of multi-medium data, device, equipment and system.

Background technology

Along with the lifting of the universal of intelligent mobile terminal and wireless network bandwidth, terminal use for the demand of transmission of video also in continuous increase.The familiar transmission of video of present user has two kinds, and one is non real-time, and another kind is real-time.A kind of mode of non real-time transmission of video, be applicable to closely transmission of video, first that video capture is good, then between terminal by transmission meanss such as bluetooths by video from a terminal transmission to another terminal, the quality of the video received by this transmission means is better, but requires that transmit leg and recipient are in limited distance; Another kind of mode, being applicable to long-distance video transmission, is that the video taken is uploaded onto the server by transmitting terminal, another terminal downloads video content from server again, such as micro-letter, the quality of the video of this non real-time transmission is lower, and the time span of the video of shooting is shorter.The video of real-time Transmission, such as existing qik and facetime technology, the transmission of video of this class ensure that the real-time of transmission, the video watched captured by transmit leg that namely receiving terminal can be real-time.

In existing real-time transmission of video technology, after user A and user B connects, user A gathers audio, video data, compression coding is carried out to audio, video data, audio, video data after compression is packaged into IP bag, sent to user B by network, in transmission data procedures, user A receives the data receiver situation message that user B feeds back, thus know the situation of network, user A can adjust the coding bit rate output of user A according to the network condition known, when the network bandwidth declines time, user A adjusts coding parameter, exports the code stream of low bit-rate; When the network bandwidth promotes, user A adjusts coding parameter, exports the data flow of high code check, and the code check of user A and the adjustment situation of bandwidth are as shown in Figure 1.Fig. 1 is that in prior art, data flow code check is with the adjustment situation schematic diagram of bandwidth, and x-axis is the time, and unit is second; Y-axis is bit rate, and unit is kbps.

As seen from Figure 1, in existing Real-time Video Transport Technology, when the network bandwidth declines, transmitting terminal can adjust coding parameter to export the code stream of low bit-rate, thus ensure audio video transmission continuity when low network band width, realize the real-time Transmission of audio frequency and video, in real-time Transmission process, server can obtain this video data for non real time user by the video data in corresponding preservation transmitting procedure, but when adopting the Real-time Video Transport Technology of prior art, due to the impact by the network bandwidth, there is fluctuation in the quality of the video data in transmitting procedure, when the network bandwidth declines, the video data that server is preserved is the low bit-rate data in real-time Transmission process, even if non real time user watches the video data that when network bandwidth declines, server is preserved again when the network bandwidth promotes like this, this non real time user still cannot obtain high-quality video data.

Summary of the invention

The embodiment of the present invention provides a kind of processing method, device, equipment and system of multi-medium data.

First aspect, the embodiment of the present invention provides a kind of sending method of multi-medium data, comprising:

Obtain coded reference information, know that described coded reference information meets pre-conditioned if judge, then encode to data to be encoded and obtain the first segment data and the second segment data respectively, the quality of wherein said first segment data is higher than the quality of described second segment data;

Described first segment data is sent to and strengthens in segmentation transmit queue, described second segment data is sent in described real-time segmentation transmit queue;

Under the prerequisite that described second segment data of preferential guarantee sends in time, described first segment data and described second segment data are sent to server, for described server according to described second segment data for user provides real-time data service, and apply described first segment data and corresponding described second segment data for user non-real-time data business be provided.

In conjunction with first aspect, in the first mode in the cards of first aspect, described under the prerequisite that preferentially described second segment data of guarantee sends in time, described first segment data and described second segment data are sent to server, comprising:

Described second segment data is sent to described server;

Within the free time sending described second segment data or after sending described second segment data or utilize and send the remaining network bandwidth of described second segment data, send described first segment data to described server.

In conjunction with the first mode in the cards of first aspect or first aspect, in the second of first aspect mode in the cards, described first segment data of described application and corresponding described second segment data provide non-real-time data business for user, comprising:

Applying described first segment data replaces corresponding described second segment data to provide non-real-time data business for user; Or

Described first segment data and corresponding described second segment data are carried out mergings generation the 4th segment data, applies described 4th segment data and replace described second segment data of correspondence to provide non-real-time data business for user.

In conjunction with first aspect, in the third mode in the cards of first aspect, described method also comprises:

Know that described coded reference information does not meet pre-conditioned if judge, then coding carried out to data to be encoded and only obtain the 3rd segment data, described 3rd segment data higher than the code check of described second segment data, and is sent in described real-time segmentation transmit queue to send to described server by the code check of wherein said 3rd segment data.

In conjunction with the third mode in the cards of first aspect, in the 4th kind of mode in the cards of first aspect, described under the prerequisite that preferentially described 3rd segment data of guarantee sends in time, described first segment data and described 3rd segment data are sent to server, comprising:

Described 3rd segment data is sent to described server;

Within the free time sending described 3rd segment data or after sending described 3rd segment data or utilize and send the remaining network bandwidth of described 3rd segment data, send described first segment data to described server.

In conjunction with the third mode in the cards of first aspect or first aspect, in the 5th kind of mode in the cards of first aspect, described coded reference information is the transmission delay that in described real-time segmentation transmit queue, current data to be sent are corresponding, is correspondingly describedly pre-conditionedly greater than the first predetermined threshold value for described transmission delay; Or,

Described coded reference information is the current network bandwidth, is correspondingly describedly pre-conditionedly less than the second predetermined threshold value for the described network bandwidth; Or,

Described coded reference information is the data volume in current real-time segmentation transmit queue, is correspondingly describedly pre-conditionedly greater than the 3rd predetermined threshold value for the data volume in described current real-time segmentation transmit queue; Or

Described coded reference information is the segment data number in current real-time segmentation transmit queue, is correspondingly describedly pre-conditionedly greater than the 4th predetermined threshold value for the segmentation number in described current real-time segmentation transmit queue.

In conjunction with the 5th kind of mode in the cards of first aspect, in the 6th kind of mode in the cards of first aspect, described transmission delay is the total amount of data/network bandwidth of current data to be sent in described real-time segmentation transmit queue, or is playing duration corresponding to the total amount of data of current data to be sent in described real-time segmentation transmit queue.

In conjunction with the third mode in the cards of first aspect, in the 7th kind of mode in the cards of first aspect, described method also comprises:

Generate the descriptor that described first segment data, described second segment data and described 3rd segment data are corresponding, and send to described server.

Second aspect, the embodiment of the present invention provides a kind of method of reseptance of multi-medium data, comprising:

The first segment data that receiving terminal sends and the second segment data, the quality of wherein said first segment data is higher than the quality of described second segment data; Described first segment data and described second segment data be described terminal in acquisition coded reference information and judge to know described coded reference information meet pre-conditioned after, carrying out encoding to data to be encoded obtains; Wherein, described first segment data is sent by the enhancing segmentation transmit queue of described terminal, and described second segment data is sent by the real-time segmentation transmit queue of described terminal; Described first segment data is preferentially ensureing to send under the prerequisite that described second segment data sends in time;

According to described second segment data for user provides real-time data service, and apply described first segment data with corresponding described second segment data for user provides non-real-time data business.

In conjunction with second aspect, in the first mode in the cards of second aspect, described first segment data of described application and corresponding described second segment data provide non-real-time data business for user, comprising:

In conjunction with the first mode in the cards of second aspect, in the third mode in the cards of second aspect, described method also comprises:

Receive the 3rd segment data that described terminal sends, the code check of wherein said 3rd segment data is higher than the code check of described second segment data, and described 3rd segment data is stored in the real-time segmentation transmit queue of described terminal; Described 3rd segment data be described terminal judgement know described coded reference information do not meet pre-conditioned after, carrying out encoding to data to be encoded obtains.

The third aspect, the embodiment of the present invention provides a kind of terminal equipment, comprising:

Acquisition module, for obtaining coded reference information, know that described coded reference information meets pre-conditioned if judge, then encode to data to be encoded and obtain the first segment data and the second segment data respectively, the quality of wherein said first segment data is higher than the quality of described second segment data;

Memory module, strengthening in segmentation transmit queue for described first segment data being sent to, described second segment data being sent in described real-time segmentation transmit queue;

Sending module, for preferentially ensureing under the prerequisite that described second segment data sends in time, described first segment data and described second segment data are sent to server, for described server according to described second segment data for user provides real-time data service, and apply described first segment data and replace corresponding described second segment data to provide non-real-time data business for user.

In conjunction with the third aspect, in the first mode in the cards of the third aspect, described sending module specifically comprises:

First transmitting element, for sending described second segment data to described server;

Second transmitting element, for within the free time sending described second segment data or after sending described second segment data or utilize and send the remaining network bandwidth of described second segment data, send described first segment data to described server.

In conjunction with the first mode in the cards of the third aspect or the third aspect, in the second of the third aspect mode in the cards, described first segment data of described application and corresponding described second segment data provide non-real-time data business for user, comprising:

In conjunction with the third aspect, in the third mode in the cards of the third aspect, described acquisition module also for: if judge know that described coded reference information does not meet pre-conditioned, then coding carried out to data to be encoded and only obtain the 3rd segment data, described 3rd segment data higher than the code check of described second segment data, and is sent in described real-time segmentation transmit queue to send to described server by the code check of wherein said 3rd segment data;

Described sending module also for: preferential ensure the prerequisite that described 3rd segment data sends in time under, described first segment data and described 3rd segment data are sent to server, for described server according to described 3rd segment data for user provides real-time data service, and apply described first segment data for user non-real-time data business be provided.

In conjunction with the third mode in the cards of the third aspect, in the 4th kind of mode in the cards of the third aspect, described sending module specifically comprises:

3rd transmitting element, for sending described 3rd segment data to described server;

4th transmitting element, for within the free time sending described 3rd segment data or after sending described 3rd segment data or utilize and send the remaining network bandwidth of described 3rd segment data, send described first segment data to described server.

In conjunction with the third mode in the cards of the third aspect or the third aspect, in the 5th kind of mode in the cards of the third aspect,

Described coded reference information is the transmission delay that in described real-time segmentation transmit queue, current data to be sent are corresponding, is describedly accordingly pre-conditionedly greater than the first predetermined threshold value for described transmission delay; Or,

In conjunction with the 5th kind of mode in the cards of the third aspect, in the 6th kind of mode in the cards of the third aspect, described transmission delay is the total amount of data/network bandwidth of current data to be sent in described real-time segmentation transmit queue, or is playing duration corresponding to the total amount of data of current data to be sent in described real-time segmentation transmit queue.

In conjunction with the third mode in the cards of the third aspect, in the 7th kind of mode in the cards of the third aspect, also comprise:

Processing module, for generating descriptor corresponding to described first segment data, described second segment data and described 3rd segment data, and sends to described server.

Fourth aspect, the embodiment of the present invention provides a kind of terminal equipment, comprising: the encoder of communication connection mutually and sending module; Wherein, described sending module comprises network bandwidth estimation module, real-time segmentation transmit queue and strengthens segmentation transmit queue; Wherein:

Described network bandwidth estimation module, for obtaining coded reference information;

Described encoder is used for, if judge to know that described coded reference information meets pre-conditioned in described network bandwidth estimation module, then encode to data to be encoded and obtain the first segment data and the second segment data respectively, the quality of wherein said first segment data is higher than the quality of described second segment data;

Described real-time segmentation transmit queue for storing described second segment data, and sends to server;

Under described enhancing segmentation transmit queue is used for the prerequisite sent in time in described second segment data of preferential guarantee, described first segment data is sent to server, for described server according to described second segment data for user provides real-time data service, and apply described first segment data and corresponding described second segment data for user non-real-time data business be provided.

In conjunction with fourth aspect, in the first mode in the cards of fourth aspect, described enhancing segmentation transmit queue specifically for, within the free time sending described second segment data or after sending described second segment data or utilize and send the remaining network bandwidth of described second segment data, send described first segment data to described server.

In conjunction with the first mode in the cards of fourth aspect or fourth aspect, in the second of fourth aspect mode in the cards, described first segment data of described application and corresponding described second segment data provide non-real-time data business for user, comprising:

In conjunction with fourth aspect, in the third mode in the cards of fourth aspect, described encoder also for, if judge to know that described coded reference information does not meet pre-conditioned in described network bandwidth estimation module, then carry out coding to data to be encoded and only obtain the 3rd segment data, the code check of wherein said 3rd segment data is higher than the code check of described second segment data;

Correspondingly, described real-time segmentation transmit queue also for storing described 3rd segment data, and sends to described server;

Described enhancing segmentation transmit queue is also for preferentially ensureing under the prerequisite that described 3rd segment data sends in time, described first segment data is sent to server, for described server according to described 3rd segment data for user provides real-time data service, and apply described first segment data for user non-real-time data business be provided.

In conjunction with the third possible implementation of fourth aspect, in the 4th kind of mode in the cards of fourth aspect, described enhancing segmentation transmit queue also for, within the free time sending described 3rd segment data or after sending described 3rd segment data or utilize and send the remaining network bandwidth of described 3rd segment data, send described first segment data to described server.

In conjunction with the third mode in the cards of fourth aspect or fourth aspect, in the 5th kind of mode in the cards of fourth aspect, described coded reference information is the transmission delay that in described real-time segmentation transmit queue, current data to be sent are corresponding, is correspondingly describedly pre-conditionedly greater than the first predetermined threshold value for described transmission delay; Or,

In conjunction with the 5th kind of mode in the cards of fourth aspect, in the 6th kind of mode in the cards of fourth aspect, described transmission delay is the total amount of data/network bandwidth of current data to be sent in described real-time segmentation transmit queue, or is playing duration corresponding to the total amount of data of current data to be sent in described real-time segmentation transmit queue.

5th aspect, the embodiment of the present invention provides a kind of server, comprising:

Receiver module, the first segment data sent for receiving terminal and the second segment data, the quality of wherein said first segment data is higher than the quality of described second segment data; Described first segment data and described second segment data be described terminal in acquisition coded reference information and judge to know described coded reference information meet pre-conditioned after, carrying out encoding to data to be encoded obtains; Wherein, described first segment data is sent by the enhancing segmentation transmit queue of described terminal, and described second segment data is sent by the real-time segmentation transmit queue of described terminal; Described first segments is preferentially ensureing to send under the prerequisite that described second segment data sends in time;

Processing module, for according to described second segment data for user provides real-time data service, and apply described first segment data and corresponding described second segment data for user non-real-time data business be provided.

In conjunction with the 5th aspect, in the first mode in the cards in the 5th, described first segment data of described application and corresponding described second segment data provide non-real-time data business for user, comprising:

In conjunction with the first mode in the cards of the 5th aspect, in the mode in the cards of the second in the 5th, described receiver module also for:

6th aspect, the embodiment of the present invention provides a kind of transmitting system of multi-medium data, comprising:

The first server to any one possible implementation of the second of the first terminal equipment to the 7th kind of any one possible implementation of the third aspect, the third aspect and the 5th aspect, the 5th aspect.

The processing method of embodiment of the present invention multi-medium data, device, equipment and system, by obtaining coded reference information, know that described coded reference information meets pre-conditioned if judge, then data to be encoded encoded and obtain the first segment data and the second segment data respectively, the quality of wherein said first segment data is higher than the quality of described second segment data, described first segment data is sent to and strengthens in segmentation transmit queue, described second segment data is sent in described real-time segmentation transmit queue, under the prerequisite that described second segment data of preferential guarantee sends in time, described first segment data and described second segment data are sent to server, for described server according to described second segment data for user provides real-time data service, and apply described first segment data with corresponding described second segment data for user provides non-real-time data business, achieve and data to be encoded are carried out the segment data that coding obtains different quality, the low-quality segment data of preferential transmission, to ensure the real-time continuous of data service, and then send high-quality segment data, and use high-quality segment data to provide non-real-time data business for user, thus make as when user provides non-real-time data business, user can obtain high-quality segment data, namely high-quality video or audio experience is obtained.

Accompanying drawing explanation

In order to be illustrated more clearly in the embodiment of the present invention or technical scheme of the prior art, be briefly described to the accompanying drawing used required in embodiment or description of the prior art below, apparently, accompanying drawing in the following describes is some embodiments of the present invention, for those of ordinary skill in the art, under the prerequisite not paying creative work, other accompanying drawing can also be obtained according to these accompanying drawings.

Fig. 1 be in prior art data flow code check with the adjustment situation schematic diagram of bandwidth;

Fig. 2 is the flow chart of the sending method embodiment one of multi-medium data of the present invention;

Fig. 3 is the flow chart of the method for reseptance embodiment two of multi-medium data of the present invention;

Fig. 4 is the flow chart of the method for reseptance mode one of multi-medium data of the present invention;

Fig. 5 is the flow chart of the method for reseptance mode two of multi-medium data of the present invention;

Fig. 6 is the flow chart of the method for reseptance mode three of multi-medium data of the present invention;

Fig. 7 is the flow chart of the method for reseptance mode four of multi-medium data of the present invention;

Fig. 8 is the flow chart of the method for sending and receiving embodiment three of multi-medium data of the present invention;

Fig. 9 is the flow chart of time-sharing send segment data specific implementation (1) of the present invention;

Figure 10 is the flow chart of time-sharing send segment data specific implementation (2) of the present invention;

Figure 11 is the flow chart that the present invention distributes bandwidth transmission segment data specific implementation (3);

Figure 12 is the flow chart that the present invention distributes bandwidth transmission segment data specific implementation (4);

Figure 13 is the structural representation of terminal equipment embodiment one of the present invention;

Figure 14 is the structural representation of terminal equipment embodiment two of the present invention;

Figure 15 is coding under heterogeneous networks bandwidth situation of the present invention and the situation of transmission schematic diagram;

Figure 16 is the structural representation of server example one of the present invention;

Figure 17 is the structural representation of multimedia data processing system embodiment one of the present invention.

Embodiment

For making the object of the embodiment of the present invention, technical scheme and advantage clearly, below in conjunction with the accompanying drawing in the embodiment of the present invention, technical scheme in the embodiment of the present invention is clearly and completely described, obviously, described embodiment is the present invention's part embodiment, instead of whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art, not making the every other embodiment obtained under creative work prerequisite, belong to the scope of protection of the invention.

The quality of segment data involved in the embodiment of the present invention can comprise subjective quality and the objective quality of segment data, wherein subjective quality comprises average suggestion value (Mean Opinion Score, be called for short MOS) and Quality of experience (Quality of Entertainment, be called for short QoE), objective quality comprises mean square deviation (Mean Square Error, be called for short MSE), Y-PSNR (Peak Signal to Noise Ratio, be called for short PSNR) and structural similarity (Structural Similarity is called for short SSIM) etc.

The factor affecting segment data quality height can include but not limited to code check.

Involved in the embodiment of the present invention coded system that coding can adopt advanced video coding (Advanced Video Coding is called for short AVC) or scalable video encoder (Scalable VideoCoding is called for short SVC) is carried out to data to be encoded.Particularly, adopt the video coding and decoding technology of AVC to carry out coding generation first segment data and the second segment data respectively to data to be encoded, the first segment data and the second segment data are have identical content but the segment data of different quality; Also the video coding and decoding technology of SVC can be adopted to encode to data to be encoded, produce layer data based on the second segment data, and then produce the first segment data as enhancement data, first segment data and the second segment data have dependence, and wherein by basal layer code check, the code check of the first segment data can add that enhancement layer code check obtains.3rd segment data also can be adopt AVC technology to carry out encoding and obtaining.

Fig. 2 is the flow chart of the sending method embodiment one of multi-medium data of the present invention, and the executive agent of the present embodiment is terminal equipment, and as shown in Figure 2, the method for the present embodiment can comprise:

Step 201, acquisition coded reference information, know that described coded reference information meets pre-conditioned if judge, then encode to data to be encoded and obtain the first segment data and the second segment data respectively, the quality of wherein said first segment data is higher than the quality of described second segment data.

Wherein, described data to be encoded can be voice data also can be video data.Described first segment data and described second segment data are the segment data with same media content different quality.Described first segment data and the second segment data can be carry out absolute coding acquisition to described data to be encoded respectively, also can be after first described second segment data of coding acquisition being carried out to described data to be encoded, and then described first segment data of acquisition of encoding further on the second segment data basis, by layer data based on the second segment data, first segment data is as enhancement data, and by basal layer code check, the code check of the first segment data can add that enhancement layer code check obtains, therefore two segment datas that described first segment data and described second segment data can be separate, also can be two segment datas having dependence.

Step 202, described first segment data is sent to strengthens in segmentation transmit queue, described second segment data is sent in described real-time segmentation transmit queue.

Wherein, described enhancing segmentation transmit queue is used for the first segment data described in buffer memory, and described real-time segmentation transmit queue is used for buffer memory second segment data.Namely data to be encoded are encoded after generation first segment data and the second segment data, described enhancing segmentation transmit queue is for storing higher described first segment data of quality, and described real-time segmentation transmit queue is for storing lower described second segment data of quality.The segment data stored in described enhancing segmentation transmit queue and described real-time segmentation transmit queue is sent to server, wherein, the second segment data stored in preferential transmission real-time segmentation transmit queue.

Step 203, preferential ensure the prerequisite that described second segment data sends in time under, described first segment data and described second segment data are sent to server, for described server according to described second segment data for user provides real-time data service, and apply described first segment data and corresponding described second segment data for user non-real-time data business be provided.

Wherein, in step 203, described under the prerequisite that preferentially described second segment data of guarantee sends in time, described first segment data and described second segment data are sent to server, can include but not limited to: send described second segment data to described server, within the free time sending described second segment data or after sending described second segment data or utilize and send the remaining network bandwidth of described second segment data, send described first segment data to described server.

Wherein, described first segment data of described application and corresponding described second segment data provide non-real-time data business for user, can adopt in the following two kinds mode according to the difference of coded system wherein a kind of: (1) when adopt the coded system of AVC to data to be encoded encode obtain the first segment data and the second segment data respectively time, then apply described first segment data and replace corresponding described second segment data to provide non-real-time data business for user; (2) when adopting the coded system of SVC to carry out coding acquisition first segment data and the second segment data to data to be encoded, then described first segment data and corresponding described second segment data are carried out mergings generation the 4th segment data, apply described 4th segment data and replace described second segment data of correspondence to provide non-real-time data business for user.

Further, in step 201, know that described coded reference information does not meet pre-conditioned if judge, then coding carried out to data to be encoded and only obtain the 3rd segment data, described 3rd segment data higher than the code check of described second segment data, and is sent in described real-time segmentation transmit queue by the code check of wherein said 3rd segment data; Under the prerequisite that described 3rd segment data of preferential guarantee sends in time, described first segment data and described 3rd segment data are sent to server, for described server according to described 3rd segment data for user provides real-time data service, and apply described first segment data for user non-real-time data business be provided.

Wherein, described under the prerequisite that preferentially described 3rd segment data of guarantee sends in time, described first segment data and described 3rd segment data are sent to server, can include but not limited to: send described 3rd segment data to described server, within the free time sending described 3rd segment data or after sending described 3rd segment data or utilize and send the remaining network bandwidth of described 3rd segment data, send described first segment data to described server.

Namely when terminal equipment sends segment data to server, segment data in the described real-time segmentation transmit queue of preferential transmission, within the free time sending the segment data in real-time segmentation transmit queue or at the segment data sent in described real-time segmentation transmit queue or the network bandwidth utilizing the segment data sent in described real-time segmentation transmit queue remaining, send the segment data strengthened in segmentation transmit queue to described server.

Above-mentioned coded reference information can be the transmission delay that in real-time segmentation transmit queue, current data to be sent are corresponding, is correspondingly describedly pre-conditionedly greater than the first predetermined threshold value for described transmission delay; Or described coded reference information is the current network bandwidth, be correspondingly describedly pre-conditionedly less than the second predetermined threshold value for the described network bandwidth; Or, described coded reference information is the data volume in current real-time segmentation transmit queue, correspondingly describedly pre-conditionedly be greater than the 3rd predetermined threshold value for the data volume in described current real-time segmentation transmit queue, wherein said data volume is the total amount of data size of all segment datas to be sent in current real-time segmentation transmit queue; Or described coded reference information is the segment data number in current real-time segmentation transmit queue, be correspondingly describedly pre-conditionedly greater than the 4th predetermined threshold value for the segmentation number in described current real-time segmentation transmit queue.Wherein, described transmission delay is the total amount of data/network bandwidth of current data to be sent in described real-time segmentation transmit queue, or can be playing duration corresponding to the total amount of data of current data to be sent in described real-time segmentation transmit queue.

Further, generate the descriptor that described first segment data, described second segment data and described 3rd segment data are corresponding, and send to described server.

The present embodiment, by obtaining coded reference information, know that described coded reference information meets pre-conditioned if judge, then data to be encoded encoded and obtain the first segment data and the second segment data respectively, the quality of wherein said first segment data is higher than the quality of described second segment data, described first segment data is sent to and strengthens in segmentation transmit queue, described second segment data is sent in described real-time segmentation transmit queue, under the prerequisite that described second segment data of preferential guarantee sends in time, described first segment data and described second segment data are sent to server, for described server according to described second segment data for user provides real-time data service, and apply described first segment data with corresponding described second segment data for user provides non-real-time data business, achieve and data to be encoded are carried out the segment data that coding obtains different quality, the low-quality segment data of preferential transmission, to ensure the real-time continuous of data service, and then send high-quality segment data, and use high-quality segment data to provide non-real-time data business for user, thus make as when user provides non-real-time data business, user can obtain high-quality segment data, namely high-quality video or audio experience is obtained.

Fig. 3 is the flow chart of the method for reseptance embodiment two of multi-medium data of the present invention, and the executive agent of the present embodiment is server, and as shown in Figure 3, the method for the present embodiment can comprise:

The first segment data that step 301, receiving terminal send and the second segment data, the quality of wherein said first segment data is higher than the quality of described second segment data; Described first segment data and described second segment data be described terminal in acquisition coded reference information and judge to know described coded reference information meet pre-conditioned after, carrying out encoding to data to be encoded obtains; Wherein, described first segment data is sent by the enhancing segmentation transmit queue of described terminal, and described second segment data is sent by the real-time segmentation transmit queue of described terminal; Described first segment data is preferentially ensureing to send under the prerequisite that described second segment data sends in time.

Step 302, according to described second segment data for user provides real-time data service, and apply described first segment data and corresponding described second segment data for user non-real-time data business be provided.

Particularly, described first segment data of described application and corresponding described second segment data provide non-real-time data business for user, can adopt in the following two kinds mode according to the difference of coded system wherein a kind of: (1) when adopt the coded system of AVC to data to be encoded encode obtain the first segment data and the second segment data respectively time, then apply described first segment data and replace corresponding described second segment data to provide non-real-time data business for user; (2) when adopting the coded system of SVC to carry out coding acquisition first segment data and the second segment data to data to be encoded, then described first segment data and corresponding described second segment data are carried out mergings generation the 4th segment data, apply described 4th segment data and replace described second segment data of correspondence to provide non-real-time data business for user.

Alternatively, receive the 3rd segment data that described terminal sends, the code check of wherein said 3rd segment data is higher than the code check of described second segment data, and described 3rd segment data is stored in the real-time segmentation transmit queue of described terminal; Described 3rd segment data be described terminal judgement know described coded reference information do not meet pre-conditioned after, carrying out encoding to data to be encoded obtains.

The present embodiment, the first segment data sent by receiving terminal and the second segment data, the quality of wherein said first segment data is higher than the quality of described second segment data, described first segment data and described second segment data be described terminal in acquisition coded reference information and judge to know described coded reference information meet pre-conditioned after, carrying out encoding to data to be encoded obtains, wherein, described first segment data is sent by the enhancing segmentation transmit queue of described terminal, and described second segment data is sent by the real-time segmentation transmit queue of described terminal, described first segment data is preferentially ensureing to send under the prerequisite that described second segment data sends in time, according to described second segment data for user provides real-time data service, and apply described first segment data with corresponding described second segment data for user provides non-real-time data business, namely by the segment data with different quality that receiving terminal sends, under preferential guarantee receives the prerequisite of low-quality segment data, receive high-quality segment data, and use high-quality segment data to provide non-real-time data business for user, thus realize under the prerequisite of the real-time continuous ensureing data service, further by the high-quality segment data of reception, make as when user provides non-real-time data business, user can obtain high-quality segment data, thus obtain high-quality video or audio experience.

Fig. 4 is the flow chart of the method for reseptance mode one of multi-medium data of the present invention, Fig. 5 is the flow chart of the method for reseptance mode two of multi-medium data of the present invention, Fig. 6 is the flow chart of the method for reseptance mode three of multi-medium data of the present invention, and Fig. 7 is the flow chart of the method for reseptance mode four of multi-medium data of the present invention.Concrete, middle server embodiment illustrated in fig. 3 receives segment data can different implementations, as Figure 4-Figure 7.Mode one, as shown in Figure 4, S401: receive segment data, wherein, segment data can be the first segment data, the second segment data or the 3rd segment data; S402: the descriptor that the segment data received by renewal is corresponding, wherein this descriptor is the descriptor of carrying in segment data.Mode two, as shown in Figure 5, S501: receive segment data, wherein, segment data can be the first segment data, the second segment data or the 3rd segment data; S502: judge whether the segment data received is the first segment data, if then perform S503; S503: use the first segment data to replace the second corresponding segment data.Mode three, as shown in Figure 6, S601: receive segment data, wherein, segment data can be the first segment data, the second segment data or the 3rd segment data; ; S602: judge whether the segment data received is the first segment data, if then perform S603; S603: use the first segment data to replace the second corresponding segment data; S604: the descriptor that the segment data received by renewal is corresponding, wherein this descriptor is the descriptor of carrying in segment data.Mode four, be with the difference of mode three, descriptor corresponding to the segment data in mode four is independently file or message, S701: receive segment data and descriptor file, wherein, segment data can be the first segment data, the second segment data or the 3rd segment data; S702: judge whether the segment data received is the first segment data, if then perform S703; S703: use the first segment data to replace the second corresponding segment data; S704: the descriptor file that the segment data received by renewal is corresponding, wherein this descriptor file is send to server by terminal separately.

The first segment data, the second segment data and the 3rd segment data involved in above-mentioned several implementation all adopt the coded system of AVC to obtain.

Further, the step upgrading descriptor in above-mentioned several implementation also can select other modes, such as, the descriptor that server receives has consistent template, or when having consistent feature or can obtain descriptor corresponding to segment data according to certain rule, server does not need the letter upgrading segment data to describe breath.Use the first segment data to replace the second corresponding segment data in above-mentioned several implementation and also can select other modes, such as, server can retain low-quality second segment data of real-time reception and high-quality first segment data simultaneously, comprises the information of low-quality second segmentation and high-quality first segmentation in the descriptor of segment data.

Fig. 8 is the flow chart of the method for sending and receiving embodiment three of multi-medium data of the present invention, and as shown in Figure 3, the method for the present embodiment can comprise:

Step 801, terminal obtain the former data of audio frequency and video, carry out compression coding to data to be encoded, produce segment data.

Wherein, the former data of audio-video document can be divided into the data to be encoded of identical duration or different duration, carry out compression coding produce segment data to each data to be encoded.

Wherein, when initially producing the segment data after compression coding, generation the 3rd segment data of can only encoding, corresponding first fragment data of coding generation while generation second segment data of also can encoding.

The 3rd segment data involved by the present embodiment is the segment data of quality higher than the second segment data, first segment data is the segment data of quality higher than the second segment data, namely the 3rd segment data and the first segment data are high-quality segment data, and the second segment data is quality lower than the low-quality segment data of the first segment data and the 3rd segment data.

If step 802 is only encoded, generation the 3rd segment data, then put into real-time segmentation queue by the 3rd segment data; If coding produces corresponding high-quality first segment data while coding generation second segment data, then the second segment data is put into real-time segmentation transmit queue, high-quality first segment data is accordingly put into and strengthens segmentation transmit queue.

Step 803, to server send segment data, preferentially the segment data in real-time segmentation transmit queue is sent to server.

Particularly, if only produce the 3rd segment data, then the 3rd segment data is put into real-time segmentation transmit queue, and the 3rd segment data is sent to server; If not only produce the second segment data but also produce the first segment data, then the second segment data is put into real-time segmentation transmit queue, the first segmentation put into and strengthen segmentation transmit queue, terminal preferentially sends the second segment data in real-time segmentation transmit queue.Within the free time of transmission second segment data or after sending the second segment data or the network bandwidth utilizing transmission second segment data remaining, send the first segment data strengthened in segmentation transmit queue to server, if the second segment data in real-time segmentation transmit queue and the first segment data strengthened in segmentation transmit queue do not send, and now terminal generation the 3rd segment data puts into real-time segmentation transmit queue, then terminal preferentially sends the second segment data in real-time segmentation transmit queue and the 3rd segment data, when in the free time sending the segment data in real-time segmentation transmit queue, or after sending the segment data in real-time segmentation transmit queue, or utilize the network bandwidth that the segment data sent in real-time segmentation transmit queue is remaining, the first segment data strengthened in segmentation transmit queue is sent to server.

Namely the segment data in preferential guarantee real-time segmentation transmit queue sends to server in time, for server for user provides real-time data service, the segment data strengthened in segmentation transmit queue is sent, for server for user provides non real-time data service under the prerequisite that the segment data ensured in real-time segmentation transmit queue sends in time.

Step 804, acquisition coded reference information, judge whether this coded reference information meets pre-conditioned, if meet pre-conditioned, perform step 805, if do not meet, perform step 806.

Wherein, the triggering mode of step 804 determining step can have multitude of different ways, particularly, it can be time triggered, namely every primary network of inspection at regular intervals performs this determining step, also can be that event triggers, trigger after namely there is a certain event, such as, when having data to send, Sampling network performs this determining step.

Wherein, coded reference information can be multiple different information, particularly, this coded reference information can be the transmission delay that in real-time segmentation transmit queue, current data to be sent are corresponding, and the correspondingly corresponding with it pre-conditioned transmission delay that is is greater than the first predetermined threshold value.Namely judge whether the transmission delay that in real-time segmentation transmit queue, current data to be sent are corresponding is greater than the first predetermined threshold value, if be greater than, meet pre-conditioned execution step 805, if be not more than, do not meet and then perform step 806.Wherein, transmission delay can be the total amount of data/network bandwidth of current data to be sent in real-time segmentation transmit queue, or also can be playing duration corresponding to the total amount of data of current data to be sent in real-time segmentation transmit queue.Wherein, the first predetermined threshold value can be arranged according to demand, and such as can arrange the first predetermined threshold value is 5 seconds.

This coded reference information also can be the current network bandwidth, and the correspondingly corresponding with it pre-conditioned network bandwidth that is is less than the second predetermined threshold value.Namely judge whether the network bandwidth is less than the second predetermined threshold value, if be less than, meet pre-conditioned execution step 805, if be not less than, do not meet and then perform step 806.Wherein, the second predetermined threshold value can be arranged according to demand, and such as can arrange the second predetermined threshold value is 500kbps.

This coded reference information can also be the data volume in current real-time segmentation transmit queue, this data volume is the total amount of data size of all segment datas to be sent in current real-time segmentation transmit queue, and the correspondingly corresponding pre-conditioned data volume be in current real-time segmentation transmit queue is greater than the 3rd predetermined threshold value.Namely judge whether the data volume in current real-time segmentation transmit queue is greater than the 3rd predetermined threshold value, if be greater than, meet pre-conditioned execution step 805, if be not less than, do not meet and then perform step 806.Wherein, the 3rd predetermined threshold value can be arranged according to demand, and such as can arrange the 3rd predetermined threshold value is 2500kb.

This coded reference information can also be the segment data number in current real-time segmentation transmit queue, and the correspondingly corresponding pre-conditioned segment data number be in current real-time segmentation transmit queue is greater than the 4th predetermined threshold value.Namely judge whether the segment data number in current real-time segmentation transmit queue is greater than the 4th predetermined threshold value, if be greater than, meet pre-conditioned execution step 805, if be not more than, do not meet and then perform step 806.Wherein the 4th predetermined threshold value can be arranged as required, such as, can equipment the 4th predetermined threshold value be 40.

Step 805, encoding and obtain the first segment data and the second segment data respectively to data to be encoded, wherein the quality of this first segment data is higher than the quality of this second segment data; And this first segment data is put into enhancing segmentation transmit queue, this second segment data is put into real-time segmentation transmit queue, and performs step 803.

Step 806, data to be encoded carried out to coding and only obtain the 3rd segment data, the 3rd segment data higher than the quality of the second segment data, and is sent in real-time segmentation transmit queue by the quality of the 3rd segment data, and performs step 803.

The segment data that step 807, server receiving terminal send, if receive again first segment data corresponding with this second segment data after receiving the second segment data, then by the first segment data and the second corresponding with it segment data for user provides non-real-time data business.

Wherein, by the first segment data and the second corresponding with it segment data for user provides non-real-time data business, specifically can adopt following two kinds of modes, (1) when the first segment data and the second segment data be obtained by AVC coded system time, then use the first segment data to substitute the second corresponding segment data for user and non-real-time data business be provided; (2) when the first segment data and the second segment data are obtained by SVC coded system, there is dependence, then use the first segment data and the second segment data to merge generation the 4th segment data, use the 4th segment data to replace the second corresponding segment data to provide non-real-time data business for user.

Generate the descriptor corresponding with each segment data when generating the first segment data, the second segment data and the 3rd segment data in above-mentioned steps, and send to server.Wherein, this descriptor can comprise initial time, time duration, the code check of segmentation, the quality information of segmentation, the resolution of video, any one information or multiple information combination such as sample rate, language of audio frequency of segmentation; And this descriptor can directly be encapsulated in each segment data, sends to server together with segment data, independently file or message also can be adopted to carry this descriptor and to send to server.First segment data and second segment data with the different code check of identical content have the method that can distinguish, such as, in the first segment data and the second segment data, carry mark, or carry differentiation information in independently file or message.

The present embodiment, after data to be encoded being carried out compression coding and sending to server, obtain coded reference information, judge whether this coded reference information meets pre-conditioned, namely when coded reference information is the transmission delay that in real-time segmentation transmit queue, current data to be sent are corresponding, then correspondingly correspondingly with it pre-conditionedly be greater than the first predetermined threshold value for transmission delay, when coded reference information is current network bandwidth, then correspondingly correspondingly with it pre-conditionedly be less than the second predetermined threshold value for the network bandwidth, when coded reference information is the data volume in current real-time segmentation transmit queue, correspondingly describedly pre-conditionedly be greater than the 3rd predetermined threshold value for the data volume in described current real-time segmentation transmit queue, when the segment data number that coded reference information is in current real-time segmentation transmit queue, correspondingly describedly pre-conditionedly be greater than the 4th predetermined threshold value for the segmentation number in described current real-time segmentation transmit queue, namely by obtaining the coded reference information of reflection network bandwidth situation, when coded reference information is any one, when judging that this coded reference information meets pre-conditioned, show that current network bandwidth declines, thus terminal self-adaptative adjustment coded system, first segment data of outputting high quality while exporting low-quality second segment data, second segment data can ensure that data can be sent to server in real time, thus guarantee that user obtains data from server in real time, ensure that the continuity listening to viewing in real time, under the prerequisite of preferential transmission the second segment data, utilize free time or remaining bandwidth that the first segment data is sent to server, thus make server by high-quality first segment data received for user provides non-real-time data business, thus for the non real-time user listening to viewing, because the first segment data is sent to server when the network bandwidth improves by terminal, so non real time user can be listened to and watch that to listen to the quality of data of watching than active user before high, further, if non real time user listens to the content that viewing terminal uploads after terminal sends all segment datas, so non real time user can be listened to and watch audiovisual content the best in quality.

Fig. 9 is the flow chart of time-sharing send segment data specific implementation (1) of the present invention, Figure 10 is the flow chart of time-sharing send segment data specific implementation (2) of the present invention, Figure 11 is the flow chart that the present invention distributes bandwidth transmission segment data specific implementation (3), and Figure 12 is the flow chart that the present invention distributes bandwidth transmission segment data specific implementation (4).

Particularly, in above-described embodiment, step 803 can choose following two kinds of send modes.Mode one, time-sharing send segment data; Mode two, distributes bandwidth and sends segment data.

Wherein mode one time-sharing send segment data can comprise two kinds of specific implementations, specific implementation (1) and specific implementation (2), and specific implementation (1) as shown in Figure 9, comprising:

S901: whether be empty, then perform S902 if not empty, then perform S903 if it is empty if detecting real-time segmentation transmit queue.

S902: send the segment data in real-time segmentation transmit queue.

S903: whether be empty, then perform S904 if not empty, then return S501 if it is empty if detecting enhancing segmentation transmit queue.

S904: send the segment data strengthened in segmentation transmit queue.

S905: segment data is sent completely, returns S901.

Specific implementation (2) as shown in Figure 10, comprising:

S1001: whether be empty, then perform S1002 if not empty, then perform S1003 if it is empty if detecting real-time segmentation transmit queue.

S1002: detect the segment data whether sending and strengthening in segmentation transmit queue, if then perform S1004, then perform S1005 if not.

S1003: whether be empty, then perform S1006 if not empty, then return S1001 if it is empty if detecting enhancing segmentation transmit queue.

S1004: suspend the segment data strengthened in segmentation transmit queue and send, and perform S1005.

S1005: send the segment data in real-time segmentation transmit queue.

S1006: send the segment data strengthened in segmentation transmit queue.

S1007: segment data is sent completely (event triggering) or segment data is not sent completely, but arrive detection time (time triggered), return S1001.

Mode two distributes bandwidth transmission segment data can comprise two kinds of specific implementations, specific implementation (3) and specific implementation (4), and specific implementation (3) as shown in figure 11, comprising:

S1101: whether be empty, then perform S1102 if not empty, then perform S1103 if it is empty if detecting real-time segmentation transmit queue.

S1102: send the segment data in real-time segmentation transmit queue, and Sampling network bandwidth, comparing cell bandwidth sum segment data code check can be passed through, namely judge whether the network bandwidth is greater than segment data code check, if the network bandwidth is greater than segment data code check, then perform S1104, otherwise, perform S1105.

S1103: whether be empty, then perform S1106 if not empty, then return S1101 if it is empty if detecting enhancing segmentation transmit queue.

S1104: utilize idle network bandwidth to send the segment data strengthened in segmentation transmit queue, wherein, idle network bandwidth≤network bandwidth-segment data code check.

S1105: obtain current network bandwidth, judge whether this network bandwidth meets pre-conditioned.

Wherein, the operation that S1105 performs can with reference to step 804, and the step after performing step 804.

S1106: send the segment data strengthened in segmentation transmit queue.

S1107: segment data is sent completely.

Specific implementation (4) as shown in figure 12, comprising:

S1201: whether be empty, then perform S1202 if not empty, then perform S1203 if it is empty if detecting real-time segmentation transmit queue.

S1202: send the segment data in real-time segmentation transmit queue, and Sampling network bandwidth, comparing cell bandwidth sum segment data code check can be passed through, namely judge whether the network bandwidth is greater than segment data code check, if the network bandwidth is greater than segment data code check, then perform S1204, otherwise, perform S1205.

S1203: whether be empty, then perform S1206 if not empty, then return S1201 if it is empty if detecting enhancing segmentation transmit queue.

S1204: utilize idle network bandwidth to send the segment data strengthened in segmentation transmit queue, wherein, idle network bandwidth≤network bandwidth-segment data code check.

S1205: if there is the segment data strengthened in segmentation transmit queue to send, then suspend the transmission of this segment data, perform S1207.

S1206: send the segment data strengthened in segmentation transmit queue, performs S1208.

S1207: obtain current network bandwidth, judge whether this network bandwidth meets pre-conditioned.

Wherein, S1207 concrete operations can with reference to step 804, and the step after performing step 804.

S1208: segment data is sent completely (event triggering) or segment data is not sent completely, but arrive detection time (time triggered), return S1201.

Sending module in terminal equipment can perform the procedure of Fig. 9-Figure 12.

Figure 13 is the structural representation of terminal equipment embodiment one of the present invention, and as shown in figure 13, the device of this enforcement can comprise: acquisition module 131, memory module 132 and sending module 133.Wherein, acquisition module 131 is for obtaining coded reference information, know that described coded reference information meets pre-conditioned if judge, then encode to data to be encoded and obtain the first segment data and the second segment data respectively, the quality of wherein said first segment data is higher than the quality of described second segment data; Memory module 132, strengthening in segmentation transmit queue for described first segment data being sent to, described second segment data being sent in described real-time segmentation transmit queue; Sending module 133, for preferentially ensureing under the prerequisite that described second segment data sends in time, described first segment data and described second segment data are sent to server, for described server according to described second segment data for user provides real-time data service, and apply described first segment data and replace corresponding described second segment data to provide non-real-time data business for user.

Wherein, described sending module 133 specifically can comprise: the first transmitting element 1331, for sending described second segment data to described server; Second transmitting element 1332, for within the free time sending described second segment data or after sending described second segment data or utilize and send the remaining network bandwidth of described second segment data, send described first segment data to described server.

Wherein, described first segment data of described application and corresponding described second segment data provide non-real-time data business for user, are specifically as follows: apply described first segment data and replace corresponding described second segment data to provide non-real-time data business for user; Or, described first segment data and corresponding described second segment data are carried out mergings generation the 4th segment data, apply described 4th segment data and replace described second segment data of correspondence to provide non-real-time data business for user.

Further, if for judging, described acquisition module 131 also knows that described coded reference information does not meet pre-conditioned, then coding carried out to data to be encoded and only obtain the 3rd segment data, described 3rd segment data higher than the code check of described second segment data, and is sent in described real-time segmentation transmit queue to send to described server by the code check of wherein said 3rd segment data; Described sending module 133 also for: preferential ensure the prerequisite that described 3rd segment data sends in time under, described first segment data and described 3rd segment data are sent to server, for described server according to described 3rd segment data for user provides real-time data service, and apply described first segment data for user non-real-time data business be provided.

Further, described sending module 133 specifically can comprise: the 3rd transmitting element 1333, for sending described 3rd segment data to described server; 4th transmitting element 1334, for within the free time sending described 3rd segment data or after sending described 3rd segment data or utilize and send the remaining network bandwidth of described 3rd segment data, send described first segment data to described server.

Described coded reference information is the transmission delay that in described real-time segmentation transmit queue, current data to be sent are corresponding, is describedly accordingly pre-conditionedly greater than the first predetermined threshold value for described transmission delay; Or described coded reference information is the current network bandwidth, be correspondingly describedly pre-conditionedly less than the second predetermined threshold value for the described network bandwidth; Or described coded reference information is the data volume in current real-time segmentation transmit queue, be correspondingly describedly pre-conditionedly greater than the 3rd predetermined threshold value for the data volume in described current real-time segmentation transmit queue; Or described coded reference information is the segment data number in current real-time segmentation transmit queue, be correspondingly describedly pre-conditionedly greater than the 4th predetermined threshold value for the segmentation number in described current real-time segmentation transmit queue.Described transmission delay is the total amount of data/network bandwidth of current data to be sent in described real-time segmentation transmit queue, or is playing duration corresponding to the total amount of data of current data to be sent in described real-time segmentation transmit queue.

Further, described terminal equipment also comprises processing module 134, for generating descriptor corresponding to described first segment data, described second segment data and described 3rd segment data, and sends to described server.

The device of the present embodiment, may be used for the technical scheme performing embodiment of the method shown in Fig. 2, it realizes principle and technique effect is similar, repeats no more herein.

Figure 14 is the structural representation of terminal equipment embodiment two of the present invention, and as shown in figure 14, the equipment of this enforcement comprises: the encoder 141 of communication connection mutually and sending module 142; Wherein, described sending module 142 comprises network bandwidth estimation module 1421, real-time segmentation transmit queue 1422 and strengthens segmentation transmit queue 1423; Wherein: described network bandwidth estimation module 1421, for obtaining coded reference information; Described encoder 141 for, if judge to know that described coded reference information meets pre-conditioned in described network bandwidth estimation module 1421, then encode to data to be encoded and obtain the first segment data and the second segment data respectively, the quality of wherein said first segment data is higher than the quality of described second segment data; Described real-time segmentation transmit queue 1422 for storing described second segment data, and sends to server; Described enhancing segmentation transmit queue 1423 is for preferentially ensureing under the prerequisite that described second segment data sends in time, described first segment data is sent to server, for described server according to described second segment data for user provides real-time data service, and apply described first segment data and corresponding described second segment data for user non-real-time data business be provided.

Wherein, described enhancing segmentation transmit queue 1423 specifically for, within the free time sending described second segment data or after sending described second segment data or utilize and send the remaining network bandwidth of described second segment data, send described first segment data to described server.

Described first segment data of described application and corresponding described second segment data provide non-real-time data business for user, are specifically as follows: apply described first segment data and replace corresponding described second segment data to provide non-real-time data business for user; Or, described first segment data and corresponding described second segment data are carried out mergings generation the 4th segment data, apply described 4th segment data and replace described second segment data of correspondence to provide non-real-time data business for user.

Further, described encoder 141 also for, if judge to know that described coded reference information does not meet pre-conditioned in described network bandwidth estimation module, then carry out coding to data to be encoded and only obtain the 3rd segment data, the code check of wherein said 3rd segment data is higher than the code check of described second segment data; Correspondingly, described real-time segmentation transmit queue also for storing described 3rd segment data, and sends to described server; Described enhancing segmentation transmit queue 1423 is also for preferentially ensureing under the prerequisite that described 3rd segment data sends in time, described first segment data is sent to server, for described server according to described 3rd segment data for user provides real-time data service, and apply described first segment data for user non-real-time data business be provided.

Described coded reference information is the transmission delay that in described real-time segmentation transmit queue, current data to be sent are corresponding, is correspondingly describedly pre-conditionedly greater than the first predetermined threshold value for described transmission delay; Or described coded reference information is the current network bandwidth, be correspondingly describedly pre-conditionedly less than the second predetermined threshold value for the described network bandwidth; Or described coded reference information is the data volume in current real-time segmentation transmit queue, be correspondingly describedly pre-conditionedly greater than the 3rd predetermined threshold value for the data volume in described current real-time segmentation transmit queue; Or described coded reference information is the segment data number in current real-time segmentation transmit queue, be correspondingly describedly pre-conditionedly greater than the 4th predetermined threshold value for the segmentation number in described current real-time segmentation transmit queue.

Described transmission delay is the total amount of data/network bandwidth of current data to be sent in described real-time segmentation transmit queue, or is playing duration corresponding to the total amount of data of current data to be sent in described real-time segmentation transmit queue.

The equipment of the present embodiment, can perform the technical scheme of embodiment of the method shown in Fig. 2, and it realizes principle and technique effect is similar, repeats no more herein.

Figure 15 is coding under heterogeneous networks bandwidth situation of the present invention and the situation of transmission schematic diagram.In order to technical scheme of the present invention is clearly described, when describing network bandwidth change with time diagram, encoder exports the transmission situation of segment data and segment data, as shown in figure 15, first, within the time period of t0 to t1, the high network bandwidth, because network bandwidth quality is better, then the encoder in terminal equipment exports the 3rd segment data, i.e. high-quality segment data, and the 3rd segment data is put into real-time segmentation transmit queue; In real time the 3rd segment data is sent by the real-time segmentation transmit queue in sending module.Afterwards, within the time period of t1 to t2, the network bandwidth declines, network bandwidth estimation module in sending module perceives the network bandwidth and declines, network bandwidth situation is fed back to encoder, now encoder exports two-way segment data, i.e. the first segment data (high-quality segment data) and the second segment data (low-quality segment data), and this first segment data is sent into enhancing segmentation transmit queue, this second segment data is sent into real-time segmentation transmit queue, and sending module preferentially sends the segment data in real-time segmentation transmit queue.After instant t 2, the network bandwidth is by returning to the high network bandwidth, network bandwidth estimation module in sending module perceives the network bandwidth and rises, network bandwidth situation is fed back to encoder, encoder adjustment coded system only exports the 3rd segment data (high-quality segment data), and the 3rd segment data is put into real-time segmentation transmit queue, the real-time segment data in real-time segmentation transmit queue being sent of sending module, sent the first segment data strengthened in segmentation transmit queue when the available free time.

Wherein, the first segment data strengthened in segmentation transmit queue is sent during the available free time, also other modes can be selected, such as, under the prerequisite that the segment data ensured in real-time segmentation transmit queue sends in time, utilize rest network bandwidth, the segment data strengthened in segmentation transmit queue is sent together with the segment data in real-time segmentation transmit queue.

Figure 16 is the structural representation of server example one of the present invention, and as shown in figure 16, the device of the present embodiment can comprise: receiver module 161 and processing module 162.Wherein, receiver module 161, the first segment data sent for receiving terminal and the second segment data, the quality of wherein said first segment data is higher than the quality of described second segment data; Described first segment data and described second segment data be described terminal in acquisition coded reference information and judge to know described coded reference information meet pre-conditioned after, carrying out encoding to data to be encoded obtains; Wherein, described first segment data is sent by the enhancing segmentation transmit queue of described terminal, and described second segment data is sent by the real-time segmentation transmit queue of described terminal; Described first segments is preferentially ensureing to send under the prerequisite that described second segment data sends in time; Processing module 162, for according to described second segment data for user provides real-time data service, and apply described first segment data and corresponding described second segment data for user non-real-time data business be provided.

Further, described receiver module 161 is also for receiving the 3rd segment data that described terminal sends, the code check of wherein said 3rd segment data is higher than the code check of described second segment data, and described 3rd segment data is stored in the real-time segmentation transmit queue of described terminal; Described 3rd segment data be described terminal judgement know described coded reference information do not meet pre-conditioned after, carrying out encoding to data to be encoded obtains.

The device of the present embodiment, may be used for the technical scheme performing embodiment of the method shown in Fig. 3, it realizes principle and technique effect is similar, repeats no more herein.

Figure 17 is the structural representation of multimedia data processing system embodiment one of the present invention, and as shown in figure 17, described multimedia data processing system comprises terminal equipment 171, server 172.

Wherein, terminal equipment 171 can adopt the structure of the device embodiment of Figure 13, and it can perform the technical scheme of its embodiment of the method accordingly, and it realizes principle and technique effect is similar, repeats no more herein.

Server 172 can adopt the structure of the device embodiment of Figure 16, and it can perform the technical scheme of its embodiment of the method accordingly, and it realizes principle and technique effect is similar, repeats no more herein.

In several embodiment provided by the present invention, should be understood that, disclosed apparatus and method, can realize by another way.Such as, device embodiment described above is only schematic, such as, the division of described unit, be only a kind of logic function to divide, actual can have other dividing mode when realizing, such as multiple unit or assembly can in conjunction with or another system can be integrated into, or some features can be ignored, or do not perform.Another point, shown or discussed coupling each other or direct-coupling or communication connection can be by some interfaces, and the indirect coupling of device or unit or communication connection can be electrical, machinery or other form.

The described unit illustrated as separating component or can may not be and physically separates, and the parts as unit display can be or may not be physical location, namely can be positioned at a place, or also can be distributed in multiple network element.Some or all of unit wherein can be selected according to the actual needs to realize the object of the present embodiment scheme.

In addition, each functional unit in each embodiment of the present invention can be integrated in a processing unit, also can be that the independent physics of unit exists, also can two or more unit in a unit integrated.Above-mentioned integrated unit both can adopt the form of hardware to realize, and the form that hardware also can be adopted to add SFU software functional unit realizes.

The above-mentioned integrated unit realized with the form of SFU software functional unit, can be stored in a computer read/write memory medium.Above-mentioned SFU software functional unit is stored in a storage medium, comprising some instructions in order to make a computer equipment (can be personal computer, server, or the network equipment etc.) or processor (processor) perform the part steps of method described in each embodiment of the present invention.And aforesaid storage medium comprises: USB flash disk, portable hard drive, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disc or CD etc. various can be program code stored medium.

Those skilled in the art can be well understood to, for convenience and simplicity of description, only be illustrated with the division of above-mentioned each functional module, in practical application, can distribute as required and by above-mentioned functions and be completed by different functional modules, internal structure by device is divided into different functional modules, to complete all or part of function described above.The specific works process of the device of foregoing description, with reference to the corresponding process in preceding method embodiment, can not repeat them here.

Last it is noted that above each embodiment is only in order to illustrate technical scheme of the present invention, be not intended to limit; Although with reference to foregoing embodiments to invention has been detailed description, those of ordinary skill in the art is to be understood that: it still can be modified to the technical scheme described in foregoing embodiments, or carries out equivalent replacement to wherein some or all of technical characteristic; And these amendments or replacement, do not make the essence of appropriate technical solution depart from the scope of various embodiments of the present invention technical scheme.

Claims

1. a sending method for multi-medium data, is characterized in that, comprising:

2. method according to claim 1, is characterized in that, described under the prerequisite that preferentially described second segment data of guarantee sends in time, described first segment data and described second segment data is sent to server, comprising:

Described second segment data is sent to described server;

3. method according to claim 1 and 2, is characterized in that, described first segment data of described application and corresponding described second segment data provide non-real-time data business for user, comprising:

4. method according to claim 1, is characterized in that, described method also comprises:

Know that described coded reference information does not meet pre-conditioned if judge, then coding carried out to data to be encoded and only obtain the 3rd segment data, described 3rd segment data higher than the code check of described second segment data, and is sent in described real-time segmentation transmit queue by the code check of wherein said 3rd segment data;

Under the prerequisite that described 3rd segment data of preferential guarantee sends in time, described first segment data and described 3rd segment data are sent to server, for described server according to described 3rd segment data for user provides real-time data service, and apply described first segment data for user non-real-time data business be provided.

5. method according to claim 4, is characterized in that, described under the prerequisite that preferentially described 3rd segment data of guarantee sends in time, described first segment data and described 3rd segment data is sent to server, comprising:

Described 3rd segment data is sent to described server;

6. the method according to claim 1 or 4, it is characterized in that, described coded reference information is the transmission delay that in described real-time segmentation transmit queue, current data to be sent are corresponding, is correspondingly describedly pre-conditionedly greater than the first predetermined threshold value for described transmission delay; Or,

7. method according to claim 6, it is characterized in that, described transmission delay is the total amount of data/network bandwidth of current data to be sent in described real-time segmentation transmit queue, or is playing duration corresponding to the total amount of data of current data to be sent in described real-time segmentation transmit queue.

8. method according to claim 4, is characterized in that, described method also comprises:

9. a method of reseptance for multi-medium data, is characterized in that, comprising:

The first segment data that receiving terminal sends and the second segment data, the quality of wherein said first segment data is higher than the quality of described second segment data; Described first segment data and described second segment data be described terminal in acquisition coded reference information and judge to know described coded reference information meet pre-conditioned after, carrying out encoding to data to be encoded obtains; Wherein, described first segment data is sent by the enhancing segmentation transmit queue of described terminal, and described second segment data is sent by the real-time segmentation transmit queue of described terminal; Described first segments is preferentially ensureing to send under the prerequisite that described second segment data sends in time;

10. method according to claim 9, is characterized in that, described first segment data of described application and corresponding described second segment data provide non-real-time data business for user, comprising:

11. methods according to claim 10, is characterized in that, described method also comprises:

12. 1 kinds of terminal equipments, is characterized in that, comprising:

Sending module, for preferentially ensureing under the prerequisite that described second segment data sends in time, described first segment data and described second segment data are sent to server, for described server according to described second segment data for user provides real-time data service, and apply described first segment data and corresponding described second segment data for user non-real-time data business be provided.

13. terminal equipments according to claim 12, is characterized in that, described sending module specifically comprises:

14. terminal equipments according to claim 12 or 13, is characterized in that, described first segment data of described application and corresponding described second segment data provide non-real-time data business for user, comprising:

15. terminal equipments according to claim 12, it is characterized in that, described acquisition module also for: if judge know that described coded reference information does not meet pre-conditioned, then coding carried out to data to be encoded and only obtain the 3rd segment data, described 3rd segment data higher than the code check of described second segment data, and is sent in described real-time segmentation transmit queue to send to described server by the code check of wherein said 3rd segment data;

16. terminal equipments according to claim 15, is characterized in that, described sending module specifically comprises:

17. terminal equipments according to claim 12 or 15, it is characterized in that, described coded reference information is the transmission delay that in described real-time segmentation transmit queue, current data to be sent are corresponding, is describedly accordingly pre-conditionedly greater than the first predetermined threshold value for described transmission delay; Or,

18. terminal equipments according to claim 17, it is characterized in that, described transmission delay is the total amount of data/network bandwidth of current data to be sent in described real-time segmentation transmit queue, or is playing duration corresponding to the total amount of data of current data to be sent in described real-time segmentation transmit queue.

19. terminal equipments according to claim 15, is characterized in that, also comprise:

20. 1 kinds of terminal equipments, is characterized in that, comprising: the encoder of communication connection mutually and sending module; Wherein, described sending module comprises network bandwidth estimation module, real-time segmentation transmit queue and strengthens segmentation transmit queue; Wherein:

21. terminal equipments according to claim 20, it is characterized in that, described enhancing segmentation transmit queue specifically for, within the free time sending described second segment data or after sending described second segment data or utilize and send the remaining network bandwidth of described second segment data, send described first segment data to described server.

22. terminal equipments according to claim 20 or 21, is characterized in that, described first segment data of described application and corresponding described second segment data provide non-real-time data business for user, comprising:

23. terminal equipments according to claim 20, it is characterized in that, described encoder also for, if judge to know that described coded reference information does not meet pre-conditioned in described network bandwidth estimation module, then carry out coding to data to be encoded and only obtain the 3rd segment data, the code check of wherein said 3rd segment data is higher than the code check of described second segment data;

24. terminal equipments according to claim 23, it is characterized in that, described enhancing segmentation transmit queue also for, within the free time sending described 3rd segment data or after sending described 3rd segment data or utilize and send the remaining network bandwidth of described 3rd segment data, send described first segment data to described server.

25. terminal equipments according to claim 20 or 23, it is characterized in that, described coded reference information is the transmission delay that in described real-time segmentation transmit queue, current data to be sent are corresponding, is correspondingly describedly pre-conditionedly greater than the first predetermined threshold value for described transmission delay; Or,

26. terminal equipments according to claim 25, described transmission delay is the total amount of data/network bandwidth of current data to be sent in described real-time segmentation transmit queue, or is playing duration corresponding to the total amount of data of current data to be sent in described real-time segmentation transmit queue.

27. 1 kinds of servers, is characterized in that, comprising:

28. servers according to claim 27, is characterized in that, described first segment data of described application and corresponding described second segment data provide non-real-time data business for user, comprising:

29. servers according to claim 28, is characterized in that, described receiver module also for:

The transmitting system of 30. 1 kinds of multi-medium datas, is characterized in that, comprising:

Terminal equipment as described in any one of claim 12-19 and the server as described in any one of claim 27-29.