Summary of the invention
Technical problem to be solved by this invention is: the real-time separating treatment of a kind of audio/video flow and synchronous method and system are provided, while ensureing host computer and slave computer transmission data, take the less network bandwidth, and can guarantee the accurate synchronization that host computer and slave computer audio/video flow are play.
For solving the problems of the technologies described above, the technical solution used in the present invention is: the real-time separating treatment of a kind of audio frequency and video and synchronous method are provided, comprise:
S1: host computer obtains and demultiplexing source of media file, to obtain video flowing and audio stream;
S2: described host computer carries out video decode and video filtering to described video flowing, to obtain video playback data, and plays described video playback data;
S3: described host computer carries out transferring to slave computer after format conversion and compression coding to described audio stream;
S4: described slave computer calls decoder described audio stream is decoded and filtering, to obtain audio frequency played data, and plays described audio frequency played data;
S5: when being started, described video playback data and audio frequency played data carry out initial synchronization, and to carrying out synchronous correction processing in described video playback data and audio frequency played data playing process.
Compared with prior art, in method of the present invention, host computer carries out demultiplexing to source of media file and obtains, after video flowing and audio stream, flowing corresponding data by host computer displaying video, and audio stream is transferred to slave computer play, thereby realize the separating treatment of audio/video flow; And before audio stream is transferred to slave computer, audio stream is carried out to format conversion and compression coding, thereby greatly reduce the network bandwidth shared while transmitting data; In addition, when starting video playback data and audio frequency played data, the method carries out initial synchronization, and to carrying out synchronous correction processing in video playback data and audio frequency played data playing process, thereby guarantee host computer and slave computer accurate synchronization when displaying video, audio frequency respectively.
Correspondingly, the present invention also provides the real-time separating treatment of a kind of audio/video flow and synchronous system, comprises host computer and slave computer, and described host computer comprises:
Demultiplexing module, for obtaining and demultiplexing source of media file, to obtain video flowing and audio stream;
Video decode and filtration module, for described video flowing is carried out to video decode and video filtering, to obtain video playback data, and play described video playback data;
The first format converting module, for carrying out format conversion and compression coding to described audio stream;
The first transport module, for transferring to slave computer by the described audio stream after format conversion and compression coding;
Described slave computer comprises:
Audio decoder and filtration module, decode and filtering to described audio stream for calling decoder, to obtain audio frequency played data, and plays described audio frequency played data;
It is characterized in that, described host computer and slave computer include:
Synchronization module, carries out initial synchronization when described video playback data and audio frequency played data are started, and to carrying out synchronous correction processing in described video playback data and audio frequency played data playing process.
By following description also by reference to the accompanying drawings, it is more clear that the present invention will become, and these accompanying drawings are used for explaining embodiments of the invention.
Embodiment
With reference now to accompanying drawing, describe embodiments of the invention, in accompanying drawing, similarly element numbers represents similar element.
Please refer to Fig. 1, the invention provides the real-time separating treatment of a kind of audio frequency and video and synchronous method, comprising:
S1: host computer obtains and demultiplexing source of media file, to obtain video flowing and audio stream;
S2: host computer carries out video decode and video filtering to video flowing, to obtain video playback data, and displaying video played data;
S3: host computer carries out transferring to slave computer after format conversion and compression coding to audio stream;
S4: slave computer calls decoder audio stream is decoded and filtering, to obtain audio frequency played data, and plays described audio frequency played data;
S5: when being started, video playback data and audio frequency played data carry out initial synchronization, and to carrying out synchronous correction processing in video playback data and audio frequency played data playing process.
Compared with prior art, in method of the present invention, host computer carries out demultiplexing to source of media file and obtains, after video flowing and audio stream, flowing corresponding data by host computer displaying video, and audio stream is transferred to slave computer play, thereby realize the separating treatment of audio/video flow; And before audio stream is transferred to slave computer, audio stream is carried out to format conversion and compression coding, thereby greatly reduce the network bandwidth shared while transmitting data; In addition, when starting video playback data and audio frequency played data, the method carries out initial synchronization, and to carrying out synchronous correction processing in video playback data and audio frequency played data playing process, thereby guarantee host computer and slave computer accurate synchronization when displaying video, audio frequency respectively.
Please refer to again Fig. 2 and Fig. 3, in another embodiment of the present invention, can realize the real-time separating treatment to audio/video flow and synchronous in wireless image-sound system.It should be noted that, in the present embodiment, hardware device used mainly comprises: host computer and slave computer, connect by wireless Wi-Fi network between the two.Wherein, host computer is mobile intelligent terminal equipment, can be mobile phone or flat-panel devices based on Android, IOS or other system (not limiting to cell phone system).Slave computer is the Specialty Hi-Fi equipment that has carried professional audio processor.The decoding of audio frequency and video is to separate with slave computer at host computer respectively to realize, and asynchronous execution.Host computer is responsible for video pictures and is play, audio stream is sent to slave computer by network transmission module and plays (utilizing synchronization module to ensure the accurate synchronization of audio frequency and video), while slave computer is recorded user voice and is passed back to host computer, finally carries out audio frequency and video merging at host computer.
Particularly, the present embodiment comprises the steps:
S201: host computer obtains and demultiplexing source of media file, to obtain video flowing and audio stream.
S202: host computer carries out video decode and video filtering to video flowing, to obtain video playback data, and displaying video played data; Be specially, host computer calls corresponding Video Decoder according to the form of video flowing, and the video flowing that demultiplexing is obtained is decoded, and carries out the output of video pictures play at mobile terminal.
S203: host computer carries out transferring to slave computer after format conversion and compression coding to audio stream; Be specially, the audio stream obtaining is carried out to format conversion, adjust and become the form that is applicable to carrying out flow transmission, then arrive slave computer by Wi-Fi wireless network transmissions.Now the data of transmission are not original audio frequency PCM data, but the audio stream data of compressing that is encoded has reduced the bandwidth occupancy of Wi-Fi network.If the just voice data in current broadcasting moment of real-time Transmission, in the time that network congestion is unsettled, voice data cannot arrive slave computer in real time, so the mode that the present invention adopts is: the process of audio stream transmission is the service doing one's best, and the slave computer that is transferred in advance as much as possible audio stream data is preserved into file.
S204: slave computer calls decoder audio stream is decoded and filtering, to obtain audio frequency played data; Be specially, slave computer receives the audio stream data of host computer transmission, preserves into file, and the decoder that meanwhile audio stream is called to specific format carries out audio decoder and filtering.
S205: gather voice data by audio input device, the audio frequency played data and the voice data that obtain after to decoding filtering by audio mixing and audio adjustment module are synthesized processing, to obtain forming new audio frequency played data, and play this new audio frequency played data by sound equipment.
S206, carries out initial synchronization when video playback data and new audio frequency played data are started, and to carrying out synchronous correction processing in video playback data and new audio frequency played data playing process.
S207, slave computer carries out transferring to host computer after audio coding and format conversion to this new audio frequency played data; Be specially, these new audio frequency played data are carried out audio coding simultaneously, convert through format converting module the audio stream that is applicable to flow transmission to again, pass through in real time Wi-Fi Internet Transmission to host computer, now the data of transmission are through the voice data of coding instead of original PCM data, have reduced greatly taking of bandwidth.
S208, host computer receives this new audio frequency played data, and after video playback data and this new audio frequency played data are play and finished, and both is carried out to audio frequency and video synthetic, to generate new media file; Be specially, host computer receives the audio stream data that comprises voice of slave computer passback, preserves and becomes file.When media play finishes, carry out the synthetic of audio frequency and video on backstage, finally obtain a homemade MV file.
Particularly, please refer to Fig. 4, S206 specifically comprises:
In the time that video starts to play, video the first frame sends a frame data bag to slave computer when broadcasting, and the form of this packet is as follows:
Frame number m |
Frame correspondence system time t1(ms) |
In the time that this frame data bag arrives slave computer, the slave computer audio stream plays of decoding at once postbacks a frame data bag to host computer simultaneously, and form is as shown above, and the frame number that frame number sends over host computer is identical.In the time that host computer receives the frame data bag that slave computer postbacks, obtain the frame correspondence system time t2 in this packet, poor with the t2 of current system time is Δ t=t2-t1, in order more accurately reasonably to add up this time difference, the interference of exclusive segment contingent situation, use the method for sample mean, this process is carried out to repetitive operation.Front 10 frames to video carry out this repetitive operation, 10 groups that obtain are carried out assembly average, obtain the mean value Δ t ' of time difference, for example, if when Δ t ' has exceeded certain predetermined threshold value (50ms), slow down the processing of the frame rate of presentation of video frames, if current frame of video is p1, within the time of Δ t '/2, maintain this frame, reach the initial synchronous object of audio frequency and video.Frame of video is slowed down the processing of frame rate instead of processed at audio frequency end is to experience in order to obtain good user, in the moment of carrying out initial synchronous correction, picture to a certain degree delay user's impact very tricklely, in the time that Δ t ' is enough little, even can not discover to some extent.
After having ensured that audio frequency and video have started synchronously, in decoding playing process, audio frequency and video are carried out to synchronous correction.
Please refer to Fig. 5, in the process of playing at audio/video decoding, need on each time break node, do a synchronous correction, the present invention is in order to obtain good user's experience, get this time interval and be 10s(in playing process every 10s of mistake carry out a synchronous correction).
The detailed process of synchronous correction is as follows:
When the present frame of video is play, host computer sends a synchronous correction packet by Wi-Fi network to slave computer, and the current reproduction time tvideo(that this packet comprised current frame number, calculate according to present frame and frame per second is taking ms as unit).The form of packet is as follows:
Frame of video sequence number n |
Video present frame time tvideo(ms) |
After slave computer receives packet, go out according to calculation of parameter such as the sample rates of the audio stream of decoding the reproduction time taudio that current audio frame is corresponding, reformulate new packet, send to host computer.The information that this packet comprises is: frame of video sequence number, video frame time tvideo, audio frame time taudio.The form of packet is as follows:
Frame of video sequence number n |
Video frame time tvideo(ms) |
Audio frame time taudio(ms) |
Host computer receives this packet, resolves the relevant information that obtains, and calculates time tvideo ' corresponding to current video frame by the frame per second of video, and total information is as follows:
1. frame of video sequence number n
2. video frame time tvideo
3. audio frame time taudio
4. current video frame time tvideo '
According to above information, the time Δ tv_ahead_a of the leading audio frame of frame of video can estimate according to following formula:
Same, 10 continuous frame of video are carried out to above repetitive operation, it is carried out to sample mean, obtain the Δ t ' average time of a leading audio frame of frame of video
v_ahead_a.As Δ t '
v_ahead_aabsolute value when exceeding certain default threshold value (such as 200ms), it is carried out to audio-visual synchronization correction.
Particularly, as Δ t '
v_ahead_awhen >0, video is ahead of audio frequency, slows down the frame rate of video playback at host computer, can take method below: at Δ t
v_ahead_atime in maintain present frame, then continue as usual to play;
As Δ t '
v_ahead_a < 0time, video lags behind audio frequency, carries out the processing of frame-skipping at host computer, according to Δ t '
v_ahead_acalculate the frame number k that should skip with current frame per second, then skip this k frame.
Please refer to Fig. 6, the present invention provides the real-time separating treatment of a kind of audio/video flow and synchronous system simultaneously again, comprises host computer 100 and slave computer 200.
Particularly, host computer 100 comprises:
Demultiplexing module 101, for obtaining and demultiplexing source of media file, to obtain video flowing and audio stream;
Video decode and filtration module 102, for video flowing is carried out to video decode and video filtering, to obtain video playback data, and displaying video played data;
The first format converting module 103, for carrying out format conversion and compression coding to audio stream;
The first transport module 104, for transferring to slave computer by the described audio stream after format conversion and compression coding.
Particularly, slave computer 200 comprises:
Audio decoder and filtration module 201, decode and filtering to audio stream for calling decoder, to obtain audio frequency played data, and audio plays played data.
It should be noted that host computer 100 and slave computer 200 include:
Synchronization module 30, carries out initial synchronization when video playback data and audio frequency played data are started, and to carrying out synchronous correction processing in video playback data and audio frequency played data playing process.
Compared with prior art, system of the present invention comprises host computer 100 and slave computer 200, host computer 100 carries out demultiplexing to source of media file and obtains after video flowing and audio stream, flow corresponding data by host computer 100 displaying videos, and audio stream is transferred to slave computer 200 play, thereby realize the separating treatment of audio/video flow; And before audio stream is transferred to slave computer 100, by the first format converting module 103, audio stream is carried out to format conversion and compression coding, thereby greatly reduce the network bandwidth shared while transmitting data; In addition, in this system, host computer 100 and slave computer 200 include synchronization module 30, can start video playback data and audio frequency played data time, carry out initial synchronization, and to carrying out synchronous correction processing in video playback data and audio frequency played data playing process, thereby guarantee host computer and slave computer accurate synchronization when displaying video, audio frequency respectively.
Particularly, synchronization module 30 comprises:
Initial synchronization unit 301, for according to the frame number m of video lead frame packet and the corresponding system time t of described video lead frame
1, frame number and the corresponding system time t of this audio frequency start frame in audio frequency start frame packet
2, adopt the poor Δ t=t of method of average timing statistics
2-t
1mean value Δ t ', and according to mean value Δ t ', described video lead frame is slowed down to frame rate processing; And
Synchronous correction unit 302, for estimating the time of the leading described current audio frame of described current video frame according to the reproduction time taudio of the current reproduction time tvideo of current video frame, the corresponding current video frame of current audio frame, and adopt the method for average to obtain the Δ t ' average time of the leading described current audio frame of described current video frame
v_ahead_a, and according to Δ t ' average time
v_ahead_aslow down the frame rate of video playback or carry out frame-skipping processing.
It should be noted that because initial synchronization and synchronous correction part are front being described in detail, therefore do not repeat them here.
Please refer to Fig. 7, in another embodiment of system of the present invention, this system can realize the functions such as recording, possesses the function of wireless Karaoke again, is a wireless image-sound system.
Particularly, the system in the present embodiment comprises host computer 100 ', slave computer 200 ', sound collection equipment 400 and sound equipment 500.Wherein, sound collection equipment 400 is for gathering voice data, and sound equipment 500 is for playing new audio frequency played data.
Particularly, slave computer 200 ', except the audio decoder and filtration module 201 ' that comprise in the first embodiment, also comprises:
Audio mixing and audio adjustment module 202, for audio frequency played data and voice data are synthesized to processing, to obtain forming new audio frequency played data;
The second format converting module 203, for carrying out audio coding and format conversion to this new audio frequency played data;
The second transport module 204, for transferring to described host computer by the new audio frequency played data after audio coding and format conversion.
Particularly, host computer 100 ', except demultiplexing module 101 ', video decode and the filtration module 102 ', the first format converting module 103 ', the first transport module 104 ' and the synchronization module 30 ' part that comprise in the first embodiment, also comprises:
Audio/video flow merges module 105, for receiving this new audio frequency played data, and after video playback data and this new audio frequency played data are play and finished, both is carried out to audio frequency and video synthetic, to generate new media file;
User interactive module 106, for realizing the mutual of described system and user.
It should be noted that the first transport module 104 ' and the second transport module 204 are WIFI module, host computer 100 is intelligent mobile terminal equipment.
As can be seen from the above description, the real-time separating treatment of audio/video flow of the present invention and synchronous method and system, realize the separating treatment of audio/video flow, the shared network bandwidth while greatly having reduced transmission data, and accurate synchronization can guarantee that host computer is distinguished displaying video, audio frequency with slave computer time.
In conjunction with most preferred embodiment, invention has been described above, but the present invention is not limited to the embodiment of above announcement, and should contain the various amendments of carrying out according to essence of the present invention, equivalent combinations.