CN105430537A - Method and server for synthesis of multiple paths of data, and music teaching system - Google Patents
Method and server for synthesis of multiple paths of data, and music teaching system Download PDFInfo
- Publication number
- CN105430537A CN105430537A CN201510851568.9A CN201510851568A CN105430537A CN 105430537 A CN105430537 A CN 105430537A CN 201510851568 A CN201510851568 A CN 201510851568A CN 105430537 A CN105430537 A CN 105430537A
- Authority
- CN
- China
- Prior art keywords
- frame
- video
- data
- road
- synthetic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/85—Assembly of content; Generation of multimedia applications
- H04N21/854—Content authoring
- H04N21/8547—Content authoring involving timestamps for synchronizing content
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
- H04N21/23424—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/234363—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by altering the spatial resolution, e.g. for clients with a lower screen resolution
Abstract
The invention discloses a method and a server for synthesis of multiple paths of data, and a music teaching system. The method for synthesis of multiple paths of data is suitable for being executed in the server. The method comprises the following steps of: receiving multiple paths of video data transmitted by a media terminal, wherein each path of video data comprises one or more video frames, and each video frame comprises a time stamp corresponding to acquisition time thereof; according to the time stamp of each path of video frame, selecting a reference point in time for aligning the video frames; according to frame rate of each path of video data, selecting one path of video data as synthesis reference data; starting from the selected reference point in time, orderly choosing one video frame in the synthesis reference data according to time order, and inquiring one frame, of which the time stamp is earlier than and most closest to that of the selected video frame, from each of the rest paths of video data; executing synthesis operation for the selected video frame and the inquired video frame, thereby obtaining a synthesized video frame of one or more code streams.
Description
Technical field
The present invention relates to the communications field, particularly relate to method, server and music lesson system that multichannel data is synthesized.
Background technology
At present, in the such as real time communication such as video conference or network direct broadcasting scheme, the terminal gathering media data can gather the media data such as frame of video and audio frame, and is transferred to server.Server, can to media play end transmission of media data after receiving media data.In addition, media server, before media play end transmission of media data, also can carry out data processing to media data.Such as, the frame of video from multi pass acquisition terminal can be synthesized picture-in-picture by server.
Such as, application number be CN200810131309.9 patent discloses a kind of conference system, comprise acquisition terminal, server and image display device.Gathered image can be sent in server by acquisition terminal.Received view data can be synthesized by server, is then transmitted to image display device by composograph.
But in existing Data Synthesis scheme, usually using a road picture as key frame, and the temporal associativity of synthesizing in picture between multiple image is very low.
Summary of the invention
For this reason, the invention provides a kind of scheme that multichannel data is synthesized newly, effectively solve at least one problem above.
According to an aspect of the present invention, provide a kind of method of synthesizing multichannel data, the method is suitable for performing in the server.The method comprises the steps.Receive the video data that multi-path media terminal sends.The every road video data received comprises one or more frame of video.Wherein each frame of video comprises should the timestamp of frame of video acquisition time.According to the timestamp of frame of video in the video data of every road, select the reference time point for aliging to received multi-path video data.According to the frame per second of every road video data, in the multi-path video data that selection receives, a road is as synthesis reference data.From selected reference time point, select a frame of video in this synthesis reference data in chronological order successively, and from the video data of received non-synthetic reference data, every road query time stamp early than and closest to a frame of selected frame of video.Synthetic operation is performed, to obtain the synthetic video frame of one or more code streams to selected frame of video and the frame of video inquired.
According to a further aspect of the invention, provide a kind of server that multichannel data is synthesized, comprise receiver, selection of reference frame device, frame per second selector and Compositing Engine.Receiver is suitable for the video data that reception multi-path media terminal sends.The every road video data received comprises one or more frame of video.Wherein each frame of video comprises should the timestamp of frame of video acquisition time.Selection of reference frame device is suitable for, according to the timestamp of frame of video in the video data of every road, selecting the reference time point for aliging to received multi-path video data.Frame per second selector is suitable for the frame per second according to every road video data, and in the multi-path video data that selection receives, a road is as synthesis reference data.Compositing Engine is suitable for from selected reference time point, select a frame of video in this synthesis reference data in chronological order successively, and from the video data of received non-synthetic reference data, every road query time stamp early than and closest to a frame of selected frame of video.Then, Compositing Engine performs synthetic operation, to obtain the synthetic video frame of one or more code streams to selected frame of video and the frame of video inquired.
According to a further aspect of the invention, provide a kind of music lesson system, comprise according to media termination of the present invention, server and media play end media termination, be suitable for gathering video data and voice data.Server is suitable for synthesizing from the media data from multi-path media terminal.Apparatus for media playing is suitable for obtaining synthetic video frame and/or Composite tone frame from server.
According to scheme of synthesizing multichannel data of the present invention, when receiving the frame of video from multi-path media terminal, according to the acquisition time of frame of video, alignment operation can be performed successively to frame of video, and then the frame of video through alignment is synthesized a road frame of video.Particularly, synthetic schemes according to the present invention makes each several part picture in synthetic video frame have higher synchronousness.Like this, can reception one road synthetic video frame be passed through according to apparatus for media playing of the present invention, to realize multi-path media terminal gather the live broadcasting of image.It should be noted that, synthetic schemes according to the present invention, for the flow medium live systems such as Online Music teaching, greatly can improve systematic function.In addition, according to the code stream expected, the synthetic video frame of multiple code stream can also be generated according to synthetic schemes of the present invention.Like this, can transmit to apparatus for media playing the frame of video code stream mated with current network speed according to media server of the present invention, thus ensure the high real-time of transfer of data, to improve the performance of live broadcast system further.
Accompanying drawing explanation
In order to realize above-mentioned and relevant object; combine description below and accompanying drawing herein to describe some illustrative aspect; these aspects indicate the various modes can putting into practice principle disclosed herein, and all aspects and equivalent aspect thereof are intended to fall in the scope of theme required for protection.Read detailed description below in conjunction with the drawings, above-mentioned and other object of the present disclosure, Characteristics and advantages will become more obvious.Throughout the disclosure, identical Reference numeral is often referred to for identical parts or element.
Fig. 1 shows the block diagram according to the present invention's exemplary music tutoring system 100;
Fig. 2 shows the block diagram to the server 200 that multichannel data synthesizes according to some embodiments of the invention;
Fig. 3 shows the flow chart to the method 300 that multichannel data synthesizes according to some embodiments of the invention; And
Fig. 4 shows the flow chart to the method 400 that multichannel data synthesizes according to some embodiments of the invention.
Embodiment
Below with reference to accompanying drawings exemplary embodiment of the present disclosure is described in more detail.Although show exemplary embodiment of the present disclosure in accompanying drawing, however should be appreciated that can realize the disclosure in a variety of manners and not should limit by the embodiment set forth here.On the contrary, provide these embodiments to be in order to more thoroughly the disclosure can be understood, and complete for the scope of the present disclosure can be conveyed to those skilled in the art.
Fig. 1 shows the block diagram according to the present invention's exemplary music tutoring system 100.As shown in Figure 1, music lesson system 100 can comprise multiple student client 110, server 120 and teacher's client 130.In music lesson system 100, student client 110 and teacher's client 130 carry out real time communication by server 120, to carry out Online Music teaching.Such as, when student plays, student client 110 may be implemented as media termination, gathers and plays the media datas such as relevant such as Audio and Video to student, and transmit these media datas by server 120 to teacher's client 130.Teacher's client 130 may be implemented as apparatus for media playing, receives and plays media data, so that teacher understands the performance situation of student in real time.Meanwhile, teacher's client 130 also may be implemented as media termination, gathers the media data of the contents such as feedback guidance that teacher plays student or teaching demonstration, and passes through server 120 and transmit to student client.Student client 110 may be implemented as apparatus for media playing, receives and plays the media data from teacher's client 130, so that teacher plays student carry out Real-time Feedback, or carries out teaching demonstration to student in real time.In a word, student client 110 and teacher's client 130 can be implemented as media termination or apparatus for media playing.In order to simplified characterization, hereinafter no longer distinguish the particular type of media termination and apparatus for media playing.Here, media data such as comprises the fingering, breath, the musical instrument sound that play an instrument and instructs the contents of courses such as official documents and correspondence, but is not limited thereto.
Usually, music lesson system has higher requirement to the aspect such as real-time and synchronism.The present invention is directed to the server link of music lesson system, propose a kind of new Data Synthesis scheme.Below in conjunction with Fig. 2, further exemplary illustration is carried out to the server in music lesson system.It should be noted that, can be used in music lesson system according to server of the present invention, but be not limited to this.Such as, also can be applied in such as video conference according to server of the present invention, compete in the real time flow medium scheme such as live.
Fig. 2 shows the block diagram to the server 200 that multichannel data synthesizes according to some embodiments of the invention.Server 200 can carry out data processing to the media data from one or more media termination, and by treated transfer of data to one or more apparatus for media playing.Although server 200 is depicted as single entity, the function of server 200 can be dispersed in multiple computing equipment, computing cluster or data center, and the assembly of server 200 can reside in multiple geographical position.
Server 200 comprises receiver 210, selection of reference frame device 220, frame per second selector 230, Compositing Engine 240 and transmitter 250.
Receiver 210 is suitable for receiving the voice data from multi-path media terminal and video data.Each media termination usually in the mode of network packet by voice data and video data transmission to apparatus for media playing.Wherein, video data refers to multiple video packets of data that receiver 210 receives successively.Each video packets of data carries out the network packet that becomes packaged by Internet Transmission for media termination to an audio frame.In an embodiment in accordance with the invention, a video packets of data message format example is:
TCP_info+AV_Info+VideoData
Wherein, TCP_info is TCP transmission protocol header.
AV_Info comprises frame of video controling parameters:
DWORDc_type Control Cooling;
_ _ int64stamp timestamp;
DWORDc_value controls numerical value.
VideoData is video compression data corresponding to a frame of video, and H.264 its compressed format be such as, but be not limited thereto.Timestamp included in AV_Info is the acquisition time of frame of video.In other words, this timestamp is the capture time that media termination gathers original image.
When receiving a video packets of data, receiver 210 can from wherein extracting frame of video (AV_Info+VideoData).In addition, receiver 210 can be configured to comprise meshwork buffering district 211, and extracted frame of video can be stored in meshwork buffering district 211.In an embodiment in accordance with the invention, frame of video to be stored in the example code in network-caching district 211 as follows for receiver 210:
m_VideoRecRing->Set((char)pBuf,nLen,(char*)&vsinfo,sizeof(AV_Info))
Wherein pBuf is the adhoc buffer of frame of video, and nLen is the length of frame of video, and vsinfo is frame of video controling parameters.
In addition, in an embodiment in accordance with the invention, meshwork buffering district 211 is configured to comprise multiple distribution district.Like this, every road media termination can a corresponding distribution district.Frame of video from same media termination can be stored in same distribution district by receiver 210.
Selection of reference frame device 220 is suitable for the timestamp of the frame of video according to every road media termination, selectes the reference time point for the multi-channel video frame that aligns.Usually, every road media termination is according to acquisition time sequential delivery frame of video.Selection of reference frame device 220 can inquire about the timestamp of the frame of video that this road is received at first from the frame of video of every road.Selection of reference frame device 220 can compare the timestamp inquired, and using time value timestamp the latest as reference time point.Selection of reference frame device 220 can to delete in the video data of every road timestamp early than the frame of video of reference time point.
Frame per second selector 230 is suitable for from the frame of video of multi-path media terminal, select a circuit-switched data as synthesis reference data.In one embodiment, frame per second selector 230 selects most Gao mono-tunnel of frame per second as synthesis reference data, but is not limited thereto.
Compositing Engine 240 can from reference time point, select a frame of video in synthesis reference data successively in chronological order, and from the every road video data outside synthesis reference data query time stamp early than and the frame of video of timestamp closest to selected frame of video.In other words, Compositing Engine 240 can perform repeatedly alignment operation.In each alignment operation, although may there is the small time difference between the timestamp of this group frame of video in the frame of video that selected and frame of video that is that inquire about are mated as one group of timestamp by Compositing Engine 240.According to one embodiment of present invention exemplary illustration will be carried out to each alignment operation below.Such as, the media termination be connected with server 200 has A and B two-way.
Acquisition frame rate from the frame of video of A road media termination is 25 frames/second, and its timestamp from reference time point is followed successively by: 40ms80ms120ms160ms
Acquisition frame rate from the frame of video of B road media termination is 10 frames/second, and its timestamp from reference time point is followed successively by: 10ms110ms210ms310ms
In order to simplified characterization, the gap here between shown timestamp in 1 second duration, and eliminates hour, minute of each timestamp and the numerical value of second unit level and illustrate only the concrete numerical value of millisecond unit level.Wherein, A road frame of video is as synthesis reference data.When an execution alignment operation, Compositing Engine 240 have selected the frame of video that timestamp is 40ms.In addition, Compositing Engine 240 from the B road of non-synthetic reference data query time stamp early than and closest to the frame of video of 40ms.The frame of video inquired is the frame of video that 10ms is corresponding.Like this, Compositing Engine 240 in this alignment operation using frame of video that the frame of video corresponding to 40ms and 10ms is mated as one group of timestamp.When performing another alignment operation, Compositing Engine 240 from the frame of video of B road query time stamp early than and closest to the frame of video of 80ms.The frame of video inquired is the frame of video that 10ms is corresponding.Like this, Compositing Engine frame of video that frame of video corresponding for 80ms and 10ms is mated as one group of timestamp.The like, 120ms and 110ms is the frame of video of one group of timestamp coupling, repeats no more here.
It should be noted that, carry out achieving time synchronized between the multi-path media terminal that communicates with server 200.In other words, the timestamp of every road frame of video has identical time reference.Compositing Engine 240 is performed to the frame of video of alignment operation and selected often group timestamp coupling according to timestamp, wherein the difference of the acquisition time of each frame of video is very little, therefore has higher synchronousness.On this basis, Compositing Engine 240 continue to often organize frame of video carry out synthetic operation time, make each several part picture in synthesized frame of video have higher synchronousness.To sum up, the frame of video from multiple media termination can be synthesized a road frame of video according to server 200 of the present invention, and in each synthetic video frame, each several part picture has higher synchronism.
In addition, the frame of video often organizing timestamp coupling can also synthesized the code stream regulating this frame of video in the process of a frame of video by Compositing Engine 240.Specifically, first Compositing Engine 240 performs decode operation to often organizing frame of video.Such as, one group of frame of video comprises 4 frame of video (namely media termination is 4 tunnels), and Compositing Engine 240 obtains the image of 4 640*480 by performing decode operation.Depend on the synthesis code stream of expectation, Compositing Engine 240 can be selected to carry out adjusted size to 4 images.Such as, it is 320*240 that Compositing Engine 240 often will open Image Adjusting by trimming operation, but is not limited thereto.In one embodiment, Compositing Engine 240 adjusts the example code of image size and is:
voidCDsCaptureDemoDlg::YUVToYUV(BYTE*pDesStr,intDesWidth,intDesHeight,BYTE*pSourceStr,intSourceWidth,intSourceHeight)
Before adjustment, the yuv image of SourceWidth=640, SourceHeight=480 and 640*480.
After adjustment conversion, become DesWidth=320, the yuv image of DesHeight=240 and 320*240.
Subsequently, Compositing Engine 240 carries out Images uniting to the image through cutting, and is encoded to a synthetic video frame.Like this, can by generating the synthetic video frame of multiple code stream to the adjustment of picture size according to Compositing Engine 240 of the present invention.
Transmitter 250 can transmit to apparatus for media playing the synthetic video frame generated.Specifically, transmitter 250 of the present invention can transmit to apparatus for media playing the synthetic video frame code stream mated with current network speed.Like this, server 200, when transmitting the video data from multiple media termination to apparatus for media playing, has higher transmission real-time.
In addition, received multi-path audio-frequency data can also be synthesized a road voice data according to server 200 of the present invention.Specifically, receiver 210 receive every road voice data and comprise one or more audio frame.Each audio frame comprises the timestamp of multiple audio frequency sampling point and this audio frame.This timestamp is such as the acquisition time of the first sampling point in multiple audio frequency sampling point.Audio frame through alignment operation according to this timestamp to multichannel voice frequency frame time of implementation alignment operation, then can be synthesized a road Composite tone frame by Compositing Engine 240.
Fig. 3 shows the flow chart to the method 300 that multichannel data synthesizes according to some embodiments of the invention.Method 300 is suitable for performing in media server according to the present invention.
As shown in Figure 3, method 300 starts from step S310.In step S310, receive the video data that multi-path media terminal sends.The every road video data received comprises one or more frame of video.Wherein each frame of video comprises should the timestamp of frame of video acquisition time.Subsequently, method enters step S320, according to the timestamp of frame of video in the video data of every road, selectes the reference time point for aliging to received multi-path video data.According to one embodiment of the invention, in step s 320, the timestamp of received every road video data, received at first frame of video is compared, and select time value timestamp is the latest as reference time point.In step s 320, can also in received every road video data, timestamp early than described reference time point frame of video perform deletion action.
In addition, method 300 also comprises step S330.In step S330, according to the frame per second of every road video data, in the multi-path video data that selection receives, a road is as synthesis reference data.According to one embodiment of the invention, in step S330, to select in the multi-path video data that receives most Gao mono-tunnel of frame per second as synthesis reference data.But be not limited thereto, depend on the frame per second expecting synthetic video frame, in step S330, a road video data of other frame per second also can be selected as synthesis reference data.
Subsequently, method 300 performs step S340.In step S340, from selected reference time point, select a frame of video in this synthesis reference data in chronological order successively, and from the video data of received non-synthetic reference data, every road query time stamp early than and the frame of video of timestamp closest to selected frame of video.As mentioned above, based on the timestamp of frame of video in step S340, from the video data of every road, have selected a frame of video, and using frame of video that selected frame of video is mated as one group of timestamp.Subsequently, method 300 can perform step S350.In step S350, synthetic operation is performed, to obtain the synthetic video frame of one or more code streams to selected frame of video and the frame of video inquired.According to one embodiment of the invention, in step S350, first decode operation is performed to selected frame of video and the frame of video inquired.Then, in step S350, synthetic operation is performed to obtain the synthetic video frame of one or more code streams described to the frame of video through decode operation.Wherein, in order to synthesize many middle code streams, to the frame of video (i.e. a two field picture) through decode operation before synthetic video frame, trimming operation can be carried out to adjust picture size.Here, the embodiment of method 300 discloses in the description of Fig. 2, repeats no more here.
Fig. 4 shows the flow chart to the method 400 that multichannel data synthesizes according to some embodiments of the invention.Method 400 is suitable for performing in media server according to the present invention.
As shown in Figure 4, method 400 starts from step S410.In step S410, receive the video data that multi-path media terminal sends.The every road video data received comprises one or more frame of video.Wherein each frame of video comprises should the timestamp of frame of video acquisition time.Subsequently, method enters step S420, according to the timestamp of frame of video in the video data of every road, selectes the reference time point for aliging to received multi-path video data.According to one embodiment of the invention, in the step s 420, the timestamp of received every road video data, received at first frame of video is compared, and select time value timestamp is the latest as reference time point.In the step s 420, can also in received every road video data, timestamp early than described reference time point frame of video perform deletion action.
In addition, method 400 also comprises step S430.In step S430, according to the frame per second of every road video data, in the multi-path video data that selection receives, a road is as synthesis reference data.According to one embodiment of the invention, in step S430, to select in the multi-path video data that receives most Gao mono-tunnel of frame per second as synthesis reference data.But be not limited thereto, depend on the frame per second expecting synthetic video frame, in step S430, a road video data of other frame per second also can be selected as synthesis reference data.
Subsequently, method 400 performs step S440.In step S440, from selected reference time point, select a frame of video in this synthesis reference data in chronological order successively, and from the video data of received non-synthetic reference data, every road query time stamp early than and the frame of video of timestamp closest to selected frame of video.As mentioned above, based on the timestamp of frame of video in step S440, from the video data of every road, have selected a frame of video, and using frame of video that selected frame of video is mated as one group of timestamp.Subsequently, method 400 can perform step S450.In step S450, synthetic operation is performed, to obtain the synthetic video frame of one or more code streams to selected frame of video and the frame of video inquired.According to one embodiment of the invention, in step S450, first decode operation is performed to selected frame of video and the frame of video inquired.Then, in step S450, synthetic operation is performed to obtain the synthetic video frame of one or more code streams described to the frame of video through decode operation.Wherein, in order to synthesize many middle code streams, to the frame of video (i.e. a two field picture) through decode operation before synthetic video frame, trimming operation can be carried out to adjust picture size.
In addition, according to one embodiment of the invention, method 400 also comprises step S460.In step S460, receive the voice data that multi pass acquisition end sends.Wherein, every road voice data comprises one or more audio frame.Each audio frame comprises the timestamp of its acquisition time corresponding.Subsequently, method 400 performs step S470.In step S470, according to the timestamp of every road voice data sound intermediate frequency frame, by received multi-path audio-frequency data sound intermediate frequency frame time of implementation alignment operation.Subsequently, method 400 enters step S480, the audio frame through alignment operation is synthesized a road Composite tone frame.In addition, method 400 also comprises step S490, to apparatus for media playing transmission synthetic video frame and/or Composite tone frame.Here, the embodiment of method 400 is open in the explanation of Fig. 2, repeats no more here.
A10, server as described in A9, wherein, described selection of reference frame device is suitable for selecting according to following manner carrying out the reference time point of aliging for received multi-path video data: compare the timestamp of received every road video data, received at first frame of video, select time value timestamp is the latest as reference time point.A11, server as described in A10, wherein said selection of reference frame device is also suitable for: in received every road video data, timestamp performs deletion action early than the frame of video of described reference time point.A12, server according to any one of A9-A11, wherein, described frame per second selector is suitable for selecting a road in the multi-path video data received as synthesis reference data according to following manner: to select in the multi-path video data received most Gao mono-tunnel of frame per second as synthesis reference data.A13, server according to any one of A9-A12, wherein, described Compositing Engine is suitable for performing synthetic operation, to obtain the synthetic video frame of one or more code streams according to following manner to selected frame of video and the frame of video inquired: perform decode operation to selected frame of video and the frame of video inquired; And synthetic operation is performed to obtain the synthetic video frame of one or more code streams described to the frame of video through decode operation.A14, server as described in A13, wherein, described Compositing Engine, before performing synthetic operation to the frame of video through decode operation, is also suitable for: to described in the frame of video of decode operation one or more trimming operation that carries out to adjust picture size.A15, server according to any one of A9-A14, wherein, described receiver is also suitable for: receive the voice data that multi-path media terminal sends, and wherein every road voice data comprises one or more audio frame, and each audio frame comprises the timestamp of its acquisition time corresponding; And described Compositing Engine is also suitable for: according to the timestamp of every road voice data sound intermediate frequency frame, by received multi-path audio-frequency data sound intermediate frequency frame time of implementation alignment operation, and the audio frame through alignment operation is synthesized a road Composite tone frame.A16, server as described in claim A15, also comprise transmitter, be suitable for transmitting described synthetic video frame and/or described Composite tone frame to apparatus for media playing.
In specification provided herein, describe a large amount of detail.But can understand, embodiments of the invention can be put into practice when not having these details.In some instances, be not shown specifically known method, structure and technology, so that not fuzzy understanding of this description.
Similarly, be to be understood that, in order to simplify the disclosure and to help to understand in each inventive aspect one or more, in the description above to exemplary embodiment of the present invention, each feature of the present invention is grouped together in single embodiment, figure or the description to it sometimes.But, the method for the disclosure should be construed to the following intention of reflection: namely the present invention for required protection requires than the feature more multiple features clearly recorded in each claim.Or rather, as claims below reflect, all features of disclosed single embodiment before inventive aspect is to be less than.Therefore, the claims following embodiment are incorporated to this embodiment thus clearly, and wherein each claim itself is as independent embodiment of the present invention.
Those skilled in the art are to be understood that the module of the equipment in example disclosed herein or unit or assembly can be arranged in equipment as depicted in this embodiment, or alternatively can be positioned in one or more equipment different from the equipment in this example.Module in aforementioned exemplary can be combined as a module or can be divided into multiple submodule in addition.
Those skilled in the art are appreciated that and adaptively can change the module in the equipment in embodiment and they are arranged in one or more equipment different from this embodiment.Module in embodiment or unit or assembly can be combined into a module or unit or assembly, and multiple submodule or subelement or sub-component can be put them in addition.Except at least some in such feature and/or process or unit be mutually repel except, any combination can be adopted to combine all processes of all features disclosed in this specification (comprising adjoint claim, summary and accompanying drawing) and so disclosed any method or equipment or unit.Unless expressly stated otherwise, each feature disclosed in this specification (comprising adjoint claim, summary and accompanying drawing) can by providing identical, alternative features that is equivalent or similar object replaces.
In addition, those skilled in the art can understand, although embodiments more described herein to comprise in other embodiment some included feature instead of further feature, the combination of the feature of different embodiment means and to be within scope of the present invention and to form different embodiments.Such as, in the following claims, the one of any of embodiment required for protection can use with arbitrary compound mode.
In addition, some in described embodiment are described as at this can by the processor of computer system or the method implemented by other device performing described function or the combination of method element.Therefore, there is the device of processor formation for implementing the method or method element of the necessary instruction for implementing described method or method element.In addition, the element described herein of device embodiment is the example as lower device: this device is for implementing the function performed by the element of the object in order to implement this invention.
As used in this, unless specifically stated so, use ordinal number " first ", " second ", " the 3rd " etc. to describe plain objects and only represent the different instances relating to similar object, and be not intended to imply the object be described like this must have the time upper, spatially, sequence aspect or in any other manner to definite sequence.
Although the embodiment according to limited quantity describes the present invention, benefit from description above, those skilled in the art understand, in the scope of the present invention described thus, it is contemplated that other embodiment.In addition, it should be noted that the language used in this specification is mainly in order to object that is readable and instruction is selected, instead of select to explain or limiting theme of the present invention.Therefore, when not departing from the scope and spirit of appended claims, many modifications and changes are all apparent for those skilled in the art.For scope of the present invention, be illustrative to disclosing of doing of the present invention, and nonrestrictive, and scope of the present invention is defined by the appended claims.
Claims (10)
1., to the method that multichannel data synthesizes, the method is suitable for performing in the server, and the method comprises:
Receive the video data that multi-path media terminal sends, the every road video data received comprises one or more frame of video, and wherein each frame of video comprises should the timestamp of frame of video acquisition time;
According to the timestamp of frame of video in the video data of every road, select the reference time point for aliging to received multi-path video data;
According to the frame per second of every road video data, in the multi-path video data that selection receives, a road is as synthesis reference data;
From selected reference time point, select a frame of video in this synthesis reference data in chronological order successively, and from the video data of received non-synthetic reference data, every road query time stamp early than and closest to a frame of selected frame of video; And
Synthetic operation is performed, to obtain the synthetic video frame of one or more code streams to selected frame of video and the frame of video inquired.
2. the method for claim 1, wherein the described step selecting reference time point for aliging to received multi-path video data comprises:
Compare the timestamp of received every road video data, received at first frame of video, select time value timestamp is the latest as reference time point.
3. method as claimed in claim 2, also comprises:
To in received every road video data, timestamp early than described reference time point frame of video perform deletion action.
4. the method according to any one of claim 1-3, wherein, in the multi-path video data that described selection receives, a road comprises as the step of synthesis reference data:
To select in the multi-path video data that receives most Gao mono-tunnel of frame per second as synthesis reference data.
5. the method according to any one of claim 1-4, wherein, described to selected frame of video and the frame of video execution synthetic operation inquired, comprise with the step of the synthetic video frame obtaining one or more code streams:
Decode operation is performed to selected frame of video and the frame of video inquired; And
Synthetic operation is performed to obtain the synthetic video frame of one or more code streams described to the frame of video through decode operation.
6. method as claimed in claim 5, wherein, describedly performs synthetic operation to the frame of video through decode operation and comprises with the step obtaining described synthetic video frame:
To described in the frame of video of decode operation one or more trimming operation that carries out to adjust picture size.
7. the method according to any one of claim 1-6, also comprises:
Receive the voice data that multi-path media terminal sends, wherein every road voice data comprises one or more audio frame, and each audio frame comprises the timestamp of its acquisition time corresponding;
According to the timestamp of every road voice data sound intermediate frequency frame, by received multi-path audio-frequency data sound intermediate frequency frame time of implementation alignment operation; And
Audio frame through alignment operation is synthesized a road Composite tone frame.
8. method as claimed in claim 7, wherein,
Described synthetic video frame and/or described Composite tone frame is transmitted to apparatus for media playing.
9., to the server that multichannel data synthesizes, comprising:
Receiver, be suitable for receiving the video data that multi-path media terminal sends, the every road video data received comprises one or more frame of video, and wherein each frame of video comprises should the timestamp of frame of video acquisition time;
Selection of reference frame device, is suitable for, according to the timestamp of frame of video in the video data of every road, selecting the reference time point for aliging to received multi-path video data;
Frame per second selector, is suitable for the frame per second according to every road video data, and in the multi-path video data that selection receives, a road is as synthesis reference data; And
Compositing Engine, be suitable for from selected reference time point, select a frame of video in this synthesis reference data in chronological order successively, and from the video data of received non-synthetic reference data, every road query time stamp early than and closest to a frame of selected frame of video
Synthetic operation is performed, to obtain the synthetic video frame of one or more code streams to selected frame of video and the frame of video inquired.
10. a music lesson system, comprising:
Media termination, is suitable for gathering video data and voice data;
The server that multichannel data is synthesized as claimed in claim 9; And
Apparatus for media playing, is suitable for obtaining synthetic video frame and/or Composite tone frame from described server.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510851568.9A CN105430537B (en) | 2015-11-27 | 2015-11-27 | Synthetic method, server and music lesson system are carried out to multichannel data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510851568.9A CN105430537B (en) | 2015-11-27 | 2015-11-27 | Synthetic method, server and music lesson system are carried out to multichannel data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105430537A true CN105430537A (en) | 2016-03-23 |
| CN105430537B CN105430537B (en) | 2018-04-17 |
Family
ID=55508420
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510851568.9A Active CN105430537B (en) | 2015-11-27 | 2015-11-27 | Synthetic method, server and music lesson system are carried out to multichannel data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105430537B (en) |
Cited By (30)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106027991A (en) * | 2016-07-12 | 2016-10-12 | 李巍 | Medical video and live broadcast all-in-one machine |
| CN106060080A (en) * | 2016-07-12 | 2016-10-26 | 李巍 | Medical video signal collection and transcoding system |
| CN106157213A (en) * | 2016-07-12 | 2016-11-23 | 李巍 | A kind of medical video image live broadcasting method |
| CN106231319A (en) * | 2016-07-14 | 2016-12-14 | 观止云（北京）信息技术有限公司 | A kind of method alignd frame by frame based on software and hardware combining |
| CN106604097A (en) * | 2016-12-07 | 2017-04-26 | 广东威创视讯科技股份有限公司 | Method and system for transmitting multipath video signals |
| CN108682436A (en) * | 2018-05-11 | 2018-10-19 | 北京海天瑞声科技股份有限公司 | Voice alignment schemes and device |
| CN108881927A (en) * | 2017-11-30 | 2018-11-23 | 北京视联动力国际信息技术有限公司 | A kind of video data synthetic method and device |
| CN108962228A (en) * | 2018-07-16 | 2018-12-07 | 北京百度网讯科技有限公司 | model training method and device |
| CN108989906A (en) * | 2018-08-22 | 2018-12-11 | 佛山龙眼传媒科技有限公司 | A kind of live video processing method and processing device |
| CN109089129A (en) * | 2018-09-05 | 2018-12-25 | 南京爱布谷网络科技有限公司 | The steady more video binding live broadcast systems of one kind and its method |
| CN109429073A (en) * | 2017-09-01 | 2019-03-05 | 杭州海康威视数字技术股份有限公司 | The method, apparatus and system for sending multi-medium data, playing multi-medium data |
| CN109600649A (en) * | 2018-08-01 | 2019-04-09 | 北京微播视界科技有限公司 | Method and apparatus for handling data |
| CN109729391A (en) * | 2018-12-18 | 2019-05-07 | 北京华夏电通科技有限公司 | A kind of sending method and system of multipath media stream |
| CN109729373A (en) * | 2018-12-27 | 2019-05-07 | 广州华多网络科技有限公司 | Mixed flow method, apparatus and storage medium, the computer equipment of stream medium data |
| CN110602522A (en) * | 2019-10-11 | 2019-12-20 | 西南民族大学 | Multi-path real-time live webRTC stream synthesis method |
| CN110636321A (en) * | 2019-09-30 | 2019-12-31 | 北京达佳互联信息技术有限公司 | Data processing method, device, system, mobile terminal and storage medium |
| CN110719496A (en) * | 2018-07-11 | 2020-01-21 | 杭州海康威视数字技术股份有限公司 | Multi-path code stream packaging and playing method, device and system |
| CN110800272A (en) * | 2018-09-28 | 2020-02-14 | 深圳市大疆软件科技有限公司 | Cluster rendering method, device and system |
| CN110913273A (en) * | 2019-11-27 | 2020-03-24 | 北京翔云颐康科技发展有限公司 | Video live broadcasting method and device |
| CN110958466A (en) * | 2019-12-17 | 2020-04-03 | 杭州当虹科技股份有限公司 | SDI signal synchronous return method based on RTMP transmission |
| CN111107299A (en) * | 2019-12-05 | 2020-05-05 | 视联动力信息技术股份有限公司 | Method and device for synthesizing multi-channel video |
| WO2020135055A1 (en) * | 2018-12-28 | 2020-07-02 | 广州市百果园信息技术有限公司 | Method, device and apparatus for adding video special effects and storage mediem |
| CN111787365A (en) * | 2020-07-17 | 2020-10-16 | 易视腾科技股份有限公司 | Multi-channel audio and video synchronization method and device |
| CN112492357A (en) * | 2020-11-13 | 2021-03-12 | 北京安博盛赢教育科技有限责任公司 | Method, device, medium and electronic equipment for processing multiple video streams |
| CN112564837A (en) * | 2019-09-25 | 2021-03-26 | 杭州海康威视数字技术股份有限公司 | Multi-path data flow synchronization method and multi-path data flow synchronization step-by-step transmission system |
| CN112584088A (en) * | 2021-02-25 | 2021-03-30 | 浙江华创视讯科技有限公司 | Method for transmitting media stream data, electronic device and storage medium |
| CN113906734A (en) * | 2019-05-31 | 2022-01-07 | 日本电信电话株式会社 | Synchronization control device, synchronization control method, and synchronization control program |
| CN114383667A (en) * | 2022-01-29 | 2022-04-22 | 重庆长安汽车股份有限公司 | Multi-sensor simulation data synchronous injection method and system |
| CN115547357A (en) * | 2022-12-01 | 2022-12-30 | 合肥高维数据技术有限公司 | Audio and video counterfeiting synchronization method and counterfeiting system formed by same |
| CN115767130A (en) * | 2022-09-27 | 2023-03-07 | 北京奇艺世纪科技有限公司 | Video data processing method, device, equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110053123A1 (en) * | 2009-08-31 | 2011-03-03 | Christopher John Lonsdale | Method for teaching language pronunciation and spelling |
| CN102340681A (en) * | 2010-07-26 | 2012-02-01 | 深圳市锐取软件技术有限公司 | 3D (three-dimensional) stereo video single-file double-video stream recording method |
| CN104994278A (en) * | 2015-06-30 | 2015-10-21 | 北京竞业达数码科技有限公司 | Method and device for synchronously processing multiple paths of videos |
| CN105100733A (en) * | 2015-08-27 | 2015-11-25 | 广东威创视讯科技股份有限公司 | Video playing method and system of mosaic display device |
-
2015
- 2015-11-27 CN CN201510851568.9A patent/CN105430537B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110053123A1 (en) * | 2009-08-31 | 2011-03-03 | Christopher John Lonsdale | Method for teaching language pronunciation and spelling |
| CN102340681A (en) * | 2010-07-26 | 2012-02-01 | 深圳市锐取软件技术有限公司 | 3D (three-dimensional) stereo video single-file double-video stream recording method |
| CN104994278A (en) * | 2015-06-30 | 2015-10-21 | 北京竞业达数码科技有限公司 | Method and device for synchronously processing multiple paths of videos |
| CN105100733A (en) * | 2015-08-27 | 2015-11-25 | 广东威创视讯科技股份有限公司 | Video playing method and system of mosaic display device |
Cited By (45)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106027991B (en) * | 2016-07-12 | 2024-02-13 | 李巍 | Medical video image live broadcast all-in-one |
| CN106060080A (en) * | 2016-07-12 | 2016-10-26 | 李巍 | Medical video signal collection and transcoding system |
| CN106157213A (en) * | 2016-07-12 | 2016-11-23 | 李巍 | A kind of medical video image live broadcasting method |
| CN106027991A (en) * | 2016-07-12 | 2016-10-12 | 李巍 | Medical video and live broadcast all-in-one machine |
| CN106060080B (en) * | 2016-07-12 | 2019-07-12 | 南京新广云信息科技有限公司 | A kind of medical video signal acquisition trans-coding system |
| CN106157213B (en) * | 2016-07-12 | 2019-06-11 | 南京新广云信息科技有限公司 | A kind of medical video image live broadcasting method |
| CN106231319A (en) * | 2016-07-14 | 2016-12-14 | 观止云（北京）信息技术有限公司 | A kind of method alignd frame by frame based on software and hardware combining |
| CN106604097A (en) * | 2016-12-07 | 2017-04-26 | 广东威创视讯科技股份有限公司 | Method and system for transmitting multipath video signals |
| CN106604097B (en) * | 2016-12-07 | 2020-08-11 | 广东威创视讯科技股份有限公司 | Method and system for transmitting multiple video signals |
| CN109429073A (en) * | 2017-09-01 | 2019-03-05 | 杭州海康威视数字技术股份有限公司 | The method, apparatus and system for sending multi-medium data, playing multi-medium data |
| CN108881927A (en) * | 2017-11-30 | 2018-11-23 | 北京视联动力国际信息技术有限公司 | A kind of video data synthetic method and device |
| CN108881927B (en) * | 2017-11-30 | 2020-06-26 | 视联动力信息技术股份有限公司 | Video data synthesis method and device |
| CN108682436A (en) * | 2018-05-11 | 2018-10-19 | 北京海天瑞声科技股份有限公司 | Voice alignment schemes and device |
| CN108682436B (en) * | 2018-05-11 | 2020-06-23 | 北京海天瑞声科技股份有限公司 | Voice alignment method and device |
| CN110719496B (en) * | 2018-07-11 | 2023-02-07 | 杭州海康威视数字技术股份有限公司 | Multi-path code stream packaging and playing method, device and system |
| CN110719496A (en) * | 2018-07-11 | 2020-01-21 | 杭州海康威视数字技术股份有限公司 | Multi-path code stream packaging and playing method, device and system |
| CN108962228A (en) * | 2018-07-16 | 2018-12-07 | 北京百度网讯科技有限公司 | model training method and device |
| CN109600649A (en) * | 2018-08-01 | 2019-04-09 | 北京微播视界科技有限公司 | Method and apparatus for handling data |
| CN108989906A (en) * | 2018-08-22 | 2018-12-11 | 佛山龙眼传媒科技有限公司 | A kind of live video processing method and processing device |
| CN109089129A (en) * | 2018-09-05 | 2018-12-25 | 南京爱布谷网络科技有限公司 | The steady more video binding live broadcast systems of one kind and its method |
| CN109089129B (en) * | 2018-09-05 | 2020-09-22 | 南京爱布谷网络科技有限公司 | Stable multi-video binding live broadcasting system and method thereof |
| CN110800272B (en) * | 2018-09-28 | 2022-04-22 | 深圳市大疆软件科技有限公司 | Cluster rendering method, device and system |
| CN110800272A (en) * | 2018-09-28 | 2020-02-14 | 深圳市大疆软件科技有限公司 | Cluster rendering method, device and system |
| CN109729391B (en) * | 2018-12-18 | 2021-07-02 | 北京华夏电通科技股份有限公司 | Method and system for sending multi-path media streams |
| CN109729391A (en) * | 2018-12-18 | 2019-05-07 | 北京华夏电通科技有限公司 | A kind of sending method and system of multipath media stream |
| CN109729373A (en) * | 2018-12-27 | 2019-05-07 | 广州华多网络科技有限公司 | Mixed flow method, apparatus and storage medium, the computer equipment of stream medium data |
| WO2020134791A1 (en) * | 2018-12-27 | 2020-07-02 | 广州华多网络科技有限公司 | Method and apparatus for mixing streaming media data, storage medium, and computer device |
| WO2020135055A1 (en) * | 2018-12-28 | 2020-07-02 | 广州市百果园信息技术有限公司 | Method, device and apparatus for adding video special effects and storage mediem |
| US11553240B2 (en) | 2018-12-28 | 2023-01-10 | Bigo Technology Pte. Ltd. | Method, device and apparatus for adding video special effects and storage medium |
| CN113906734A (en) * | 2019-05-31 | 2022-01-07 | 日本电信电话株式会社 | Synchronization control device, synchronization control method, and synchronization control program |
| CN112564837B (en) * | 2019-09-25 | 2022-05-06 | 杭州海康威视数字技术股份有限公司 | Multi-path data flow synchronization method and multi-path data flow synchronization step-by-step transmission system |
| CN112564837A (en) * | 2019-09-25 | 2021-03-26 | 杭州海康威视数字技术股份有限公司 | Multi-path data flow synchronization method and multi-path data flow synchronization step-by-step transmission system |
| CN110636321A (en) * | 2019-09-30 | 2019-12-31 | 北京达佳互联信息技术有限公司 | Data processing method, device, system, mobile terminal and storage medium |
| CN110602522B (en) * | 2019-10-11 | 2021-08-03 | 西南民族大学 | Multi-path real-time live webRTC stream synthesis method |
| CN110602522A (en) * | 2019-10-11 | 2019-12-20 | 西南民族大学 | Multi-path real-time live webRTC stream synthesis method |
| CN110913273A (en) * | 2019-11-27 | 2020-03-24 | 北京翔云颐康科技发展有限公司 | Video live broadcasting method and device |
| CN111107299A (en) * | 2019-12-05 | 2020-05-05 | 视联动力信息技术股份有限公司 | Method and device for synthesizing multi-channel video |
| CN110958466A (en) * | 2019-12-17 | 2020-04-03 | 杭州当虹科技股份有限公司 | SDI signal synchronous return method based on RTMP transmission |
| CN111787365A (en) * | 2020-07-17 | 2020-10-16 | 易视腾科技股份有限公司 | Multi-channel audio and video synchronization method and device |
| CN112492357A (en) * | 2020-11-13 | 2021-03-12 | 北京安博盛赢教育科技有限责任公司 | Method, device, medium and electronic equipment for processing multiple video streams |
| CN112584088B (en) * | 2021-02-25 | 2021-07-06 | 浙江华创视讯科技有限公司 | Method for transmitting media stream data, electronic device and storage medium |
| CN112584088A (en) * | 2021-02-25 | 2021-03-30 | 浙江华创视讯科技有限公司 | Method for transmitting media stream data, electronic device and storage medium |
| CN114383667A (en) * | 2022-01-29 | 2022-04-22 | 重庆长安汽车股份有限公司 | Multi-sensor simulation data synchronous injection method and system |
| CN115767130A (en) * | 2022-09-27 | 2023-03-07 | 北京奇艺世纪科技有限公司 | Video data processing method, device, equipment and storage medium |
| CN115547357A (en) * | 2022-12-01 | 2022-12-30 | 合肥高维数据技术有限公司 | Audio and video counterfeiting synchronization method and counterfeiting system formed by same |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105430537B (en) | 2018-04-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105430537A (en) | Method and server for synthesis of multiple paths of data, and music teaching system | |
| KR102529711B1 (en) | Receiving device, transmitting device, and data processing method | |
| JP2004525545A (en) | Webcast method and system for synchronizing multiple independent media streams in time | |
| US20120099656A1 (en) | Transmitting system, receiving device, and a video transmission method | |
| CA2695577C (en) | Apparatus, systems and methods to synchronize communication of content to a presentation device and a mobile device | |
| CN105340280B (en) | Content supply device, Content supply method, storage medium, terminal installation and contents providing system | |
| CN101998116A (en) | Method, system and equipment for realizing multi-view video service | |
| JP6329964B2 (en) | Transmission device, transmission method, reception device, and reception method | |
| CN105516090A (en) | Media play method, device and music teaching system | |
| CN105429984A (en) | Media play method, equipment and music teaching system | |
| RU2656093C2 (en) | Content supply device, content supply method, program, terminal device and content supply system | |
| CN105429983A (en) | Media data acquisition method, media terminal and music teaching system | |
| KR102499231B1 (en) | Receiving device, sending device and data processing method | |
| CN111147362B (en) | Multi-user instant messaging method, system, device and electronic equipment | |
| KR20160077066A (en) | Transmission device, transmission method, reception device, and reception method | |
| CN105359539B (en) | Content supply device, Content supply method, terminal installation and contents providing system | |
| KR102137858B1 (en) | Transmission device, transmission method, reception device, reception method, and program | |
| CN105430453A (en) | Media data acquisition method, media terminal and online music teaching system | |
| US11431770B2 (en) | Method, system, apparatus, and electronic device for managing data streams in a multi-user instant messaging system | |
| CN102045586A (en) | Network device, information processing apparatus, stream switching method and content distribution system | |
| JP2020162090A (en) | Transmission node, broadcast station system, control node and transmission control method | |
| CN109640162A (en) | Code stream conversion method and system | |
| CN106797342A (en) | Video network | |
| KR20170012225A (en) | Reception apparatus, reception method, transmission apparatus, and transmission method | |
| CN105284118A (en) | Content provision device, content provision method, program, terminal device, and content provision system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |