CN105429983A - Media data acquisition method, media terminal and music teaching system - Google Patents
Media data acquisition method, media terminal and music teaching system Download PDFInfo
- Publication number
- CN105429983A CN105429983A CN201510846324.1A CN201510846324A CN105429983A CN 105429983 A CN105429983 A CN 105429983A CN 201510846324 A CN201510846324 A CN 201510846324A CN 105429983 A CN105429983 A CN 105429983A
- Authority
- CN
- China
- Prior art keywords
- frame
- audio
- video
- sent
- buffering area
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 42
- 239000000872 buffer Substances 0.000 claims abstract description 114
- 230000005540 biological transmission Effects 0.000 claims abstract description 81
- 239000000284 extract Substances 0.000 claims abstract description 8
- 230000003139 buffering effect Effects 0.000 claims description 77
- 238000000151 deposition Methods 0.000 claims description 11
- 230000001105 regulatory effect Effects 0.000 claims 2
- 230000006835 compression Effects 0.000 description 9
- 238000007906 compression Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 241001269238 Data Species 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 150000001875 compounds Chemical class 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000009191 jumping Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
- G09B15/00—Teaching music
- G09B15/001—Boards or like means for providing an indication of chords
- G09B15/002—Electrically operated systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/433—Content storage operation, e.g. storage operation in response to a pause request, caching operations
- H04N21/4335—Housekeeping operations, e.g. prioritizing content for deletion because of storage space restrictions
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/439—Processing of audio elementary streams
- H04N21/4392—Processing of audio elementary streams involving audio buffer management
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
- H04N21/440236—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by media transcoding, e.g. video is transformed into a slideshow of still pictures, audio is converted into text
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/45—Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts
- H04N21/4508—Management of client data or end-user data
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Databases & Information Systems (AREA)
- Computer Networks & Wireless Communication (AREA)
- Business, Economics & Management (AREA)
- Physics & Mathematics (AREA)
- Educational Administration (AREA)
- Educational Technology (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
The invention discloses a media data acquisition method, a media terminal and a music teaching system. The media terminal comprises a video acquisition unit, a video buffer zone, an audio acquisition unit, an audio buffer zone, a sending buffer zone, a transmission unit and a control unit; the video acquisition unit is used for acquiring an image and coding the image into a video frame; the video buffer zone is suitable for storing the video frame; the audio acquisition unit is used for acquiring sound and coding the sound into an audio frame; the audio buffer zone is suitable for storing the audio frame; the sending buffer zone is used for storing data frames to be sent; each data frame is the video frame or the audio frame; the transmission unit is suitable for transmitting the data frames to be sent to a media server; the control unit is suitable for detecting the audio buffer zone and pushing the audio frame in the audio buffer zone into the sending buffer zone; and, when the audio buffer zone is empty, the control unit extracts the video frame from the video buffer zone and pushes the video frame into the sending buffer zone if the number of the data frames to be sent is not beyond a threshold value and the video buffer zone is not empty.
Description
Technical field
The present invention relates to the communications field, particularly relate to the method, media termination and the music lesson system that gather media data.
Background technology
At present, in the such as real time communication such as video conference or network direct broadcasting scheme, the terminal gathering media data can gather the media data such as frame of video and audio frame, and to broadcasting end transmission of media data.Such as, encapsulate the transmission of rear open network together with frame of video can being carried out with audio frame.Or frame of video and audio frame carry out encapsulating and transmitting by terminal separately.
But existing network condition is complicated and changeable, such as, there is the problem such as network jitter, intermittent interruption.When sending Voice & Video data by network, there is the problem such as network delay, network congestion in acquisition terminal.Therefore, when media play end obtains Voice & Video data from acquisition terminal, face the disconnected slack problem of card.
Summary of the invention
For this reason, the invention provides a kind of scheme of collection media data newly, effectively solve at least one problem above.
According to an aspect of the present invention, provide a kind of media termination, comprise video acquisition unit, screen buffer, audio collection unit, audio buffer, transmission buffering area, transmission unit and control unit.Video acquisition unit is suitable for gathering image and being encoded to frame of video.Screen buffer is suitable for depositing the frame of video from video acquisition unit.Audio collection unit is suitable for gathering sound and being encoded to audio frame.Audio buffer is suitable for depositing the audio frame from audio collection unit.Send buffering area to be suitable for depositing one or more Frame to be sent.Wherein each Frame to be sent is the frame of video from screen buffer or the audio frame from audio buffer.Transmission unit is suitable for one or more data frame transfer to be sent to media server.Control unit is suitable for detecting audio buffer and is pushed to by audio frame wherein and sends in buffering area, and when being empty in audio buffer, judges whether that the quantity of Frame to be sent in transmission buffering area does not exceed threshold value and screen buffer is not empty.If the quantity of Frame to be sent does not exceed threshold value and screen buffer is not sky, control unit extracts a frame frame of video and is pushed to and sends in buffering area from screen buffer.
According to a further aspect of the invention, a kind of method gathering media data is provided.The method is suitable for performing in media termination.This media termination comprises transmission buffering area.Send buffering area to be suitable for depositing one or more Frame to be sent.Wherein each Frame to be sent is frame of video or audio frame.The method comprises the following steps.Gather image and be encoded to frame of video, and this frame of video is stored in screen buffer.Gather sound and be encoded to audio frame, and this audio frame is deposited in audio buffer.Detect audio buffer and audio frame is wherein pushed to and send in buffering area.If audio buffer is empty, then judge whether that the quantity of Frame to be sent in transmission buffering area does not exceed threshold value and screen buffer is not sky.If the quantity of Frame to be sent do not exceed threshold value and screen buffer for empty, then from this screen buffer, extract a frame frame of video and be pushed to and send in buffering area.By described one or more data frame transfer to be sent to media server.
According to a further aspect of the invention, provide a kind of music lesson system, comprise according to media termination of the present invention, media server and media play end.Media server is suitable for audio frame and the frame of video of the transmission of receiving media terminal.Media play end is suitable for obtaining and audio plays frame and frame of video from media server.
According to medium data acquisition scheme of the present invention, preferentially audio buffer sound intermediate frequency frame can be pushed to and send buffering area, and be empty in audio buffer and the quantity sending Frame in buffering area does not exceed threshold value time, just frame of video in screen buffer can be pushed to and send buffering area.Like this, preferential all the time audio frame can be transmitted according to medium data acquisition scheme of the present invention, and in the gap of transmission of audio frame, frame of video is transmitted.Particularly, when the network bandwidth is lower (quantity sending Frame in buffering area exceedes threshold value), medium data acquisition scheme according to the present invention can stop to transmission buffering area pushing video frame, and is normally pushed to by audio buffer sound intermediate frequency frame in transmission buffering area.Such working method can effectively reduce the data volume that transmission unit needs to send when the network bandwidth is lower, thus the effective real-time Transmission ensureing audio frame.In other words, medium data acquisition scheme according to the present invention achieves the prioritised transmission to audio frame, thus avoids card that media play end plays sound and to break problem.It should be noted that, in the occasion that the sound importance such as music teaching are higher, ensure the real-time Transmission of voice data according to medium data acquisition scheme of the present invention, and media play end can play continuous, distortionless voice data, thus greatly improve the Experience Degree of user.
Accompanying drawing explanation
In order to realize above-mentioned and relevant object; combine description below and accompanying drawing herein to describe some illustrative aspect; these aspects indicate the various modes can putting into practice principle disclosed herein, and all aspects and equivalent aspect thereof are intended to fall in the scope of theme required for protection.Read detailed description below in conjunction with the drawings, above-mentioned and other object of the present disclosure, Characteristics and advantages will become more obvious.Throughout the disclosure, identical Reference numeral is often referred to for identical parts or element.
Fig. 1 shows the block diagram according to the present invention's exemplary music tutoring system 100;
Fig. 2 shows the block diagram of media termination 200 according to some embodiments of the invention; And
Fig. 3 shows the flow chart of the method 300 of the collection media data according to some embodiments of the present invention.
Embodiment
Below with reference to accompanying drawings exemplary embodiment of the present disclosure is described in more detail.Although show exemplary embodiment of the present disclosure in accompanying drawing, however should be appreciated that can realize the disclosure in a variety of manners and not should limit by the embodiment set forth here.On the contrary, provide these embodiments to be in order to more thoroughly the disclosure can be understood, and complete for the scope of the present disclosure can be conveyed to those skilled in the art.
Fig. 1 shows the block diagram according to the present invention's exemplary music tutoring system 100.As shown in Figure 1, music lesson system 100 can comprise multiple student client 110, server 120 and teacher's client 130.In music lesson system 100, student client 110 and teacher's client 130 carry out real time communication by server 120, to carry out Online Music teaching.Such as, when student plays, student client 110 may be implemented as media termination, gathers student and plays the media datas such as relevant such as Audio and Video, and transmit these media datas by server 120 to teacher's client 130.Teacher's client 130 may be implemented as media play end, receives and plays media data, so that teacher understands the performance situation of student in real time.Meanwhile, teacher's client 130 also may be implemented as media termination, gathers the media data of the contents such as feedback guidance that teacher plays student or teaching demonstration, and passes through server 120 and transmit to student client.Student client 110 may be implemented as media play end, receives and plays the media data from teacher's client 130, so that teacher plays student carry out Real-time Feedback, or carries out teaching demonstration to student in real time.In a word, student client 110 and teacher's client 130 can be implemented as media termination and media broadcast terminal.Here, media data such as comprises the fingering, breath, the musical instrument sound that play an instrument and instructs the contents of courses such as official documents and correspondence, but is not limited thereto.
Usually, music lesson system 100 faces the problem of transmission bandwidth instability, the situations such as such as network jitter, network intermittent interruption.But high-quality music teaching has higher requirement to the real-time of media data, the aspect such as synchronism and fluency.The present invention is directed to the collection media data link of music lesson system, propose a kind of new media termination.Below in conjunction with Fig. 2, further exemplary illustration is carried out to the media termination in music lesson system.Certainly, media termination can be student client or teacher's client, in order to simplified characterization, hereinafter no longer distinguishes the particular type of media termination.Equally, student client 110 and teacher's client 130 can be implemented as media play end.It should be noted that, can be used in music lesson system according to media termination of the present invention, but be not limited to this.Such as, also can be applied in such as video conference according to media termination of the present invention, compete in the real time flow medium scheme such as live.
Fig. 2 shows the block diagram of media termination 200 according to some embodiments of the invention.Here, computing equipment can be configured to media termination 200.And computing equipment can be implemented as a part for small size portable (or mobile) electronic equipment, these electronic equipments can be such as cell phone, personal digital assistant (PDA), personal media player equipment, wireless network browsing apparatus, individual helmet, application specific equipment or the mixing apparatus that can comprise any function above.Computing equipment can also be embodied as the personal computer comprising desktop computer and notebook computer configuration, but is not limited thereto.
As shown in Figure 2, media termination 200 comprises video acquisition unit 210, screen buffer 220, audio collection unit 230, audio buffer 240, sends buffering area 250, transmission unit 260 and control unit 270.
Video acquisition unit 210 is suitable for gathering image and being encoded to frame of video.Such as, the scene that video acquisition unit 210 couples of students play an instrument is taken, to obtain sequence of frames of video.According to one embodiment of the invention, video acquisition unit 210 comprises camera 211 and coding unit 212.Camera 211 is suitable for gathering original image frame.The acquisition parameter of original image frame is such as 640*480 size and 25 frames/second, but is not limited thereto.In addition, video acquisition unit 210 when often catching a frame original image frame, can also record current time value as the very first time stamp of catching this original image frame.According to one embodiment of the invention, the format sample of original image frame is:
{dwstampvideodata}
Wherein, dwstamp is very first time stamp, and videodata is the picture frame of YUV420 form.
Coding unit 212 is suitable for encoding to original image frame.Such as, the coding of H.264 coding unit 212 can carry out original image frame (the high compression digital video coding-coding device standard that the joint video team (JVT, JointVideoTeam) constituted jointly by ITU-T Video Coding Experts group (VCEG) and ISO/IEC dynamic image expert group (MPEG) proposes) form.The coding parameter of coding unit 212 comprises complete picture group (GOP).Length such as 100 frames of GOP, namely an image sets has 100 frame frame of video.Here, GOP starts with I frame, follows some P frames subsequently.In adjacent P interframe, also there is B frame.Such as, the partial sequence of a GOP is IPBPBPPPPBP.Wherein, I frame is intracoded frame, contains complete image information, and is not rebuilding image with reference to any additional information.P frame is forward-predictive-coded frames, is predicted by the P frame before it or I frame.B frame is bi-directional predicted condensed frame.When picture frame being compressed into B frame, coding unit 212 carrys out distilled edition frame according to the difference of adjacent former frame, this frame and rear frame data.The coding parameter of coding unit 212 can also comprise coding frame per second and coding size, but is not limited thereto.According to one embodiment of the invention, the example code that coding unit 212 performs encoding operation is as follows:
Ret＝CLDC_Open(width,height,bitrate,mode)；
Width: the width of video compression
Height: the height of video compression
Bitrate: the target bit rate of video compression
Mode: video compressed mode controls
// image compression engine initializes
Ret＝CLDC_Encode(pBuf,dwbase,m_pVideoBuffer,nEncoderLen)；
PBuf: the image data buffer of collection
Dwbase: the length of raw image data
M_pVideoBuffer: the buffering of image compression
NEncoderLen: the length after image compression
The H264 image code stream comprising SPS and PPS is produced after // Image Coding
Ret＝CLDC_Close()；
// Image Coding engine-off
Screen buffer 220 is suitable for depositing the frame of video that video acquisition unit 210 generates.Here, screen buffer 220 is such as buffer circle.Such as, screen buffer 220 can store 20 frame data of up-to-date generation all the time.
Audio collection unit 230 is suitable for gathering sound and being encoded to audio frame.In an embodiment in accordance with the invention, audio collection unit 230 can carry out the coding of ACC (AdvancedAudioCoding, Advanced Audio Coding) form to gathered sound, generation code check is 192kbps.Here, music when audio collection unit 230 can gather instrument playing and breath, but be not limited thereto.In an embodiment in accordance with the invention, to carry out the correlative code example of audio collection and encoding operation as follows for audio collection unit 230:
IntnSamples=AUDIO_SAMPLERATE; // audio sample rate
IntnChannels=1; // channel number
IntnBits=16; The figure place of // audio sample
IntnAudioBitrate=192000; The bit rate that // audio frequency exports
intnRet＝0；
nRet＝m_pFaacCodec->Open(nSamples,nChannels,nBits,nAudioBitrate)；
// audio compression parameter is set, and open audio compressor
nCodecRet＝m_pFaacCodec->Encode(pBuf,nLen,streamabuffer,nEncodeLen)；
PBuf: original voice data
NLen: the length of voice data
Streamabuffer: the buffering that coding exports
NEncodeLen: the data length after coding
// carry out audio coding, the data after coding are [ADTS head]+[audio compression data]
nRet＝m_pFaacCodec->Close()；
// audio coding terminates
In addition, audio collection unit 230 can also catch the time value of gathered audio frame, and is recorded as the second timestamp.The second timestamp that each audio frame is corresponding is such as the acquisition time of this frame first audio sample point.
Audio buffer 240 is suitable for depositing the audio frame that audio collection unit 230 generates.Here, audio buffer 240 is such as buffer circle.Every frame audio frame can also comprise second timestamp corresponding with it.Like this, media play end, when getting audio frame and the frame of video of media termination 200 transmission, synchronously can process audio frame and frame of video according to very first time stamp and the second timestamp.
Send buffering area 250 to be suitable for storing one or more Frame to be sent.Wherein each Frame to be sent is from the frame of video of screen buffer 220 or the audio frame from audio buffer 240.
Transmission unit 260 is suitable for Frame to be sent in transmission buffering area 250 to be transferred to media server successively.Like this, media server can transmit Frame to be sent to media play end.Media play termination is received and playing video data and voice data.
As mentioned above, the Frame deposited and wait for that transmission unit 260 transmits is sent in buffering area 250.Control unit 270 is suitable for extracting Frame from audio buffer 220 and screen buffer 240 and is pushed to and sends buffering area 250.Generally speaking, in the stream media systems such as such as music teaching, the importance of sound is higher than the data such as video information.Control unit 270, when each execution push operation, preferentially detects audio buffer 220.A kind of situation is that audio buffer 240 has audio frame, and control unit 270 extracts a frame audio frame and is pushed to transmission buffering area 250.Another kind of situation is, control unit 270 detects that audio buffer 240 is for empty.In other words, when control unit 270 performs this push operation, audio buffer 240 does not need the audio frame sent.Whether control unit 270 continuation detects video frame buffers 220 is empty and whether the quantity of Frame to be sent in detection transmission buffering area 250 exceedes threshold value.Threshold value is such as 5.Here, why will detect Frame in transmission buffering area 250 and whether exceed threshold value, be to judge whether current transmission unit 260 is in blocked state (i.e. Network Abnormal) by sending number of data frames in buffering area 250.In other words, Frame in transmission buffering area 250, when not being in blocked state, can be carried out Internet Transmission by transmission unit 260 in time.So, send buffering area 250 and can not accumulate the Frame exceeding threshold value.Otherwise Frame in transmission buffering area 250, when being in blocked state, can not be carried out Internet Transmission by transmission unit 260 in time.The quantity sending the Frame of accumulation in buffering area 250 can exceed threshold value.It should be noted that, the present invention does not do too much restriction to detecting screen buffer 220 and detecting the order sending buffering area 250.In one embodiment, whether first control unit 270 detects screen buffer 220 is empty, and to send in buffering area 250 data frame data whether superthreshold at screen buffer 220 for no longer detecting time empty.In other words, if screen buffer 220 is empty, then this push operation is terminated.Here, the time interval of twice adjacent push operation is such as 10 milliseconds.In addition, if screen buffer 220 is not empty, control unit 270 detects and to send in buffering area 250 data frame data amount whether superthreshold.In yet another embodiment, control unit 270 first detects and sends buffering area 250, and when number of data frames does not exceed threshold value, detects whether screen buffer 220 is empty.In implementing according to the present invention one, the example code of control unit 270 course of work is as follows:
As mentioned above, control unit 270 detect screen buffer 220 for empty and number of data frames to be sent does not exceed threshold value time, extract a frame frame of video from screen buffer 220 and be pushed to and send buffering area 250.To sum up, preferentially audio buffer 240 sound intermediate frequency frame can be pushed to according to control unit 270 of the present invention and send buffering area 250, and be empty in audio buffer 240 and the quantity sending Frame in buffering area does not exceed threshold value time, just frame of video in screen buffer 220 can be pushed to and send buffering area 250.Like this, preferential all the time audio frame can be transmitted according to media termination 200 of the present invention, and in the gap of transmission of audio frame, frame of video is transmitted.Particularly, when the network bandwidth is lower (quantity sending Frame in buffering area 250 exceedes threshold value), media termination 200 according to the present invention can stop to transmission buffering area 250 pushing video frame, and is normally pushed to by audio buffer 240 sound intermediate frequency frame in transmission buffering area 250.Such working method can effectively reduce the data volume that transmission unit 260 needs to send when the network bandwidth is lower, thus the effective real-time Transmission ensureing audio frame.In other words, media termination 200 according to the present invention achieves the prioritised transmission to audio frame, thus avoids card that media play end plays sound and to break problem.It should be noted that in the occasion that the sound importance such as music teaching are higher, ensure the real-time Transmission of voice data according to media termination of the present invention, and then media play end can play continuous, distortionless voice data.
In addition, when the quantity sending Frame in buffering area 250 is greater than threshold value, control unit 270 confirms that current network is abnormal.Control unit 270 is also suitable for calculating transmission unit 260 to the transmission code check sending frame of video in buffering area, so that video acquisition unit 210 regulates according to the generation code check of this transmission code check to frame of video.According to one embodiment of the invention, control unit 270 can be added up in the scheduled time (such as 2 seconds), and transmission unit to the quantity forwarded sending frame of video in buffering area, and then calculates transmission code check.Subsequently, video acquisition unit 210 can regulate image acquisition parameter and coding parameter, to regulate the generation code check of frame of video according to transmission code check.Such as, video acquisition unit 210 can regulate the size (resolution) or acquisition frame rate gathering original image.Again such as, video acquisition unit 220 can regulate coding parameter when original image frame being carried out to encoding operation.Coding parameter such as comprises the coding size etc. of coding frame per second, generating video frame.Like this, video acquisition unit 210, after adjustment generates code check, makes the transmission code check generating code check and transmission unit 260 pairs of frame of video match.Further, transmission unit 260 can send in real time to frame of video, and effectively reduce and send the situation that number of data frames in buffering area 250 exceedes threshold value and occur, and avoid because video frame rate in screen buffer 220 is too high and can not by the problem transmitted in time.In addition, because video acquisition unit can regulate coding frame per second, the sequence of the frame of video that transmission unit 260 is transmitted is uniform on acquisition time.Like this, media play end avoids the excessive problem of jumping with picture of play frame of video time delay.
Fig. 3 shows the flow chart of the method 300 of collection media data according to some embodiments of the invention.Method 300 is suitable for performing in media termination according to the present invention.
As shown in Figure 3, method 300 starts from step S310.In step S310, gather sound and be encoded to audio frame, and this audio frame is stored in audio buffer.Method 300 can also comprise step S320, gathers image and is encoded to frame of video, and this frame of video being stored in screen buffer.Here, audio buffer and screen buffer are such as loop buffers.In addition, in step s 320, the current time capturing original image corresponding to frame of video can also be recorded, stab as the very first time.Each frame of video can also comprise corresponding very first time stamp.In step S310, the acquisition time of all right record audio frame, as the second timestamp.Here the second timestamp is such as the timestamp of audio frame first sampled point.Audio frame can also comprise the second timestamp.Such media play end, when getting audio frame and the frame of video of media termination 200 transmission, synchronously can process audio frame and frame of video according to very first time stamp and the second timestamp.
Media termination according to the present invention comprises transmission buffering area.Send buffering area to be suitable for depositing one or more Frame to be sent.Each Frame to be sent is the frame of video from screen buffer or the audio frame from audio buffer.For the audio frame be stored in audio buffer and the frame of video left in screen buffer, method 300 controls by performing step S330, S340 and the S350 Internet Transmission to audio frame and frame of video.In step S330, detect audio buffer and audio frame be wherein pushed in transmission buffering area.In addition, if detect that in step S330 audio buffer is empty, then method 300 performs step S340.In step S340, judge whether that the quantity of Frame to be sent in transmission buffering area does not exceed threshold value and screen buffer is not sky.Specifically, according to one embodiment of present invention whether, in step S340, first detecting screen buffer is empty.If screen buffer is empty, then the current frame of video not needing to send, method 300 continues to perform step S330.If frame of video is not empty, then continues to detect the quantity sending Frame to be sent in buffering area and whether exceed threshold value.If number of data frames to be sent exceedes threshold value, then current transmission network congestion is described.In order to prioritised transmission audio frame, then frame of video can not be pushed to and send buffering area, but continue to perform step S330.According to another embodiment of the present invention, in step S340, whether the quantity first detecting Frame to be sent in current transmission buffering area exceedes threshold value.If exceed threshold value, then no longer detect screen buffer, but continue to perform step S330.
In addition, if in step S340, detect that the quantity of Frame to be sent does not exceed threshold value, and screen buffer is not empty, then perform step S350.In step S350, from screen buffer, extract a frame frame of video and be pushed to transmission buffering area.
As mentioned above, method 300 according to the present invention is suitable for generating audio frame and frame of video by performing step S310 and S320, and by performing step S330, S340 and S350, the audio frame that will send and frame of video are selected, and then Frame to be sent is stored in transmission buffering area.For Frame in transmission buffering area, method 300 carries out Internet Transmission by performing step S360.In step S360, will data frame transfer to be sent in buffering area be sent to media server.According to method 300 of the present invention, execution mode is consistent with the working method of media termination in Fig. 2 200 more specifically, repeats no more here.
A10, method as described in A8 or A9, wherein, described screen buffer, described audio buffer and described transmission buffering area are loop buffer.A11, method according to any one of A8-A10, wherein, the described quantity judging whether to send Frame to be sent in buffering area does not exceed threshold value and the step that screen buffer is not sky comprises: whether detect screen buffer is empty, if be not empty, whether the quantity continuing to detect Frame to be sent in described transmission buffering area exceedes threshold value.A12, method according to any one of A8-A11, wherein, the described quantity judging whether to send Frame to be sent in buffering area does not exceed threshold value and the step that screen buffer is not sky comprises: whether the quantity detecting Frame to be sent in described transmission buffering area exceedes threshold value, and whether when not exceeding threshold value, detecting screen buffer is empty.A13, method according to any one of A8-A12, wherein, described frame of video comprises very first time stamp, and this very first time stamp is the capture time value of image corresponding to this frame of video; Described audio frame comprises the second timestamp, and this second timestamp is the capture time value of the sound that this audio frame is corresponding.A14, method according to any one of A8-A13, wherein said threshold value is 5.
In specification provided herein, describe a large amount of detail.But can understand, embodiments of the invention can be put into practice when not having these details.In some instances, be not shown specifically known method, structure and technology, so that not fuzzy understanding of this description.
Similarly, be to be understood that, in order to simplify the disclosure and to help to understand in each inventive aspect one or more, in the description above to exemplary embodiment of the present invention, each feature of the present invention is grouped together in single embodiment, figure or the description to it sometimes.But, the method for the disclosure should be construed to the following intention of reflection: namely the present invention for required protection requires than the feature more multiple features clearly recorded in each claim.Or rather, as claims below reflect, all features of disclosed single embodiment before inventive aspect is to be less than.Therefore, the claims following embodiment are incorporated to this embodiment thus clearly, and wherein each claim itself is as independent embodiment of the present invention.
Those skilled in the art are to be understood that the module of the equipment in example disclosed herein or unit or assembly can be arranged in equipment as depicted in this embodiment, or alternatively can be positioned in one or more equipment different from the equipment in this example.Module in aforementioned exemplary can be combined as a module or can be divided into multiple submodule in addition.
Those skilled in the art are appreciated that and adaptively can change the module in the equipment in embodiment and they are arranged in one or more equipment different from this embodiment.Module in embodiment or unit or assembly can be combined into a module or unit or assembly, and multiple submodule or subelement or sub-component can be put them in addition.Except at least some in such feature and/or process or unit be mutually repel except, any combination can be adopted to combine all processes of all features disclosed in this specification (comprising adjoint claim, summary and accompanying drawing) and so disclosed any method or equipment or unit.Unless expressly stated otherwise, each feature disclosed in this specification (comprising adjoint claim, summary and accompanying drawing) can by providing identical, alternative features that is equivalent or similar object replaces.
In addition, those skilled in the art can understand, although embodiments more described herein to comprise in other embodiment some included feature instead of further feature, the combination of the feature of different embodiment means and to be within scope of the present invention and to form different embodiments.Such as, in the following claims, the one of any of embodiment required for protection can use with arbitrary compound mode.
In addition, some in described embodiment are described as at this can by the processor of computer system or the method implemented by other device performing described function or the combination of method element.Therefore, there is the device of processor formation for implementing the method or method element of the necessary instruction for implementing described method or method element.In addition, the element described herein of device embodiment is the example as lower device: this device is for implementing the function performed by the element of the object in order to implement this invention.
As used in this, unless specifically stated so, use ordinal number " first ", " second ", " the 3rd " etc. to describe plain objects and only represent the different instances relating to similar object, and be not intended to imply the object be described like this must have the time upper, spatially, sequence aspect or in any other manner to definite sequence.
Although the embodiment according to limited quantity describes the present invention, benefit from description above, those skilled in the art understand, in the scope of the present invention described thus, it is contemplated that other embodiment.In addition, it should be noted that the language used in this specification is mainly in order to object that is readable and instruction is selected, instead of select to explain or limiting theme of the present invention.Therefore, when not departing from the scope and spirit of appended claims, many modifications and changes are all apparent for those skilled in the art.For scope of the present invention, be illustrative to disclosing of doing of the present invention, and nonrestrictive, and scope of the present invention is defined by the appended claims.
Claims (10)
1. a media termination, comprising:
Video acquisition unit, is suitable for gathering image and being encoded to frame of video;
Screen buffer, is suitable for depositing the frame of video from described video acquisition unit;
Audio collection unit, is suitable for gathering sound and being encoded to audio frame;
Audio buffer, is suitable for depositing the audio frame from described audio collection unit;
Send buffering area, be suitable for depositing one or more Frame to be sent, wherein each Frame to be sent is from the frame of video of described screen buffer or the audio frame from described audio buffer;
Transmission unit, is suitable for described one or more data frame transfer to be sent to media server; And
Control unit, is suitable for detecting audio buffer and is pushed to by audio frame wherein in transmission buffering area, and when being empty in audio buffer, judges whether that the quantity of Frame to be sent in transmission buffering area does not exceed threshold value and screen buffer is not empty,
If the quantity of Frame to be sent do not exceed threshold value and screen buffer for empty, then from this screen buffer, extract a frame frame of video and be pushed to and send in buffering area.
2. media termination as claimed in claim 1, wherein,
Described control unit is also suitable for when detecting that the quantity sending Frame to be sent in buffering area is greater than threshold value, detects transmitting element to the transmission code check of frame of video, and generates rate adjust parameter according to this transmission code check; And
Described video acquisition unit is suitable for the generation code check regulating frame of video according to this rate adjust parameter.
3. media termination as claimed in claim 1 or 2, wherein, described screen buffer, described audio buffer and described transmission buffering area are loop buffer.
4. the media termination according to any one of claim 1-3, wherein, described control unit is suitable for judging whether that the quantity of Frame to be sent in transmission buffering area does not exceed threshold value and screen buffer is not sky according to following manner:
Whether be empty, if be not empty, whether the quantity continuing to detect Frame to be sent in described transmission buffering area exceedes threshold value if detecting screen buffer.
5. the media termination according to any one of claim 1-3, wherein, described control unit is suitable for judging whether that the quantity of Frame to be sent in transmission buffering area does not exceed threshold value and screen buffer is not sky according to following manner:
Whether the quantity detecting Frame to be sent in described transmission buffering area exceedes threshold value, and whether when not exceeding threshold value, detecting screen buffer is empty.
6. the media termination according to any one of claim 1-5, wherein,
Described frame of video comprises very first time stamp, and this very first time stabs the capture time value for image corresponding to this frame of video;
Described audio frame comprises the second timestamp, and this second timestamp is the capture time value of the sound that this audio frame is corresponding.
7. the media termination according to any one of claim 1-6, wherein, described threshold value is 5.
8. gather a method for media data, be suitable for performing in media termination, this media termination comprises transmission buffering area, and be suitable for depositing one or more Frame to be sent, wherein each Frame to be sent is frame of video or audio frame, and the method comprises:
Gather image and be encoded to frame of video, and this frame of video is stored in screen buffer;
Gather sound and be encoded to audio frame, and this audio frame is deposited in audio buffer;
Detecting audio buffer and being pushed to by audio frame wherein sends in buffering area;
If audio buffer is empty, then judge whether that the quantity of Frame to be sent in transmission buffering area does not exceed threshold value and screen buffer is not sky,
If the quantity of Frame to be sent do not exceed threshold value and screen buffer for empty, then from this screen buffer, extract a frame frame of video and be pushed to and send in buffering area; And
By described one or more data frame transfer to be sent to media server.
9. method as claimed in claim 8, also comprises:
When detecting that the quantity sending Frame to be sent in buffering area is greater than threshold value, detecting transmitting element to the transmission code check of frame of video, and generating rate adjust parameter according to this transmission code check; And
The generation code check of frame of video is regulated according to this rate adjust parameter.
10. a music lesson system, comprising:
Media termination according to any one of claim 1-7;
Media server, is suitable for the audio frame and the frame of video that receive the transmission of described media termination; And
Media play end, is suitable for obtaining from this media server and playing described audio frame and frame of video.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510846324.1A CN105429983B (en) | 2015-11-27 | 2015-11-27 | Acquire method, media termination and the music lesson system of media data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510846324.1A CN105429983B (en) | 2015-11-27 | 2015-11-27 | Acquire method, media termination and the music lesson system of media data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105429983A true CN105429983A (en) | 2016-03-23 |
| CN105429983B CN105429983B (en) | 2018-09-14 |
Family
ID=55507925
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510846324.1A Active CN105429983B (en) | 2015-11-27 | 2015-11-27 | Acquire method, media termination and the music lesson system of media data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105429983B (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106791575A (en) * | 2016-12-22 | 2017-05-31 | 上海寰视网络科技有限公司 | The control method and equipment of a kind of data is activation |
| CN107566769A (en) * | 2017-09-27 | 2018-01-09 | 维沃移动通信有限公司 | A kind of video recording method and mobile terminal |
| CN109361945A (en) * | 2018-10-18 | 2019-02-19 | 广州市保伦电子有限公司 | The meeting audiovisual system and its control method of a kind of quick transmission and synchronization |
| CN110535810A (en) * | 2018-05-25 | 2019-12-03 | 视联动力信息技术股份有限公司 | A kind of processing method and terminal of video data |
| CN111182220A (en) * | 2020-01-09 | 2020-05-19 | 西安万像电子科技有限公司 | Image processing apparatus, remote device, and communication system |
| CN111585684A (en) * | 2020-05-14 | 2020-08-25 | 武汉大学 | Multi-path camera time alignment method and system for networked monitoring video analysis |
| CN112689160A (en) * | 2020-11-27 | 2021-04-20 | 烟台艾睿光电科技有限公司 | Video transmission method and device applied to image acquisition equipment |
| CN113905266A (en) * | 2021-10-11 | 2022-01-07 | 杭州萤石软件有限公司 | Audio and video data processing method, device, equipment and storage medium |
| WO2022218165A1 (en) * | 2021-04-13 | 2022-10-20 | Oppo广东移动通信有限公司 | Method and apparatus for receiving audio data packet, method and apparatus for sending audio data packet, and electronic device |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1481547A (en) * | 2000-12-15 | 2004-03-10 | ���˹���Ѷ��� | Encoding audio signals |
| CN1744592A (en) * | 2005-09-29 | 2006-03-08 | 西安交通大学 | Multi-source flow media transmission QoS control method |
| CN101030938A (en) * | 2007-02-05 | 2007-09-05 | 北京大学 | System and method for organizing and transmitting QoS self-adaptive P2P flow medium data |
| CN101340590A (en) * | 2008-08-14 | 2009-01-07 | 北京创毅视讯科技有限公司 | Multiplex apparatus and multiplex method |
| CN101479686A (en) * | 2006-06-27 | 2009-07-08 | 皇家飞利浦电子股份有限公司 | A device and a method for managing power consumption of a plurality of data processing units |
| CN102223516A (en) * | 2010-04-14 | 2011-10-19 | 奥多比公司 | Media quality enhancement between connected media communication devices |
-
2015
- 2015-11-27 CN CN201510846324.1A patent/CN105429983B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1481547A (en) * | 2000-12-15 | 2004-03-10 | ���˹���Ѷ��� | Encoding audio signals |
| CN1744592A (en) * | 2005-09-29 | 2006-03-08 | 西安交通大学 | Multi-source flow media transmission QoS control method |
| CN101479686A (en) * | 2006-06-27 | 2009-07-08 | 皇家飞利浦电子股份有限公司 | A device and a method for managing power consumption of a plurality of data processing units |
| CN101030938A (en) * | 2007-02-05 | 2007-09-05 | 北京大学 | System and method for organizing and transmitting QoS self-adaptive P2P flow medium data |
| CN101340590A (en) * | 2008-08-14 | 2009-01-07 | 北京创毅视讯科技有限公司 | Multiplex apparatus and multiplex method |
| CN102223516A (en) * | 2010-04-14 | 2011-10-19 | 奥多比公司 | Media quality enhancement between connected media communication devices |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106791575A (en) * | 2016-12-22 | 2017-05-31 | 上海寰视网络科技有限公司 | The control method and equipment of a kind of data is activation |
| CN106791575B (en) * | 2016-12-22 | 2019-11-15 | 上海寰视网络科技有限公司 | A kind of control method and equipment that data are sent |
| CN107566769A (en) * | 2017-09-27 | 2018-01-09 | 维沃移动通信有限公司 | A kind of video recording method and mobile terminal |
| CN110535810A (en) * | 2018-05-25 | 2019-12-03 | 视联动力信息技术股份有限公司 | A kind of processing method and terminal of video data |
| CN109361945A (en) * | 2018-10-18 | 2019-02-19 | 广州市保伦电子有限公司 | The meeting audiovisual system and its control method of a kind of quick transmission and synchronization |
| CN111182220A (en) * | 2020-01-09 | 2020-05-19 | 西安万像电子科技有限公司 | Image processing apparatus, remote device, and communication system |
| WO2021139418A1 (en) * | 2020-01-09 | 2021-07-15 | 西安万像电子科技有限公司 | Image processing apparatus, remote device, and communication system |
| CN111585684A (en) * | 2020-05-14 | 2020-08-25 | 武汉大学 | Multi-path camera time alignment method and system for networked monitoring video analysis |
| CN112689160A (en) * | 2020-11-27 | 2021-04-20 | 烟台艾睿光电科技有限公司 | Video transmission method and device applied to image acquisition equipment |
| WO2022218165A1 (en) * | 2021-04-13 | 2022-10-20 | Oppo广东移动通信有限公司 | Method and apparatus for receiving audio data packet, method and apparatus for sending audio data packet, and electronic device |
| CN113905266A (en) * | 2021-10-11 | 2022-01-07 | 杭州萤石软件有限公司 | Audio and video data processing method, device, equipment and storage medium |
| CN113905266B (en) * | 2021-10-11 | 2024-04-30 | 杭州萤石软件有限公司 | Audio and video data processing method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105429983B (en) | 2018-09-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105429983A (en) | Media data acquisition method, media terminal and music teaching system | |
| US11240543B2 (en) | Synchronizing playback of segmented video content across multiple video playback devices | |
| KR101396628B1 (en) | Streaming encoded video data | |
| US9521469B2 (en) | Carriage of quality information of content in media formats | |
| CN105430537A (en) | Method and server for synthesis of multiple paths of data, and music teaching system | |
| CN101917613B (en) | Acquiring and coding service system of streaming media | |
| KR100800716B1 (en) | Apparatus and method for tranmitting and receiving moving picture data using a close range communication | |
| KR20160110472A (en) | Streaming multiple encodings encoded using different encoding parameters | |
| CN102811368A (en) | Mobile video live broadcasting system | |
| CN104620595A (en) | Proactive video frame dropping | |
| US7619645B2 (en) | Audio visual media encoding system | |
| CN105072506B (en) | The transmission method and system of live teaching broadcast video | |
| CN105516090A (en) | Media play method, device and music teaching system | |
| CN104038844A (en) | Mobile live broadcast system based on MPEG-2 standard | |
| CN105429984A (en) | Media play method, equipment and music teaching system | |
| CN105430453B (en) | Medium data acquisition method, media termination and Online Music tutoring system | |
| CN108881931B (en) | Data buffering method and network equipment | |
| KR20150003316A (en) | Method and device for evaluating media quality | |
| CN113892274B (en) | Synchronization control device, synchronization control method, and synchronization control program | |
| JP2020198509A (en) | Synchronous control device, synchronous control method, and synchronous control program | |
| CN109040818B (en) | Audio and video synchronization method, storage medium, electronic equipment and system during live broadcasting | |
| JP2005086362A (en) | Data multiplexing method, data transmitting method and data receiving method | |
| JP6193569B2 (en) | RECEPTION DEVICE, RECEPTION METHOD, AND PROGRAM, IMAGING DEVICE, IMAGING METHOD, AND PROGRAM, TRANSMISSION DEVICE, TRANSMISSION METHOD, AND PROGRAM | |
| Hoppe et al. | Video Streaming over IP using the DASH Technique | |
| JP6400163B2 (en) | Reception device, reception method, transmission device, transmission method, and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |