CN112702559A - Recorded broadcast abnormity feedback method, system, equipment and readable storage medium - Google Patents
Recorded broadcast abnormity feedback method, system, equipment and readable storage medium Download PDFInfo
- Publication number
- CN112702559A CN112702559A CN202110304631.2A CN202110304631A CN112702559A CN 112702559 A CN112702559 A CN 112702559A CN 202110304631 A CN202110304631 A CN 202110304631A CN 112702559 A CN112702559 A CN 112702559A
- Authority
- CN
- China
- Prior art keywords
- video
- recording
- stream
- recorded broadcast
- broadcasting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 43
- 230000002159 abnormal effect Effects 0.000 claims abstract description 97
- 230000005540 biological transmission Effects 0.000 claims abstract description 48
- 230000015654 memory Effects 0.000 claims description 21
- 238000004088 simulation Methods 0.000 claims description 9
- 230000004927 fusion Effects 0.000 description 28
- 230000011664 signaling Effects 0.000 description 13
- 238000010586 diagram Methods 0.000 description 11
- 238000004590 computer program Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 5
- 230000005856 abnormality Effects 0.000 description 4
- 230000010365 information processing Effects 0.000 description 4
- 238000011084 recovery Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000011144 upstream manufacturing Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005538 encapsulation Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/14—Systems for two-way working
- H04N7/15—Conference systems
- H04N7/155—Conference systems involving storage of or access to video conference sessions
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/23424—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
- H04N21/4425—Monitoring of client processing errors or hardware failure
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/76—Television signal recording
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/76—Television signal recording
- H04N5/91—Television signal processing therefor
- H04N5/92—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback
- H04N5/9201—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback involving the multiplexing of an additional signal and the video signal
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Business, Economics & Management (AREA)
- Marketing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Databases & Information Systems (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
The invention discloses a recording and broadcasting abnormal feedback method, a system, equipment and a readable storage medium, wherein the method comprises the steps of establishing a media stream transmission channel between recording and broadcasting services, sending an audio-mixed audio stream and a fused video stream to the recording and broadcasting services through the media stream transmission channel, enabling the recording and broadcasting services to write the audio-mixed audio stream and the fused video stream into a video file and write index information of the audio-mixed audio stream and the fused video stream into a video index file, receiving a recording and broadcasting abnormal message sent by the recording and broadcasting services through the media stream transmission channel, sending the recording and broadcasting abnormal message to an SFU (small form-factor user) to enable the SFU to feed back the recording and broadcasting abnormal message to a video conference terminal and/or a video conference control platform, multiplexing the media stream transmission channel of the recording and broadcasting services to send the recording and broadcasting abnormal message in the recording and broadcasting services to the SFU, and realizing very concise timely feedback of the recording and broadcasting abnormal message to a user under the condition that the recording and broadcasting services do not enter a conference, the reliability of the recorded broadcast service can be greatly improved.
Description
Technical Field
The present application relates to the field of recorded broadcast technologies, and in particular, to a recorded broadcast exception feedback method, system, device, and readable storage medium.
Background
With the popularization of the mobile internet, the demand of the video conference is more and more increased. In a video conference, each participating terminal can acquire images and sound, and send the images and sound to a Selective Forwarding Unit (SFU) after encoding, and the SFU sends the images and sound sent by each terminal to other terminals according to conference requirements, so that audio and video calls of multi-party participants are realized. In an important video conference, a user often records and broadcasts the video conference, and the recording and broadcasting service enters or does not enter the conference to acquire media streams required by recording and broadcasting for recording and broadcasting, so that the reliability of the recording and broadcasting service is extremely important. However, some failures in the recording and broadcasting service can not be completely avoided, for example, during the recording process, failures such as a disk write error, a full disk space, a disk disconnection, a file system error and the like may occur, and if the failures are not fed back to the user in time, the recording data may be lost. For the case that the recorded broadcast service does not enter the conference, the recorded broadcast service does not have an identity, and therefore the fault in the recorded broadcast service cannot be reported to the SFU, and the fault in the recorded broadcast service cannot be fed back to the user.
Disclosure of Invention
The embodiment of the invention provides a recording and broadcasting abnormity feedback method, a system, equipment and a readable storage medium, which are used for solving the technical problem that a fault in a video conference recording and broadcasting service cannot be timely fed back to a user and video data can be lost under the condition that the recording and broadcasting service does not enter a meeting.
In order to solve the above problem, in a first aspect, an embodiment of the present invention provides a recorded broadcast exception feedback method, including: acquiring multiple paths of video streams and multiple paths of audio streams of the video conference from the SFU; the method comprises the steps of fusing multiple audio streams to obtain a mixed audio stream, and fusing multiple video streams to obtain a fused video stream; creating a media stream transmission channel between the recorded broadcast service and the recorded broadcast service, and sending the audio mixing audio stream and the fused video stream to the recorded broadcast service through the media stream transmission channel so that the recorded broadcast service writes the audio mixing audio stream and the fused video stream into a video file and writes the index information of the audio mixing audio stream and the fused video stream into a video index file; and receiving recorded broadcast abnormal messages sent by the recorded broadcast service through a media stream transmission channel, and sending the recorded broadcast abnormal messages to the SFU, so that the SFU feeds back the recorded broadcast abnormal messages to the video conference terminal and/or the video conference control platform.
Optionally, the recorded broadcast exception message includes: and the abnormal writing state of the video recording file when the audio mixing audio stream and the fusion video stream are written into the video recording file and/or the abnormal writing state of the video recording index file when the index information of the audio mixing audio stream and the fusion video stream is written into the video recording index file.
Optionally, after the mixed audio stream and the fused video stream are sent to the recording and playing service through the media stream transmission channel, the recording and playing abnormality feedback method further includes: periodically acquiring the data volume of the video files from the recording and broadcasting service; calculating the video recording duration according to the video recording starting time and the current time; and sending the data volume, the video recording time length and the recording and broadcasting abnormal message of the video recording file to the SFU, so that the SFU feeds back the data volume, the video recording time length and the recording and broadcasting abnormal message of the video recording file to the video conference terminal and/or the video conference control platform.
Optionally, a media streaming channel is created with the recorded broadcast service via a proprietary protocol.
In a second aspect, an embodiment of the present invention provides a recording and broadcasting exception feedback system, including: the system comprises an analog terminal, a plug flow client and a plug flow server; the simulation terminal is used for acquiring multiple audio streams and multiple video streams of the video conference from the SFU, fusing the multiple audio streams to obtain one mixed audio stream, fusing the multiple video streams to obtain one fused video stream, and outputting the mixed audio stream and the fused video stream to the plug-flow client; the stream pushing client is used for receiving the audio mixing audio stream and the fused video stream, creating a media stream transmission channel between the stream pushing client and the stream pushing server, and sending the audio mixing audio stream and the fused video stream to the stream pushing server through the media stream transmission channel; the stream pushing server is used for receiving the audio mixing audio stream and the fusion video stream, writing the audio mixing audio stream and the fusion video stream into a video file, writing the index information of the audio mixing audio stream and the fusion video stream into a video index file, acquiring a recording and broadcasting abnormal message, and sending the recording and broadcasting abnormal message to the stream pushing client through a media stream transmission channel; the stream pushing client is also used for receiving the recorded broadcast abnormal message and outputting the recorded broadcast abnormal message to the simulation terminal; the simulation terminal is further used for receiving the recorded broadcast abnormal message and sending the recorded broadcast abnormal message to the SFU, so that the SFU feeds back the recorded broadcast abnormal message to the video conference terminal and/or the video conference control platform.
Optionally, the recorded broadcast exception message includes: and the abnormal writing state of the video recording file when the audio mixing audio stream and the fusion video stream are written into the video recording file and/or the abnormal writing state of the video recording index file when the index information of the audio mixing audio stream and the fusion video stream is written into the video recording index file.
Optionally, the stream pushing client is further configured to periodically obtain the data volume of the video file from the stream pushing server, calculate the video recording duration according to the video recording start time and the current time, and output the data volume of the video file and the video recording duration to the analog terminal; the simulation terminal is further used for receiving the data volume and the video recording time length of the video recording file and sending the data volume, the video recording time length and the recording and broadcasting abnormal message of the video recording file to the SFU, so that the SFU can feed back the data volume, the video recording time length and the recording and broadcasting abnormal message of the video recording file to the video conference terminal and/or the video conference control platform.
Optionally, the push streaming client creates a media streaming channel with the push streaming server through a private protocol.
In a third aspect, an embodiment of the present invention provides an electronic device, including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executable by the at least one processor to cause the at least one processor to perform the recorded broadcast exception feedback method as in the first aspect or any embodiment of the first aspect.
In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, where computer instructions are stored, and the computer instructions are configured to cause a computer to execute the recorded broadcast exception feedback method according to the first aspect or any implementation manner of the first aspect.
The recorded broadcast abnormal feedback method, system, device and readable storage medium provided by the embodiments of the present invention obtain multiple video streams and multiple audio streams of a video conference from an SFU, fuse the multiple audio streams to obtain a mixed audio stream, fuse the multiple video streams to obtain a fused video stream, create a media stream transmission channel between the recorded broadcast service and the recorded broadcast service, and send the mixed audio stream and the fused video stream to the recorded broadcast service through the media stream transmission channel, so that the recorded broadcast service writes the mixed audio stream and the fused video stream into a video file and writes index information of the mixed audio stream and the fused video stream into a video index file, receives a recorded broadcast abnormal message sent by the recorded broadcast service through the media stream transmission channel, and sends the recorded broadcast abnormal message to the SFU, so that the SFU feeds back the recorded broadcast abnormal message to a video conference terminal and/or a video conference control platform, therefore, the recording and broadcasting abnormal information in the recording and broadcasting service is sent to the SFU by multiplexing the media stream transmission channel of the recording and broadcasting service, the recording and broadcasting abnormal information can be fed back to a user in time in a very concise manner under the condition that the recording and broadcasting service does not enter a meeting, and the reliability of the recording and broadcasting service can be greatly improved; no additional virtual terminal is added to the recording and broadcasting service to interact with the SFU, so that the additional pressure of signaling on the SFU can be avoided; and an additional abnormal information feedback channel is not added to the recorded broadcast service, so that the pressure of the recorded broadcast service is not increased.
The foregoing description is only an overview of the technical solutions of the present application, and the present application can be implemented according to the content of the description in order to make the technical means of the present application more clearly understood, and the following detailed description of the present application is given in order to make the above and other objects, features, and advantages of the present application more clearly understandable.
Drawings
Fig. 1 is a schematic flow chart of a recording and broadcasting abnormality feedback method according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of a video conference system according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a recording and broadcasting abnormality feedback system according to an embodiment of the present invention;
fig. 4 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The embodiment of the invention provides a recording and broadcasting abnormity feedback method, as shown in fig. 1, the method comprises the following steps:
s101, obtaining multiple video streams and multiple audio streams of the video conference from the SFU.
Specifically, the video conference system structure is shown in fig. 2, and includes a plurality of video conference terminals 14, SFUs 15, media stream processing devices 16, recording and playing services 17, video recording storage 18, and the like, where the SFUs include signaling services 151 and media services 152. The recording and broadcasting abnormity feedback method provided by the embodiment of the invention is applied to a media stream processing device of a video conference system. In the video conference, the logic of the signaling service is as follows: 1. when the video conference starts recording, an analog terminal ID is generated for the video conference, the analog terminal ID is called RecordID, and one RecordID is generated for one video conference. 2. And adding a recorded broadcast state reporting interface for reporting the recorded broadcast abnormal message to the signaling service 151 by the media stream processing device 16. 3. And adding a recorded broadcast state notification message for sending the notification to the video conference terminal and/or the video conference control platform which have subscribed the message when the recorded broadcast abnormal message is changed. When the videoconference needs to be recorded, the signaling service 151 may send a recordd to the media stream processing device 16, and the media stream processing device 16 may use the recordd to start an analog terminal, initiate a registration session with the SFU15, and establish a connection channel with the media service 152. Then, a signaling interface is called to acquire an audio stream synchronization information source (SSRC) and multi-picture layout information required for recording and playing and a multi-channel video stream SSRC of the video conference from a signaling service 151 of the SFU15, the video stream SSRC required for recording and playing is determined according to a preset strategy, the multi-picture layout information and the multi-channel video stream SSRC of the video conference, and then a multi-channel audio stream corresponding to the audio stream SSRC required for recording and playing and a multi-channel video stream corresponding to the video stream SSRC required for recording and playing are acquired from a media service 152 of the SFU 15.
S102, fusing multiple audio streams to obtain one audio mixing audio stream, and fusing multiple video streams to obtain one fused video stream; specifically, a channel of mixed audio stream is obtained by fusing multiple channels of audio streams, and a channel of fused video stream is obtained by fusing multiple channels of video streams, so that the channel of mixed audio stream and the channel of fused video stream received by the recording and playing service 17 are standardized media streams, and the recording and playing service 17 can support the browser plug-in-free recording and playing.
S103, a media stream transmission channel between the recorded broadcast service and the recorded broadcast service is established, and the audio mixing audio stream and the fused video stream are sent to the recorded broadcast service through the media stream transmission channel, so that the recorded broadcast service writes the audio mixing audio stream and the fused video stream into a video file and writes index information of the audio mixing audio stream and the fused video stream into a video index file; specifically, a media streaming channel with the recording and playing service 17 may be created through a specific protocol, and the upstream data of the media streaming channel may include: recording and broadcasting request, audio mixing audio stream and fusion video stream. And then sends a recording request to the recording and broadcasting service 17 through the media streaming channel, and sends the mixed audio stream and the fused video stream to the recording and broadcasting service 17 through the media streaming channel. The recording and broadcasting service 17 encapsulates the received audio mixing audio stream and the fusion video stream after receiving the recording and broadcasting request, the audio mixing audio stream and the fusion video stream, generates index information of the audio mixing audio stream and the fusion video stream after encapsulation, writes the encapsulated audio mixing audio stream and the fusion video stream into a video file in the video storage 18, and writes the index information of the audio mixing audio stream and the fusion video stream into a video index file in the video storage 18.
And S104, receiving recorded broadcast abnormal messages sent by the recorded broadcast service through the media stream transmission channel, and sending the recorded broadcast abnormal messages to the SFU, so that the SFU feeds back the recorded broadcast abnormal messages to the video conference terminal and/or the video conference control platform. Specifically, the recorded broadcast exception message includes a recorded broadcast error message and a recorded broadcast error recovery message. The recording and broadcasting service 17 may acquire and record the writing status of the video file and the video index file in real time, update the recorded writing status in real time, and then check the recorded writing status according to a preset time period. If the writing state of the current video file and/or the writing state of the video index file are abnormal, determining a recording and broadcasting error; if the writing state of the current video file and the writing state of the video index file are normal, and the writing state of the video file and/or the writing state of the video index file in the last time period are abnormal, determining that the recorded broadcast error is recovered; then a recorded broadcast exception or recorded broadcast exception recovery message may be sent to signaling service 151 of SFU15 to cause signaling service 151 of SFU15 to feed back the recorded broadcast exception message to videoconferencing endpoint 14 and/or the videoconferencing control platform.
The recording and broadcasting abnormity feedback method provided by the embodiment of the invention comprises the steps of acquiring a plurality of video streams and a plurality of audio streams of a video conference from an SFU, fusing the plurality of audio streams to obtain a mixed audio stream, fusing the plurality of video streams to obtain a fused video stream, creating a media stream transmission channel between the recording and broadcasting service, sending the mixed audio stream and the fused video stream to the recording and broadcasting service through the media stream transmission channel, so that the recording and broadcasting service writes the mixed audio stream and the fused video stream into a video file and writes index information of the mixed audio stream and the fused video stream into a video index file, receiving recording and broadcasting abnormity information sent by the recording and broadcasting service through the media stream transmission channel, sending the recording and broadcasting abnormity information to the SFU, so that the SFU feeds back the recording and broadcasting abnormity information to a video conference terminal and/or a video conference control platform, and multiplexing the media stream transmission channel of the recording and broadcasting abnormity information in the recording and broadcasting service to the SFU, the recording and broadcasting abnormal information can be fed back to the user very simply and timely under the condition that the recording and broadcasting service does not enter a meeting, so that the reliability of the recording and broadcasting service can be greatly improved; no additional virtual terminal is added to the recording and broadcasting service to interact with the SFU, so that the additional pressure of signaling on the SFU can be avoided; and an additional abnormal information feedback channel is not added to the recorded broadcast service, so that the pressure of the recorded broadcast service is not increased.
In an alternative embodiment, the recorded broadcast exception message includes: and the abnormal writing state of the video recording file when the audio mixing audio stream and the fusion video stream are written into the video recording file and/or the abnormal writing state of the video recording index file when the index information of the audio mixing audio stream and the fusion video stream is written into the video recording index file.
Specifically, the abnormal writing state of the video file when the audio mixing stream and the fusion video stream are written into the video file and/or the abnormal writing state of the video index file when the index information of the audio mixing stream and the fusion video stream is written into the video index file includes: disk write errors, disk space fullness, disk drop, file system errors, error recovery, and the like. Preferably, the writing status can be expressed by an assigned method, for example, for a normal writing status, the normal writing status can be expressed by zero, for an abnormal writing status, the writing status with full disk space can be expressed by 1, the writing status with a disk writing error can be expressed by 2, and the like.
The abnormal writing state of the video file when the audio mixing audio stream and the fusion video stream are written into the video file and/or the abnormal writing state of the video index file when the index information of the audio mixing audio stream and the fusion video stream is written into the video index file are set in the recorded broadcast abnormal message, so that a user can quickly know the error type in the recorded broadcast process, and the problem in the recorded broadcast process is pertinently solved.
In an optional embodiment, after the mixed audio stream and the merged video stream are sent to the recording service through the media stream transmission channel, the recording anomaly feedback method further includes: periodically acquiring the data volume of the video files from the recording and broadcasting service; calculating the video recording duration according to the video recording starting time and the current time; and sending the data volume, the video recording time length and the recording and broadcasting abnormal message of the video recording file to the SFU, so that the SFU feeds back the data volume, the video recording time length and the recording and broadcasting abnormal message of the video recording file to the video conference terminal and/or the video conference control platform.
Specifically, the media stream processing device 16 may further record when receiving the recording and broadcasting abnormal message sent by the recording and broadcasting service 17 through the media stream transmission channel, update the recorded recording and broadcasting abnormal message in real time, periodically call a video file size query interface of the recording and broadcasting service 17, obtain the size of the current video file from the recording and broadcasting service 17, calculate the video recording duration according to the video recording start time and the current time, and send the data volume, the video recording duration, and the recording and broadcasting abnormal message of the video file to the SFU15, so that the SFU15 feeds back the data volume, the video recording duration, and the recording and broadcasting abnormal message of the video file to the video conference terminal and/or the video conference control platform.
In the embodiment of the invention, the data volume of the video file is periodically acquired from the recording and broadcasting service; calculating the video recording duration according to the video recording starting time and the current time; the data volume, the video recording time length and the recorded broadcast abnormal message of the video recording file are sent to the SFU, so that the SFU can feed back the data volume, the video recording time length and the recorded broadcast abnormal message of the video recording file to the video conference terminal and/or the video conference control platform, the data volume, the video recording time length and the recorded broadcast abnormal message of the video recording file can be periodically fed back to the video conference terminal and/or the video conference control platform, the recorded broadcast abnormal message cannot be frequently fed back, and the user experience can be improved.
In an alternative embodiment, the media streaming channel with the recorded broadcast service may be created through a proprietary protocol.
In particular, the private protocol may be implemented based on any protocol implemented by TCP, as well as HTTP protocol. The private protocol upstream data includes: the recording and broadcasting request and the audio and video media stream data comprise recording and broadcasting abnormal information in a downlink mode. The media stream transmission channel between the recording and broadcasting service 17 is created through the private protocol, so that the media stream transmission channel can realize the transmission of the media stream and the transmission of the recording and broadcasting abnormal message, and an abnormal information feedback channel does not need to be additionally created between the recording and broadcasting service 17 and the media stream processing device 16, and the pressure of the recording and broadcasting service 17 is not increased.
An embodiment of the present invention provides a recording and broadcasting abnormality feedback system, as shown in fig. 3, including:
the system comprises an analog terminal 11, a push streaming client 12 and a push streaming server 13.
Specifically, the analog terminal 11 and the push streaming client 12 may be provided in the media stream processing device 16, and the push streaming server 13 may be provided in the recording and broadcasting service 17.
The analog terminal 11 is configured to obtain multiple audio streams and multiple video streams of the video conference from the SFU15, merge the multiple audio streams to obtain one mixed audio stream, merge the multiple video streams to obtain one merged video stream, and output the mixed audio stream and the merged video stream to the stream pushing client 12.
The detailed description of the analog terminal 11 is described in steps S101-S102 in the above embodiments, and is not repeated herein.
The push streaming client 12 is configured to receive the mixed audio stream and the fused video stream, create a media stream transmission channel with the push streaming server 13, and send the mixed audio stream and the fused video stream to the push streaming server 13 through the media stream transmission channel. The detailed description of the push flow client 12 is described in step S103 in the above embodiment, and is not repeated herein.
The push streaming server 13 is configured to receive the audio mixing audio stream and the fusion video stream, write the audio mixing audio stream and the fusion video stream into a video file, write index information of the audio mixing audio stream and the fusion video stream into a video index file, acquire an abnormal recording and broadcasting message, and send the abnormal recording and broadcasting message to the push streaming client 12 through a media stream transmission channel.
The stream pushing client 12 is further configured to receive the recorded broadcast abnormal message, and output the recorded broadcast abnormal message to the analog terminal 11.
The analog terminal 11 is further configured to receive the recorded broadcast exception message, and send the recorded broadcast exception message to the SFU15, so that the SFU15 feeds back the recorded broadcast exception message to the video conference terminal 14 and/or the video conference control platform. The detailed description of the analog terminal 11 is described in step S104 in the above embodiment, and is not repeated herein.
The recording and broadcasting abnormity feedback system provided by the embodiment of the invention obtains a plurality of video streams and a plurality of audio streams of a video conference from an SFU, fuses the plurality of audio streams to obtain a sound mixing audio stream, fuses the plurality of video streams to obtain a fused video stream, creates a media stream transmission channel between the recording and broadcasting service, and sends the sound mixing audio stream and the fused video stream to the recording and broadcasting service through the media stream transmission channel, so that the recording and broadcasting service writes the sound mixing audio stream and the fused video stream into a video file and writes index information of the sound mixing audio stream and the fused video stream into a video index file, receives recording and broadcasting abnormity information sent by the recording and broadcasting service through the media stream transmission channel, and sends the recording and broadcasting abnormity information to the SFU, so that the SFU feeds back the recording and broadcasting abnormity information to a video conference terminal and/or a video conference control platform, and then multiplexes the media stream transmission channel of the recording and broadcasting abnormity information in the recording and broadcasting service to the SFU, the recording and broadcasting abnormal information can be fed back to the user very simply and timely under the condition that the recording and broadcasting service does not enter a meeting, so that the reliability of the recording and broadcasting service can be greatly improved; no additional virtual terminal is added to the recording and broadcasting service to interact with the SFU, so that the additional pressure of signaling on the SFU can be avoided; and an additional abnormal information feedback channel is not added to the recorded broadcast service, so that the pressure of the recorded broadcast service is not increased.
In an alternative embodiment, the recorded broadcast exception message includes: and the abnormal writing state of the video recording file when the audio mixing audio stream and the fusion video stream are written into the video recording file and/or the abnormal writing state of the video recording index file when the index information of the audio mixing audio stream and the fusion video stream is written into the video recording index file.
Specifically, the abnormal writing state of the video file when the audio mixing stream and the fusion video stream are written into the video file and/or the abnormal writing state of the video index file when the index information of the audio mixing stream and the fusion video stream is written into the video index file includes: disk write errors, disk space fullness, disk drop, file system errors, error recovery, and the like. Preferably, the writing status can be expressed by an assigned method, for example, for a normal writing status, the normal writing status can be expressed by zero, for an abnormal writing status, the writing status with full disk space can be expressed by 1, the writing status with a disk writing error can be expressed by 2, and the like.
The abnormal writing state of the video file when the audio mixing audio stream and the fusion video stream are written into the video file and/or the abnormal writing state of the video index file when the index information of the audio mixing audio stream and the fusion video stream is written into the video index file are set in the recorded broadcast abnormal message, so that a user can quickly know the error type in the recorded broadcast process, and the problem in the recorded broadcast process is pertinently solved.
In an optional embodiment, the stream pushing client 12 is further configured to periodically obtain the data volume of the video file from the stream pushing server 13, calculate the video recording duration according to the video recording start time and the current time, and output the data volume of the video file and the video recording duration to the analog terminal 11; the analog terminal 11 is further configured to receive the data volume and the recording duration of the video file, and send the data volume, the recording duration and the recording and broadcasting exception message of the video file to the SFU15, so that the SFU15 feeds back the data volume, the recording duration and the recording and broadcasting exception message of the video file to the video conference terminal and/or the video conference control platform.
Specifically, the stream pushing client 12 may further record when receiving a recorded broadcast abnormal message sent by the stream pushing server 13 through the media stream transmission channel, update the recorded broadcast abnormal message in real time, periodically call a video file size query interface of the stream pushing server 13, obtain the size of the current video file from the stream pushing server 13, calculate a video recording duration according to the video recording start time and the current time, and output the data volume of the video file and the video recording duration to the analog terminal 11; the analog terminal 11 is further configured to send the data volume, the video recording duration, and the recording and broadcasting exception message of the video recording file to the signaling service 151 of the SFU15, so that the signaling service 151 of the SFU15 feeds back the data volume, the video recording duration, and the recording and broadcasting exception message of the video recording file to the video conference terminal and/or the video conference control platform.
In the embodiment of the invention, the data volume of the video file is periodically acquired from the streaming server, the video recording duration is calculated according to the video recording starting time and the current time, the data volume of the video file and the video recording duration are output to the simulation terminal, and the simulation terminal sends the data volume of the video file, the video recording duration and the recording and broadcasting abnormal message to the SFU, so that the SFU feeds back the data volume of the video file, the video recording duration and the recording and broadcasting abnormal message to the video conference terminal and/or the video conference control platform, the data volume of the video file, the video recording duration and the recording and broadcasting abnormal message can be periodically fed back to the video conference terminal and/or the video conference control platform, the recording and broadcasting abnormal message can not be frequently fed back, and the user experience can be improved.
In an alternative embodiment, the push streaming client 12 creates a media streaming channel with the push streaming server 13 via a proprietary protocol.
In particular, the private protocol may be implemented based on any protocol implemented by TCP, as well as HTTP protocol. The private protocol upstream data includes: the recording and broadcasting request and the audio and video media stream data comprise recording and broadcasting abnormal information in a downlink mode. The push streaming client 12 creates a media stream transmission channel with the push streaming server 13 through a private protocol, so that the media stream transmission channel can realize both media stream transmission and recording and broadcasting abnormal message transmission, and thus no additional abnormal information feedback channel needs to be created between the push streaming client 12 and the push streaming server 13, and the pressure of recording and broadcasting service is not increased.
Based on the same inventive concept as the recorded broadcast exception feedback method in the foregoing embodiment, the present invention further provides an electronic device, as shown in fig. 4, including: a processor 31 and a memory 32, wherein the processor 31 and the memory 32 may be connected by a bus or other means, and the connection by the bus is illustrated in fig. 4 as an example.
The processor 31 may be a Central Processing Unit (CPU). The Processor 31 may also be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, or combinations thereof.
The memory 32 is a non-transitory computer readable storage medium, and can be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as program instructions/modules corresponding to the recorded broadcast exception feedback method in the embodiment of the present invention. The processor 31 executes various functional applications and data processing of the processor by running non-transitory software programs, instructions and modules stored in the memory 32, that is, implements the recording and broadcasting exception feedback method in the above method embodiment.
The memory 32 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created by the processor 31, and the like. Further, the memory 32 may include high speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory 32 may optionally include memory located remotely from the processor 31, and these remote memories may be connected to the processor 31 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
One or more of the modules described above are stored in memory 32 and, when executed by processor 31, perform the recorded broadcast exception feedback method of the embodiment shown in fig. 1.
The details of the electronic device may be understood with reference to the corresponding related description and effects in the embodiment shown in fig. 1, and are not described herein again.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a Flash Memory (Flash Memory), a Hard Disk (Hard Disk Drive, abbreviated as HDD), a Solid State Drive (SSD), or the like; the storage medium may also comprise a combination of memories of the kind described above.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable information processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable information processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable information processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable information processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (10)
1. A recording and broadcasting abnormity feedback method is characterized by comprising the following steps:
acquiring multiple paths of video streams and multiple paths of audio streams of the video conference from a Selective Forwarding Unit (SFU);
the multi-channel audio streams are fused to obtain a channel of mixed audio stream, and the multi-channel video streams are fused to obtain a channel of fused video stream;
creating a media stream transmission channel between the recorded broadcast service and the recorded broadcast service, and sending the audio mixing audio stream and the fused video stream to the recorded broadcast service through the media stream transmission channel, so that the recorded broadcast service writes the audio mixing audio stream and the fused video stream into a video recording file and writes index information of the audio mixing audio stream and the fused video stream into a video recording index file;
and receiving the recorded broadcast abnormal message sent by the recorded broadcast service through the media stream transmission channel, and sending the recorded broadcast abnormal message to the SFU, so that the SFU feeds the recorded broadcast abnormal message back to the video conference terminal and/or the video conference control platform.
2. The recorded broadcast exception feedback method of claim 1, wherein the recorded broadcast exception message comprises:
and the abnormal writing state of the video recording file when the audio mixing audio stream and the fused video stream are written into the video recording file and/or the abnormal writing state of the video recording index file when the index information of the audio mixing audio stream and the fused video stream is written into the video recording index file.
3. The recorded broadcast anomaly feedback method according to claim 1, further comprising, after transmitting the mixed audio stream and the merged video stream to the recorded broadcast service through the media stream transmission channel:
periodically acquiring the data volume of the video files from the recording and broadcasting service;
calculating the video recording duration according to the video recording starting time and the current time;
and sending the data volume of the video file, the video recording time length and the recording and broadcasting abnormal message to the SFU, so that the SFU feeds back the data volume of the video file, the video recording time length and the recording and broadcasting abnormal message to a video conference terminal and/or a video conference control platform.
4. The recorded broadcast anomaly feedback method of claim 1,
a media streaming channel is created with the recorded broadcast service through a proprietary protocol.
5. A recorded broadcast anomaly feedback system, comprising: the system comprises an analog terminal, a plug flow client and a plug flow server;
the analog terminal is used for acquiring multiple audio streams and multiple video streams of a video conference from the selective forwarding unit SFU, fusing the multiple audio streams to obtain one mixed audio stream, fusing the multiple video streams to obtain one fused video stream, and outputting the mixed audio stream and the fused video stream to the stream pushing client;
the push streaming client is used for receiving the audio mixing audio stream and the fused video stream, creating a media stream transmission channel between the push streaming client and the push streaming server, and sending the audio mixing audio stream and the fused video stream to the push streaming server through the media stream transmission channel;
the push streaming server is configured to receive the audio mixing audio stream and the fused video stream, write the audio mixing audio stream and the fused video stream into a video file, write index information of the audio mixing audio stream and the fused video stream into a video index file, acquire an abnormal recording and broadcasting message, and send the abnormal recording and broadcasting message to the push streaming client through the media stream transmission channel;
the stream pushing client is also used for receiving the recorded broadcast abnormal message and outputting the recorded broadcast abnormal message to the simulation terminal;
the simulation terminal is further configured to receive the recording and broadcasting exception message and send the recording and broadcasting exception message to the SFU, so that the SFU feeds back the recording and broadcasting exception message to the video conference terminal and/or the video conference control platform.
6. The recorded broadcast exception feedback system of claim 5, wherein the recorded broadcast exception message comprises:
and the abnormal writing state of the video recording file when the audio mixing audio stream and the fused video stream are written into the video recording file and/or the abnormal writing state of the video recording index file when the index information of the audio mixing audio stream and the fused video stream is written into the video recording index file.
7. The recorded broadcast anomaly feedback system in accordance with claim 5,
the plug-flow client is also used for periodically acquiring the data volume of the video file from the plug-flow server, calculating the video recording duration according to the video recording starting time and the current time, and outputting the data volume of the video file and the video recording duration to the analog terminal;
the simulation terminal is further configured to receive the data volume of the video file and the video recording duration, and send the data volume of the video file, the video recording duration and the recording and broadcasting exception message to the SFU, so that the SFU feeds back the data volume of the video file, the video recording duration and the recording and broadcasting exception message to a video conference terminal and/or a video conference control platform.
8. The recorded broadcast anomaly feedback system in accordance with claim 5,
and the stream pushing client establishes a media stream transmission channel with the stream pushing server through a private protocol.
9. An electronic device, comprising:
at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to cause the at least one processor to perform the recorded broadcast exception feedback method of any of claims 1-4.
10. A computer-readable storage medium storing computer instructions for causing a computer to perform the recorded broadcast anomaly feedback method of any one of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110304631.2A CN112702559B (en) | 2021-03-23 | 2021-03-23 | Recorded broadcast abnormity feedback method, system, equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110304631.2A CN112702559B (en) | 2021-03-23 | 2021-03-23 | Recorded broadcast abnormity feedback method, system, equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112702559A true CN112702559A (en) | 2021-04-23 |
| CN112702559B CN112702559B (en) | 2021-07-09 |
Family
ID=75516798
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110304631.2A Active CN112702559B (en) | 2021-03-23 | 2021-03-23 | Recorded broadcast abnormity feedback method, system, equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112702559B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113259700A (en) * | 2021-05-17 | 2021-08-13 | 青岛海信移动通信技术股份有限公司 | Management device, server, video storage method and system |
| CN114257771A (en) * | 2021-12-21 | 2022-03-29 | 杭州海康威视数字技术股份有限公司 | Video playback method and device for multi-channel audio and video, storage medium and electronic equipment |
| CN114827633A (en) * | 2022-06-17 | 2022-07-29 | 浙江华创视讯科技有限公司 | Media stream disaster tolerance method, device and related equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101303880A (en) * | 2008-06-30 | 2008-11-12 | 北京中星微电子有限公司 | Method and apparatus for recording and playing audio-video document |
| US7593781B1 (en) * | 2003-02-05 | 2009-09-22 | American Megatrends, Inc. | Audio feedback test |
| CN102055949A (en) * | 2009-11-02 | 2011-05-11 | 华为终端有限公司 | Recording method, device and system of multimedia conference and rewinding method and device |
| CN110213529A (en) * | 2019-04-23 | 2019-09-06 | 视联动力信息技术股份有限公司 | A kind of condition processing method and system of monitor video recording arrangement |
| CN111131760A (en) * | 2019-12-31 | 2020-05-08 | 视联动力信息技术股份有限公司 | Video recording method and device |
-
2021
- 2021-03-23 CN CN202110304631.2A patent/CN112702559B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7593781B1 (en) * | 2003-02-05 | 2009-09-22 | American Megatrends, Inc. | Audio feedback test |
| CN101303880A (en) * | 2008-06-30 | 2008-11-12 | 北京中星微电子有限公司 | Method and apparatus for recording and playing audio-video document |
| CN102055949A (en) * | 2009-11-02 | 2011-05-11 | 华为终端有限公司 | Recording method, device and system of multimedia conference and rewinding method and device |
| CN110213529A (en) * | 2019-04-23 | 2019-09-06 | 视联动力信息技术股份有限公司 | A kind of condition processing method and system of monitor video recording arrangement |
| CN111131760A (en) * | 2019-12-31 | 2020-05-08 | 视联动力信息技术股份有限公司 | Video recording method and device |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113259700A (en) * | 2021-05-17 | 2021-08-13 | 青岛海信移动通信技术股份有限公司 | Management device, server, video storage method and system |
| CN114257771A (en) * | 2021-12-21 | 2022-03-29 | 杭州海康威视数字技术股份有限公司 | Video playback method and device for multi-channel audio and video, storage medium and electronic equipment |
| CN114257771B (en) * | 2021-12-21 | 2023-12-01 | 杭州海康威视数字技术股份有限公司 | Video playback method and device for multipath audio and video, storage medium and electronic equipment |
| CN114827633A (en) * | 2022-06-17 | 2022-07-29 | 浙江华创视讯科技有限公司 | Media stream disaster tolerance method, device and related equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112702559B (en) | 2021-07-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112702559B (en) | Recorded broadcast abnormity feedback method, system, equipment and readable storage medium | |
| EP2658232A1 (en) | Method and system for an optimized multimedia communications system | |
| CN112738140A (en) | Video stream transmission method, device, storage medium and equipment based on WebRTC | |
| CN109089129B (en) | Stable multi-video binding live broadcasting system and method thereof | |
| CN103109528A (en) | System and method for the control and management of multipoint conferences | |
| US20070008969A1 (en) | Apparatuses and methods for delivering data stream content to consumer devices | |
| CN112738451B (en) | Video conference recording and playing method, device, equipment and readable storage medium | |
| CN110198493A (en) | Media data method for down loading, device, computer equipment, storage medium and system | |
| CN101350741A (en) | Method, apparatus and system for notifying real time stream protocol event | |
| CN104469248A (en) | Video session method and device, server and terminal | |
| CN105656742A (en) | Multi-looped network stream media multicast system and method based on MOST | |
| CN112422880A (en) | Video conference method, terminal device and readable storage medium | |
| US20210021659A1 (en) | Delivery apparatus, delivery method, and program | |
| CN108632681B (en) | Method, server and terminal for playing media stream | |
| CN112532719B (en) | Information stream pushing method, device, equipment and computer readable storage medium | |
| CN114710472B (en) | AR video call processing method and device and communication equipment | |
| JP4917497B2 (en) | Video distribution device, distribution video switching method, distribution video switching program, and distribution video switching program recording medium | |
| CN114979687B (en) | Wheat connecting control method and device based on edge calculation | |
| CN110392285B (en) | Media stream processing method and device | |
| CN112995573B (en) | Video conference live broadcasting system and method | |
| CN114374816A (en) | Information interaction method based on video conference, server and terminal equipment | |
| US8838805B2 (en) | Method and apparatus for establishing a streamed media session | |
| CN115776554A (en) | Conference processing method, conference processing device, electronic equipment and storage medium | |
| CN112804564A (en) | Media stream processing method, device and equipment for video conference and readable storage medium | |
| CN116471399A (en) | Video telephone service testing method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |