CN115065829A - Multi-person wheat connecting method and related equipment - Google Patents
Multi-person wheat connecting method and related equipment Download PDFInfo
- Publication number
- CN115065829A CN115065829A CN202210439610.6A CN202210439610A CN115065829A CN 115065829 A CN115065829 A CN 115065829A CN 202210439610 A CN202210439610 A CN 202210439610A CN 115065829 A CN115065829 A CN 115065829A
- Authority
- CN
- China
- Prior art keywords
- stream
- microphone
- audio
- wheat
- anchor
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/2187—Live feed
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
- H04N21/2368—Multiplexing of audio and video streams
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/439—Processing of audio elementary streams
- H04N21/4394—Processing of audio elementary streams involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
- H04N21/44008—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/14—Systems for two-way working
- H04N7/15—Conference systems
- H04N7/152—Multipoint control units therefor
Abstract
The invention discloses a multi-person wheat connecting method and related equipment. The method comprises the following steps: under the condition of receiving the connecting request message, pulling video streams and audio streams sent by N clients of the anchor broadcasts to be connected based on the same connecting request message; respectively decoding video streams and audio streams sent by different clients; mixing based on the decoded video stream to generate a continuous stream; mixing the streams according to the decoded audio streams to generate a microphone connecting audio stream; and sending the target connecting to the wheat video stream and the target connecting to the wheat audio stream corresponding to each main broadcasting to be connected to the wheat. Compared with the prior art, the scheme provided by the embodiment of the application establishes a mixed flow service method for each anchor respectively, and can greatly reduce the flow pulling operation, the mixed flow times and the encoding and decoding times, thereby greatly reducing the resource consumption in the wheat connecting process.
Description
Technical Field
The specification relates to the field of live broadcast wheat connection, in particular to a multi-person wheat connection method and related equipment.
Background
In the live broadcast process, the live broadcast by multiple anchor broadcasters is a popular live broadcast mode, in the live broadcast technology, an MCU (Multipoint conference Unit) is a very common mixed stream scheme for live broadcast, in the mixed stream of MCU, a server needs to mix streams of anchor broadcasters participating in live broadcast, and the pictures after mixing streams are provided for viewers and anchor broadcasters in live broadcast. The mixed flow process comprises video processing processes such as decoding, picture scaling and superposition, encoding and the like, and the video processing needs to consume a large amount of CPU resources.
In the existing MCU mixed stream, a mixed stream service is established for each of the main broadcasting and audience terminals that connect to the wheat. The mixed flow process comprises video processing processes such as decoding, picture scaling and superposition, encoding and the like, a large amount of CPU resources are consumed for video processing, and the popularization and application of the MCU technology are limited due to huge resource consumption.
Disclosure of Invention
A series of concepts in a simplified form are introduced in the summary section, which is described in further detail in the detailed description section. This summary of the invention is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
In order to provide an MCU mixed flow scheme with low resource consumption, in a first aspect, the present invention provides a multi-person wheat-connecting method for a client, where the method includes:
under the condition of receiving the connecting request message, pulling video streams and audio streams sent by N clients of the anchor broadcasts to be connected based on the same connecting request message;
respectively decoding video streams and audio streams sent by different clients;
mixing and encoding based on the decoded video stream to generate a continuous stream;
mixing and encoding according to the decoded audio stream to generate a continuous audio stream;
and sending the target connecting to the wheat video stream and the target connecting to the wheat audio stream corresponding to each main broadcasting to be connected to the wheat.
Optionally, the mixing and encoding based on the decoded video stream to generate a continuous-microphone video stream includes:
the N decoded video streams are mixed and encoded to generate a common continuous stream.
Optionally, the mixing and encoding according to the decoded audio stream to generate a microphone audio stream includes:
and mixing and coding the N-1 decoded audio streams to generate N types of special continuous microphone audio streams, wherein each to-be-connected microphone main broadcast corresponds to one type of special continuous microphone audio stream, and each type of special continuous microphone audio stream does not comprise the audio corresponding to the to-be-connected microphone main broadcast.
Optionally, the sending, to each anchor to be connected, a target connected-to-microphone video stream and a target connected-to-microphone audio stream corresponding to the anchor to be connected includes:
and sending the common connecting microphone video stream and the corresponding special connecting microphone audio stream to each main broadcasting of the connecting microphones.
Optionally, the method further includes:
mixing and encoding the N decoded audio streams to generate a spectator audio stream;
and sending the common video stream and the audience audio stream to an audience-associated client.
Optionally, the method further includes:
under the condition of receiving a first connecting request, distributing a mixed flow and coding server with the lowest load according to the ID of a live broadcast room of a main broadcast sending the first connecting request and creating mixed flow and coding service, wherein the main broadcast sending the first connecting request is the main broadcast creating the connecting service;
updating a mixed flow and coding service database according to the ID of the main broadcast of the first linting request, the mixed flow and coding server and the mixed flow and coding service, wherein the mixed flow and coding service database comprises the corresponding relation among the ID of the main broadcast, the mixed flow and coding server and the mixed flow and coding service;
and under the condition of receiving a second wheat connecting request, selecting corresponding mixed flow and coding service for the main broadcast sending the second wheat connecting request in the mixed flow and coding service database according to the ID of the direct broadcast room requesting the wheat connecting main broadcast.
Optionally, the mixing and encoding based on the decoded video stream to generate a continuous-microphone video stream includes:
and mixing and encoding based on the decoded video stream, and inserting SEI information to generate the Lianmai video stream, wherein the SEI information is used for the client to adjust the video layout of the multiple anchor broadcasts.
In a second aspect, the present invention further provides a multi-person wheat-connecting device, including:
the pull stream unit is used for pulling the video stream and the audio stream sent by the N clients of the main broadcasts to be connected based on the same connecting request message under the condition of receiving the connecting request message;
the decoding unit is used for respectively decoding video streams and audio streams sent by different clients;
a first mixed stream encoding unit for mixing and encoding based on the decoded video stream to generate a Lianmai video stream;
a second mixed stream encoding unit for mixing and encoding the decoded audio stream to generate a continuous stream;
and the stream pushing unit is used for sending the target connecting microphone video stream and the target connecting microphone audio stream corresponding to each to-be-connected microphone anchor.
In a third aspect, an electronic device includes: a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor is configured to implement the steps of the multi-person microphone connecting method according to any one of the first aspect when executing the computer program stored in the memory.
In a fourth aspect, the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the multi-person microphone connecting method according to any one of the above-mentioned first aspects.
In summary, the multi-person wheat connecting method provided by the application comprises the following steps: under the condition of receiving the connecting request message, pulling video streams and audio streams sent by N clients of the anchor broadcasts to be connected based on the same connecting request message; decoding each of said video stream and said audio stream; mixing based on the decoded video stream to generate a continuous stream; mixing the streams according to the decoded audio stream to generate a continuous stream audio stream; and sending the connected-to-microphone video stream and the connected-to-microphone audio stream corresponding to each to-be-connected-to-microphone main broadcast. According to the scheme provided by the embodiment of the application, in the process of connecting the multiple main broadcasts with the wheat, mixed flow operation of the multiple main broadcasts is performed in one mixed flow service, the mixed flow service only needs to perform one-time stream pulling operation on each main broadcast to be connected with the wheat, only needs to perform one-time decoding operation, and mixed flow is performed according to the decoded video stream and audio stream so as to meet the wheat connecting requirements of different main broadcasts to be connected with the wheat. Compared with the prior art, the method for respectively establishing the mixed flow service for each anchor can greatly reduce the times of flow pulling operation and decoding, thereby greatly reducing the resource consumption in the process of connecting the wheat.
The multi-person continuous wheat method of the present invention, and other advantages, objects, and features of the invention will be in part apparent from the following description and in part will become apparent to those skilled in the art upon examination of the following and practice of the invention.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the specification. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
FIG. 1 is a block diagram of a multi-user wheat linkage method in the prior art
Fig. 2 is a schematic flow chart of a multi-person wheat-connecting method according to an embodiment of the present disclosure;
fig. 3 is a schematic diagram illustrating another multi-person wheat-connecting method according to an embodiment of the present disclosure;
fig. 4 is a schematic diagram illustrating a scheduling optimization principle of a multi-user wheat-connecting method according to an embodiment of the present application;
fig. 5 is a schematic view of a distribution structure of a multi-person continuous microphone video according to an embodiment of the present disclosure;
fig. 6 is a schematic structural diagram of a multi-person wheat connecting device according to an embodiment of the present disclosure;
fig. 7 is a schematic structural diagram of a multi-user microphone connecting electronic device according to an embodiment of the present disclosure.
Detailed Description
According to the scheme provided by the embodiment of the application, in the process of connecting the multiple main broadcasts with the wheat, mixed flow operation of the multiple main broadcasts is performed in one mixed flow service, the mixed flow service only needs to perform one-time stream pulling operation on each main broadcast to be connected with the wheat, only needs to perform one-time decoding operation, and mixed flow is performed according to the decoded video stream and audio stream so as to meet the wheat connecting requirements of different main broadcasts to be connected with the wheat. Compared with the prior art, the method for establishing the mixed flow service for each anchor respectively can greatly reduce the times of flow pulling operation, mixed flow and coding, thereby greatly reducing the resource consumption in the process of connecting the wheat.
The terms "first," "second," "third," "fourth," and the like in the description and in the claims of the present application and in the drawings described above, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be practiced otherwise than as specifically illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments.
Fig. 1 shows a possible multi-user connected mode in the prior art, which establishes different mixed-flow services a, B, C and mixed-flow services corresponding to viewers respectively for different anchor a3, anchor B3 and anchor C3. Mixed-stream service a is main broadcasting A3 mixed B3 and C3 streams, mixed-stream service B3 is main broadcasting B3 mixed A3 and C3 streams, mixed-stream service C3 is main broadcasting C3 mixed A3 and B3 streams, and mixed-stream service viewers are viewer mixed A3 streams, B3 streams and C3 streams for viewer-side viewing. Three anchor needs 3 mixed flows, and spectator needs 1 mixed flow. When the number of the main broadcasts to be connected is N, N main broadcasts and audiences need N +1 mixed flows. Meanwhile, in the mixed flow process, three anchor casts a3, B3 and C3 need to perform three times of stream pulling, three times of decoding and four times of encoding, which brings a large amount of performance overhead, and the overhead of resources has a linear relationship with the number of anchor casts, so that the architecture brings a large burden to service resources.
Referring to fig. 2, a schematic flow chart of a multi-user microphone connecting method provided in an embodiment of the present application is used in a server, and specifically includes:
s110, under the condition of receiving the connecting request message, pulling video streams and audio streams sent by N clients of the main broadcasts to be connected based on the same connecting request message;
for example, in order to reduce consumption of service resources, in the method provided in this embodiment of the present application, a server, when receiving a continuous microphone request, places mixed stream operations of a plurality of main broadcasters requesting continuous microphone in the same continuous microphone request group in the same mixed stream service to complete, as shown in fig. 3, through the same mixed stream service 22, pulls a live stream of a main broadcaster a22, a main broadcaster B22, and a main broadcaster C33, and compared with a method for separately constructing mixed stream services for each main broadcaster, the streaming pulling only needs to be performed once, and resource saving can be achieved in a streaming pulling process, resources consumed by a modified streaming process are 1/N of resources consumed by an original scheme, where N is the number of main broadcasters waiting for continuous microphone.
S120, decoding video streams and audio streams sent by different clients respectively;
illustratively, a pulled live stream (including a video stream and an audio stream) is decoded, and a video stream and an audio stream of each anchor are respectively obtained, and since all mixed stream services of all anchors are integrated into one mixed stream service, only one decoding is needed, as shown in fig. 3, under the condition that there are 3 anchor broadcasters to be connected, only one decoding of a, B, and C is needed to obtain the decoded video stream and audio stream. Compared with a method for respectively constructing mixed flow service for each anchor, decoding only needs to be carried out once, resources can be saved in the decoding process, resources consumed in the modified decoding process are 1/N of the resources consumed in the original scheme, wherein N is the number of anchors to be connected.
S130, mixing and coding are carried out on the basis of the decoded video stream to generate a Lianmai video stream;
illustratively, the decoded video stream can be multiplexed according to the requirement of live broadcasting, and mixed stream and encoding are performed according to different video mixed stream schemes to generate a continuous-broadcasting video stream. It should be noted that the mixing scheme of the video streams may be determined according to different requirements of the anchor end or the user end, but all adopt the video streams after the decoding operation of step S120.
S140, mixing and coding are carried out according to the decoded audio stream to generate a microphone connecting audio stream;
illustratively, the decoded audio stream may be multiplexed according to the requirement of live broadcasting, and mixed stream and encoded according to different audio mixed stream schemes to generate a continuous microphone audio stream. It should be noted that the mixing scheme of the audio streams may be determined according to different requirements of the anchor end or the user end, but all the audio streams after the decoding operation of step S120 are adopted.
And S150, sending the target connecting to the wheat video stream and the target connecting to the wheat audio stream corresponding to each to-be-connected wheat anchor.
For example, after different connected-to-wheat video streams and different connected-to-wheat audio streams are generated according to different mixed flow schemes of a user side according to a main broadcasting end, the corresponding connected-to-wheat video streams and the corresponding connected-to-wheat audio streams are sent to the corresponding connected-to-wheat main broadcasting, so that a plurality of to-be-connected main broadcasting connected-to-wheat operations are realized.
To sum up, according to the scheme provided by the embodiment of the present application, in the process of connecting multiple anchor broadcasters, the mixing and encoding operations of the multiple anchor broadcasters are performed in a mixing and encoding service, the mixing and encoding service only needs to perform a stream pulling operation on each anchor broadcaster to be connected, and only needs to perform a decoding operation, and mixing and encoding are performed according to the decoded video stream and audio stream, so as to meet the requirements of connecting different anchor broadcasters to be connected. Compared with the prior art, the method for respectively establishing the mixed flow service for each anchor can greatly reduce the times of flow pulling operation, mixed flow and coding, thereby greatly reducing the resource consumption in the process of connecting the wheat.
In some examples, the mixing and encoding based on the decoded video stream in step S130 to generate the continuous microphone video stream includes:
the N decoded video streams are mixed and encoded to generate a common continuous stream.
Illustratively, in the mixed video stream service, about 80% of the performance overhead in mixed stream operations (picture cropping, scaling and overlaying) and in encoding, all video streams of N to-be-connected-main-broadcasts can be mixed and encoded to generate one common-connected-main video stream containing all to-be-connected-main-broadcast videos. As shown in fig. 3, only a common connected-to-many video stream H264ABC is generated for the video stream, which includes all the main broadcaster a22, main broadcaster B22 and main broadcaster C33 videos, and the video mixing and encoding operation can reduce the resource consumption to 1/N of the original consumption compared to the existing method of establishing one mixing and encoding service per main broadcaster, where N is the number of main broadcasters to be connected.
In summary, according to the method provided by the embodiment of the present application, since N to-be-connected main broadcasts share one mixed stream and coding service, when performing video mixed stream and coding operation, only one common connected-to-main video stream including all main broadcasts is required to be generated, and compared with the existing method for establishing a mixed stream and coding service for each main broadcast, resources consumed by the video mixed stream operation can be reduced to 1/N of the original method, and resource consumption in the wheat connecting process is greatly reduced.
In some examples, the mixing and encoding from the decoded audio stream to generate a joined audio stream includes:
and mixing and coding the N-1 decoded audio streams to generate N types of special continuous microphone audio streams, wherein each to-be-connected microphone main broadcast corresponds to one type of special continuous microphone audio stream, and each type of special continuous microphone audio stream does not comprise the audio corresponding to the to-be-connected microphone main broadcast.
For example, during the course of connecting to the wheat by the anchor, in the received audio stream, the anchor does not want to hear its own voice, or there is a delay between its sounding and receiving of the voice, which may interfere with the live broadcast of the anchor. Therefore, different audio mixed flow and coding schemes can be formulated for each anchor, namely a special connected-to-wheat audio stream is generated for each anchor, the special connected-to-wheat audio stream corresponding to each anchor does not comprise the audio of the anchor, the influence of the audio of the anchor on the live broadcast of the anchor can be effectively eliminated, the mixed flow and coding consumption resources of the audio are low, and the resource consumption in the process of connecting to wheat is not greatly influenced. As shown in fig. 3, the private live audio stream corresponding to anchor a22 includes a mix of anchor B22 audio and anchor C22 audio, the private live audio stream corresponding to anchor B22 includes a mix of anchor a22 audio and anchor C22 audio, and the private live audio stream corresponding to anchor C22 includes a mix of anchor a22 audio and anchor B22 audio. It is understood that, when the number of the intended connected wheat anchor is N, each dedicated connected wheat audio stream includes N-1 anchors of audio, and a total of N connected wheat audio streams can be generated.
In summary, the method provided by the embodiment of the present application formulates the dedicated wheat-connected audio stream for each anchor, and each dedicated wheat-connected audio stream does not include the audio of the corresponding anchor, so that the influence of the audio of the anchor on the live broadcast of the anchor is effectively avoided, and the live broadcast comfort level of the anchor is improved.
In some examples, the sending, to each of the to-be-connected anchor broadcasters, a connected-to-live video stream and a connected-to-live audio stream corresponding to the to-be-connected anchor broadcasters includes:
and sending the common connecting microphone video stream (namely the target connecting microphone video stream) and the corresponding special connecting microphone audio stream (namely the target connecting microphone audio stream) to each to-be-connected microphone main broadcasting.
Illustratively, public connecting microphone video streams and special connecting microphone audio streams corresponding to each anchor are established, the public connecting microphone and the special connecting microphone audio streams are combined to form connecting microphone streams corresponding to different anchors, and the corresponding connecting microphone streams are sent to corresponding anchor terminals. As shown in fig. 3, a anchor a stream, an anchor B stream, and an anchor C stream are combined for anchor a22, anchor B22, and anchor C22, respectively, where the anchor a stream includes a private barley mixing stream BC in OPUS format and a common video stream ABC in H264 format, the anchor B stream includes a private barley mixing stream AC in OPUS format and a common video stream ABC in H264 format, and the anchor C stream includes a private barley mixing stream AB in OPUS format and a common video stream ABC in H264 format, and the anchor a stream is transmitted to anchor a22, the anchor B stream is transmitted to anchor B22, and the anchor C stream is transmitted to anchor C22.
In some examples, the method further comprises:
mixing and encoding the N decoded audio streams to generate a spectator audio stream;
and sending the common video stream and the audience audio stream to an audience-associated client.
Illustratively, as the client of the audience, it is desirable to hear all the sound of the main broadcasting, therefore, the audio streams of the N main broadcasting to be connected are mixed and encoded to generate an audience audio stream, and a common video stream including all the main broadcasting pictures and an audience audio stream including all the main broadcasting audio are transmitted to the client of the audience, and the audience can receive all the main broadcasting pictures and sound. As shown in fig. 3, the decoded audios corresponding to the anchor a22, the anchor B22, and the anchor B33 are mixed and encoded to generate a viewer audio stream including an AAC-formatted audio mix ABC, and the viewer audio stream and the common video stream H264ABC are pushed to the CDN system and distributed by the CDN to the client associated with the viewer.
In summary, the method provided by the embodiment of the present application can directly pull the common video stream adopted by the anchor terminal, remix and encode the common video stream into an audience audio stream including all anchors based on the decoded audio, and provides a method for enabling the audience terminal with low resource consumption to watch live online broadcasting.
In some examples, the method further comprises:
under the condition of receiving a first connecting request, distributing a mixed flow and coding server with the lowest load according to the ID of a live broadcast room of a main broadcast sending the first connecting request and creating mixed flow and coding service, wherein the main broadcast sending the first connecting request is the main broadcast creating the connecting service;
updating a mixed flow and coding service database according to the ID of the main broadcast of the first linting request, the mixed flow and coding server and the mixed flow and coding service, wherein the mixed flow and coding service database comprises the corresponding relation among the ID of the main broadcast, the mixed flow and coding server and the mixed flow and coding service;
and under the condition of receiving a second wheat connecting request, selecting corresponding mixed flow and coding service for the main broadcast sending the second wheat connecting request in the mixed flow and coding service database according to the ID of the live broadcasting room requesting the wheat connecting main broadcast.
For example, the servers may be divided into a management server and a mixed flow and coding server, in a case where the management server receives a first microphone connecting request, that is, a request for creating a microphone connecting room (from in fig. 4), the management server (mixed flow scheduling in fig. 4) selects one mixed flow and coding server (mixed flow in fig. 4) with the lowest load from among a plurality of mixed flow and coding servers to create a mixed flow service for the mixed flow and coding server. And updating the corresponding relation between the ID of the live broadcast room of the first main broadcasting for connecting the wheat and the mixed flow and coding server and the mixed flow and coding service to a mixed flow and coding service database, and inquiring the mixed flow and coding server and the mixed flow and coding service corresponding to the ID of the live broadcast room in the mixed flow service database according to the ID of the live broadcast room of the main broadcasting for requesting connection in the second main broadcasting request under the condition of receiving the second main broadcasting request sent by other main broadcasting, thereby realizing the connection of the wheat by a plurality of main broadcasting. And in the case that all the main broadcasters finish connecting the wheat, eliminating the corresponding relation of the live room ID, the mixed stream and the coding server and the mixed stream and the coding service corresponding to the room of connecting the wheat in the mixed stream service database.
In summary, according to the method provided by the application, when the anchor requests to create a connected-to-wheat room, the management server creates mixed flow and coding services for the mixed flow and coding server with the lowest distribution load, and the mixed flow and coding service data database is established to ensure that the anchor to be connected corresponding to the same connected-to-wheat request message completes mixed flow and coding in the same mixed flow service, so that the multiplexing of pulling flow and decoding operation can be ensured, and the resource consumption in the connected-to-wheat process can be effectively reduced.
In some examples, the mixing based on the decoded video stream to generate the continuous-microphone video stream includes:
and mixing and encoding based on the decoded video stream, and inserting SEI information to generate the Lianmai video stream, wherein the SEI information is used for the client to adjust the video layout of the multiple anchor broadcasts.
Illustratively, the above architecture addresses the performance issue, but at the same time introduces new issues: the layout is not flexible enough. ABC three anchor as shown in fig. 5 may want to see different pictures, for example, B anchor may want to be in the upper left corner, C anchor may want to be in the upper left corner, and all anchors in the next figure see pictures that are a anchor in the upper left corner, which obviously cannot meet the requirements of all anchors, so we also need to further optimize the implementation of free layout.
Assuming that the resolution of fig. 5 is 1280x720, the a anchor occupies one fourth of the upper right corner, the B anchor occupies one fourth of the lower left corner, and the C anchor occupies one half of the right side, we insert layout information corresponding to three anchors ABC into the SEI information in h264, the type of SEI is unregistered user data, and SEI is inserted only in front of an I frame, the layout information of fig. 5 is packed as a json:
when the layout is changed, the layout information needs to be updated, because the SEI information is only inserted in front of the I frame, when the layout is updated, the I frame needs to be forcibly output by an encoder, and the method for forcibly outputting the I frame in the ffmpeg is to set the pict _ TYPE as AV _ PICTURE _ TYPE _ I.
The client obtains the layout information by analyzing the SEI information in the front of the H264I frame, cuts the layout information according to the layout information of the video to respectively obtain the independent pictures of the three anchor players ABC, and when the layout changes, the corresponding cutting coordinates need to be updated. As shown in the above figure, cutting the top left 640X360 to obtain a single picture of anchor a, cutting the bottom left 640X360 to obtain a single picture of anchor B, cutting the right 640X720 to obtain a single picture of anchor C, and then freely laying out according to the user's requirements. Such as BC, want to place themselves in the upper left corner, all of which can be achieved by cropping individual pictures.
In summary, the method provided by the embodiment of the present application, by performing mixing and encoding on the decoded video stream and inserting the SEI information, can realize that the picture layout can be adjusted as required by the main broadcast or the audience when watching the live video.
Referring to fig. 6, an embodiment of a multi-person microphone connecting device in the embodiment of the present application may include:
a stream pulling unit 21, configured to pull, in a case that a mic connecting request message is received, video streams and audio streams sent by N clients of the anchor mcast to be connected based on the same mic connecting request message;
a decoding unit 22, configured to decode video streams and audio streams sent by different clients respectively;
a first mixed stream encoding unit 23 for mixing and encoding based on the decoded video stream to generate a Lianmai video stream;
a second mixed stream encoding unit 24, configured to mix and encode the decoded audio stream to generate a continuous mixed stream;
and the stream pushing unit 25 is configured to send a target connected-to-microphone video stream and a target connected-to-microphone audio stream corresponding to each to-be-connected-to-microphone anchor.
As shown in fig. 7, an electronic device 300 is further provided in the present embodiment, which includes a memory 310, a processor 320, and a computer program 311 stored in the memory 320 and executable on the processor, and when the computer program 311 is executed by the processor 320, the steps of any method of multiple persons connecting to a microphone are implemented.
Since the electronic device described in this embodiment is a device used for implementing a multi-user microphone device in this embodiment, based on the method described in this embodiment, a person skilled in the art can understand the specific implementation manner of the electronic device of this embodiment and various variations thereof, so that how to implement the method in this embodiment by the electronic device is not described in detail herein, and as long as the person skilled in the art implements the device used for implementing the method in this embodiment, the device falls within the scope of protection intended by this application.
In a specific implementation, the computer program 311 may implement any of the embodiments corresponding to fig. 1 when executed by a processor.
It should be noted that, in the foregoing embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to relevant descriptions of other embodiments for parts that are not described in detail in a certain embodiment.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Embodiments of the present application further provide a computer program product, where the computer program product includes computer software instructions, and when the computer software instructions are run on a processing device, the processing device is caused to execute the flow of multiple persons connecting to a microphone as in the embodiment corresponding to fig. 1.
The computer program product includes one or more computer instructions. The procedures or functions according to the embodiments of the present application are all or partially generated when the computer program instructions are loaded and executed on a computer. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored on a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wire (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). A computer-readable storage medium may be any available medium that a computer can store or a data storage device, such as a server, a data center, etc., that is integrated with one or more available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, a division of a unit is merely a logical division, and an actual implementation may have another division, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
Units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method of the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, an optical disk, or other various media capable of storing program codes.
The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.
Claims (10)
1. A multi-person microphone connecting method for a server, comprising:
under the condition of receiving the connecting request message, pulling video streams and audio streams sent by N clients of the anchor broadcasts to be connected based on the same connecting request message;
respectively decoding video streams and audio streams sent by different clients;
mixing and encoding based on the decoded video stream to generate a Lianmai video stream;
mixing and encoding according to the decoded audio stream to generate a continuous microphone audio stream;
and sending the target connecting to the wheat video stream and the target connecting to the wheat audio stream corresponding to each main broadcasting to be connected to the wheat.
2. The method of claim 1, wherein the mixing and encoding based on the decoded video stream to generate a Lianmai video stream comprises:
the N decoded video streams are mixed and encoded to generate a common continuous stream.
3. The method of claim 2, wherein the mixing and encoding from the decoded audio stream to generate a Lianmai audio stream, comprises:
and mixing and coding the N-1 decoded audio streams to generate N types of special continuous microphone audio streams, wherein each to-be-connected microphone main broadcast corresponds to one type of special continuous microphone audio stream, and each type of special continuous microphone audio stream does not comprise audio corresponding to the to-be-connected microphone main broadcast.
4. The method of claim 3, wherein said sending each of the to-be-connected-microphone anchor a target connected-microphone video stream and a target connected-microphone audio stream corresponding thereto comprises:
and sending the common connecting microphone video stream and the corresponding special connecting microphone audio stream to each main broadcasting of the connecting microphones.
5. The method of claim 2, further comprising:
mixing and encoding the N decoded audio streams to generate a spectator audio stream;
and sending the common video stream and the audience audio stream to an audience-associated client.
6. The method of claim 1, further comprising:
under the condition of receiving a first connecting request, distributing a mixed flow and coding server with the lowest load according to the ID of a live broadcast room of a main broadcast sending the first connecting request and creating mixed flow and coding service, wherein the main broadcast sending the first connecting request is the main broadcast creating the connecting service;
updating a mixed flow and coding service database according to the ID of the main broadcast of the first linting request, the mixed flow and coding server and the mixed flow and coding service, wherein the mixed flow and coding service database comprises the corresponding relation among the ID of the main broadcast, the mixed flow and coding server and the mixed flow and coding service;
and under the condition of receiving a second wheat connecting request, selecting corresponding mixed flow service for the main broadcast sending the second wheat connecting request in the mixed flow service database according to the ID of the live broadcast room requesting the wheat connecting main broadcast.
7. The method of claim 1, wherein the mixing and encoding based on the decoded video stream to generate a Lianmai video stream comprises:
mixing and encoding based on the decoded video stream and inserting SEI information to generate a live video stream, wherein the SEI information is used for a client to adjust video layout of a plurality of anchor broadcasts.
8. A multi-person microphone connecting device, comprising:
the pull stream unit is used for pulling the video stream and the audio stream sent by the N clients of the anchor broadcasts to be connected based on the same connecting request message under the condition of receiving the connecting request message;
the decoding unit is used for respectively decoding video streams and audio streams sent by different clients;
a first mixed stream encoding unit for mixing and encoding based on the decoded video stream to generate a Lianmai video stream;
a second mixed stream encoding unit for mixing and encoding the decoded audio stream to generate a continuous stream;
and the stream pushing unit is used for sending the target connecting microphone video stream and the target connecting microphone audio stream corresponding to each to-be-connected microphone anchor.
9. An electronic device, comprising: memory, processor and computer program stored in the memory and executable on the processor, characterized in that the processor is adapted to carry out the steps of the multi-person microphone method according to any one of claims 1 to 7 when executing the computer program stored in the memory.
10. A computer-readable storage medium having stored thereon a computer program, characterized in that: the computer program when executed by a processor implementing a multi-person microphone connecting method as claimed in any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210439610.6A CN115065829A (en) | 2022-04-25 | 2022-04-25 | Multi-person wheat connecting method and related equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210439610.6A CN115065829A (en) | 2022-04-25 | 2022-04-25 | Multi-person wheat connecting method and related equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115065829A true CN115065829A (en) | 2022-09-16 |
Family
ID=83196805
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210439610.6A Pending CN115065829A (en) | 2022-04-25 | 2022-04-25 | Multi-person wheat connecting method and related equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115065829A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117499688A (en) * | 2023-12-29 | 2024-02-02 | 淘宝（中国）软件有限公司 | Method, equipment and storage medium for processing audio and video confluence in live broadcast continuous wheat |
| WO2024087197A1 (en) * | 2022-10-28 | 2024-05-02 | 广州酷狗计算机科技有限公司 | Live stream switching method and apparatus, server, terminal, and program product |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110505489A (en) * | 2019-08-08 | 2019-11-26 | 咪咕视讯科技有限公司 | Method for processing video frequency, communication equipment and computer readable storage medium |
| CN111147362A (en) * | 2020-01-16 | 2020-05-12 | 腾讯科技（深圳）有限公司 | Multi-user instant messaging method, system, device and electronic equipment |
| CN113542895A (en) * | 2021-07-07 | 2021-10-22 | 北京字节跳动网络技术有限公司 | Live broadcast method and device, computer equipment and storage medium |
| CN114173162A (en) * | 2021-12-15 | 2022-03-11 | 湖南快乐阳光互动娱乐传媒有限公司 | Interactive live broadcast system, maintenance method of publish-subscribe relationship and related equipment |
-
2022
- 2022-04-25 CN CN202210439610.6A patent/CN115065829A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110505489A (en) * | 2019-08-08 | 2019-11-26 | 咪咕视讯科技有限公司 | Method for processing video frequency, communication equipment and computer readable storage medium |
| CN111147362A (en) * | 2020-01-16 | 2020-05-12 | 腾讯科技（深圳）有限公司 | Multi-user instant messaging method, system, device and electronic equipment |
| CN113542895A (en) * | 2021-07-07 | 2021-10-22 | 北京字节跳动网络技术有限公司 | Live broadcast method and device, computer equipment and storage medium |
| CN114173162A (en) * | 2021-12-15 | 2022-03-11 | 湖南快乐阳光互动娱乐传媒有限公司 | Interactive live broadcast system, maintenance method of publish-subscribe relationship and related equipment |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024087197A1 (en) * | 2022-10-28 | 2024-05-02 | 广州酷狗计算机科技有限公司 | Live stream switching method and apparatus, server, terminal, and program product |
| CN117499688A (en) * | 2023-12-29 | 2024-02-02 | 淘宝（中国）软件有限公司 | Method, equipment and storage medium for processing audio and video confluence in live broadcast continuous wheat |
| CN117499688B (en) * | 2023-12-29 | 2024-05-03 | 淘宝（中国）软件有限公司 | Method, equipment and storage medium for processing audio and video confluence in live broadcast continuous wheat |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3562163A1 (en) | Audio-video synthesis method and system | |
| CN115065829A (en) | Multi-person wheat connecting method and related equipment | |
| CN110910860B (en) | Online KTV implementation method and device, electronic equipment and storage medium | |
| JP2011182109A (en) | Content playback device | |
| CN112019927B (en) | Video live broadcast method, microphone connecting equipment, live broadcast system and storage medium | |
| CN111147362B (en) | Multi-user instant messaging method, system, device and electronic equipment | |
| CA3000847C (en) | Gateway multi-view video stream processing for second-screen content overlay | |
| CN113542660A (en) | Method, system and storage medium for realizing conference multi-picture high-definition display | |
| CN106792244A (en) | Net cast method and device | |
| CN106209824A (en) | The cloud edit methods of data, system and the client of cloud editor | |
| KR100384757B1 (en) | Distributed internet broadcasting method and system using camera and screen capture | |
| CN114205637A (en) | Whiteboard audio and video synchronization method, device, equipment and storage medium | |
| JPWO2019188393A1 (en) | Information processing device, information processing method, transmission device, and transmission method | |
| CN114143600A (en) | Live broadcast picture adjusting method, device, equipment and storage medium | |
| CN112383794A (en) | Live broadcast method, live broadcast system, server and computer storage medium | |
| WO2019218741A1 (en) | Information transmission method and device, storage medium, and electronic device | |
| US10637904B2 (en) | Multimedia streaming service presentation method, related apparatus, and related system | |
| JP2009284268A (en) | Multicast broadcasting system and receiver | |
| EP3197151B1 (en) | Mosaic service presentation/delivery method and apparatus | |
| US20210227005A1 (en) | Multi-user instant messaging method, system, apparatus, and electronic device | |
| JP2022095777A (en) | Broadcast service Communication network distribution device and method | |
| CN101252670A (en) | Apparatus and method for processing conference television | |
| JP2002314960A (en) | Multi-view broadcasting service system and device or method or recording medium or its program related therewith | |
| US11778011B2 (en) | Live streaming architecture with server-side stream mixing | |
| KR101242478B1 (en) | Real time personal broadcasting system using media jockey based on multi-angle |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |