CN113301359A - Audio and video processing method and device and electronic equipment - Google Patents
Audio and video processing method and device and electronic equipment Download PDFInfo
- Publication number
- CN113301359A CN113301359A CN202010753100.7A CN202010753100A CN113301359A CN 113301359 A CN113301359 A CN 113301359A CN 202010753100 A CN202010753100 A CN 202010753100A CN 113301359 A CN113301359 A CN 113301359A
- Authority
- CN
- China
- Prior art keywords
- audio
- video
- live
- user
- game
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 87
- 238000009877 rendering Methods 0.000 claims abstract description 37
- 238000012545 processing Methods 0.000 claims description 85
- 238000000034 method Methods 0.000 claims description 38
- 230000008569 process Effects 0.000 claims description 17
- 238000004590 computer program Methods 0.000 claims description 11
- 238000009826 distribution Methods 0.000 claims description 5
- 238000005070 sampling Methods 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 12
- 238000003860 storage Methods 0.000 description 11
- 230000001360 synchronised effect Effects 0.000 description 5
- 230000000694 effects Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000009467 reduction Effects 0.000 description 3
- 230000006978 adaptation Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000005215 recombination Methods 0.000 description 1
- 230000006798 recombination Effects 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001755 vocal effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/2187—Live feed
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
- H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/439—Processing of audio elementary streams
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Databases & Information Systems (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
An audio and video processing method and device for live broadcasting and electronic equipment are disclosed. The audio and video processing method for live broadcasting comprises the following steps: acquiring user audio related to live broadcast operation; acquiring operation audio and operation video related to the live operation; mixing the user audio and the operation audio to obtain mixed audio; and mixing the mixed audio and the operation video and performing picture rendering of the operation video to obtain multimedia data for live broadcasting. Therefore, privacy disclosure of the anchor can be avoided, convenience in use of the anchor is improved, and stability and picture quality of live broadcasting are improved.
Description
Technical Field
The present application relates to the field of multimedia data processing, and more particularly, to an audio and video processing method and apparatus, and an electronic device.
Background
In recent years, as the speed of new infrastructure such as 5G is increased, two major problems of bandwidth and low delay which restrict the development of various cloud-based services are solved, and various cloud-based services are also in a stage of high-speed development in the next few years.
Live broadcast is a typical cloud-based service, and as described above, the development of the live broadcast industry is entering the maturity stage, and various live broadcast platforms develop cloud services in a dispute so as to further improve the competitiveness of platform content services. The live broadcast platform is used as a production and consumption platform of the content and is mainly used for connecting the anchor and the audience.
In the live broadcasting industry, a host broadcasts contents by live broadcasting various operations and states of the owner, such as playing games, singing dancing, or production labor, and making commentary. In the live broadcast process, the video of the live broadcast is recorded by using the terminal equipment of the live broadcast, such as a computer, a mobile phone, etc., and transmitted to the live broadcast platform for being watched by the audience.
However, there may be some problems in the live broadcasting process by the terminal device that records the anchor. For example, if the anchor is not aware, some screenshots other than the content to be live may be recorded, which may violate the privacy of the user; in addition, the screen recording itself requires a certain performance overhead, which presents adaptation problems for relatively low-end terminal devices, such as mobile phones and computers.
Taking the currently common live broadcast of a mobile phone game as an example, the method mainly includes screen recording and live broadcast, recording a game screen and microphone sound on a mobile phone, simultaneously performing audio and video coding, and then uploading the audio and video coding to a Content Delivery Network (CDN) server. Therefore, when the user switches the mobile phone application or the picture, some screen pictures outside the game may be recorded, and the privacy of the user may be violated; the traditional live broadcast video is mainly H.264 coding, has high requirements on the bandwidth and stability of an uplink network, and influences the live broadcast experience of a common anchor; and the screen recording has certain performance overhead, and the problem of adaptation exists for low-end mobile phones.
It is therefore desirable to provide an improved audiovisual processing scheme for live broadcasts.
Disclosure of Invention
The present application is proposed to solve the above-mentioned technical problems. Embodiments of the present application provide an audio and video processing method and apparatus for live broadcast, and an electronic device, which mix an acquired user audio related to a live broadcast operation with an operation audio and an operation video related to the live broadcast operation to obtain multimedia data for live broadcast, thereby avoiding privacy disclosure of a main broadcast, improving convenience of use of the main broadcast, and improving stability and picture quality of the live broadcast.
According to an aspect of the present application, there is provided an audio and video processing method for live broadcasting, including: acquiring user audio related to live broadcast operation; acquiring operation audio and operation video related to the live operation; mixing the user audio and the operation audio to obtain mixed audio; and mixing the mixed audio and the operation video and performing picture rendering of the operation video to obtain multimedia data for live broadcasting.
In the above audio and video processing method for live broadcasting, acquiring a user audio related to a live broadcasting operation includes: acquiring encoded user audio related to the live operation from a terminal device, the encoded user audio being obtained by encoding a sound of the user acquired using a microphone.
In the above method for processing live audio/video, the method further includes: and acquiring parameters of audio processing and/or video processing from the terminal equipment.
In the above audio and video processing method for live broadcast, the parameter of audio processing includes at least one of a sampling rate, a channel, and a bit rate of audio; the parameters of the video processing comprise at least one of resolution, frame rate, code rate and key frame time interval of the video.
In the above method for processing live audio/video, the method further includes: and collecting user videos related to the live broadcast operation from the terminal equipment.
In the above audio and video processing method for live broadcasting, acquiring an operation audio and an operation video related to the live broadcasting operation includes: acquiring a barrage video related to the live broadcast operation as at least one part of the operation video.
In the above audio and video processing method for live broadcasting, acquiring an operation audio and an operation video related to the live broadcasting operation includes: and acquiring the existing operation audio and operation video which are prepared in advance as the operation audio and operation video related to the live operation.
In the above audio and video processing method for live broadcasting, mixing the user audio and the operation audio to obtain a mixed audio includes: mixing the user audio and the operation audio; and encoding the mixed audio according to the parameters of the audio processing to obtain mixed audio.
In the above audio and video processing method for live broadcasting, mixing the user audio and the operation audio to obtain a mixed audio includes: receiving a first volume of the user audio and a second volume of the operation audio from the terminal device; and mixing the user audio and the operation audio according to the first volume and the second audio to obtain mixed audio.
In the above audio and video processing method for live broadcasting, mixing the mixed audio with the operation video and performing picture rendering of the operation video to obtain multimedia data for live broadcasting includes: and carrying out video stream coding on the operation video according to the parameters of the video processing.
In the above audio and video processing method for live broadcasting, mixing the mixed audio with the operation video and performing picture rendering of the operation video to obtain multimedia data for live broadcasting includes: synchronizing the mixed audio and the encoded video stream to obtain the multimedia data for live broadcast.
In the above audio and video processing method for live broadcasting, mixing the mixed audio with the operation video and performing picture rendering of the operation video to obtain multimedia data for live broadcasting includes: synchronizing the operation video and the user video to obtain a mixed video; and performing picture rendering on the mixed video and synchronizing with the mixed audio to obtain multimedia data for live broadcasting.
In the above audio and video processing method for live broadcasting, synchronizing the operation video with the user video to obtain a mixed video includes: synchronizing the operation video and the user video according to experience; or performing video identification on the operation video and the user video and synchronizing the operation video and the user video based on the video identification result.
In the above audio and video processing method for live broadcasting, rendering the mixed video and synchronizing with the mixed audio to obtain multimedia data for live broadcasting includes: synchronizing the mixed video subjected to picture rendering with the mixed audio according to experience; or performing video recognition on the mixed video subjected to the picture rendering, performing audio recognition on the mixed audio, and synchronizing the mixed video subjected to the picture rendering with the mixed audio based on the results of the video recognition and the audio recognition.
In the above method for processing live audio/video, the method further includes: and sending the multimedia data for live broadcasting to a content distribution network server.
In the above audio and video processing method for live broadcasting, before acquiring user audio related to a live broadcasting operation, the method further includes: and sending a message for prompting the user to send the user audio to the terminal equipment of the user carrying out the live broadcast operation.
In the audio and video processing method for live broadcasting, the live broadcasting comprises at least one of game live broadcasting, education live broadcasting, tour guide live broadcasting and remote medical live broadcasting.
In the audio and video processing method for live broadcasting, live broadcasting comprises multi-person live broadcasting.
In the above audio and video processing method for live broadcasting, the user audio is an audio of a first person in the live broadcasting of multiple persons, and the operation audio and the operation video are an audio and a video of a second person in the live broadcasting of multiple persons.
In the above method for processing live video and audio, the first person is an invited party in the live multiplayer broadcast, and the second person is a host party in the live multiplayer broadcast.
According to another aspect of the application, an audio and video processing method for live game is provided, which includes: in the process that a game anchor plays a game on a cloud server for live game play through terminal equipment, anchor audio related to the game is acquired from the terminal equipment; obtaining game audio and game video related to the game from the cloud server; mixing the anchor audio with the game audio to obtain mixed audio; and processing the mixed audio and the game video to obtain multimedia data for game live broadcast.
According to still another aspect of the present application, there is provided an audio and video processing apparatus for live broadcasting, including: a user acquisition unit for acquiring a user audio related to a live operation; the server acquisition unit is used for acquiring operation audio and operation video related to the live broadcast operation; an audio mixing unit configured to mix the user audio and the operation audio to obtain a mixed audio; and the audio and video mixing unit is used for mixing the mixed audio with the operation video and performing picture rendering on the operation video to obtain multimedia data for live broadcasting.
According to another aspect of the present application, there is provided an audio and video processing apparatus for live game, including: the system comprises an anchor audio acquisition unit, a terminal device and a server, wherein the anchor audio acquisition unit is used for acquiring anchor audio related to a game from the terminal device in the process that the game anchor plays the game on the cloud server for live game through the terminal device; a game audio and video acquisition unit for acquiring a game audio and a game video related to the game from the cloud server; an audio mixing unit for mixing the anchor audio with the game audio to obtain a mixed audio; and the audio and video mixing unit is used for processing the mixed audio and the game video to obtain multimedia data for live game broadcast.
According to still another aspect of the present application, there is provided an electronic apparatus including: a processor; and a memory in which are stored computer program instructions which, when executed by the processor, cause the processor to perform an audiovisual processing method for live broadcast as described above or an audiovisual processing method for live game as described above.
According to yet another aspect of the present application, there is provided a computer readable medium having stored thereon computer program instructions which, when executed by a processor, cause the processor to perform an audiovisual processing method for live broadcast as described above or an audiovisual processing method for live game as described above.
The audio and video processing method and device for live broadcasting and the electronic equipment can mix the audio and video of the user related to the obtained live broadcasting operation with the operation audio and the operation video related to the live broadcasting operation to obtain multimedia data for live broadcasting, so that privacy disclosure of a main broadcast is avoided, convenience of use of the main broadcast is improved, and stability and picture quality of the live broadcast are improved.
Drawings
The above and other objects, features and advantages of the present application will become more apparent by describing in more detail embodiments of the present application with reference to the attached drawings. The accompanying drawings are included to provide a further understanding of the embodiments of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the principles of the application. In the drawings, like reference numbers generally represent like parts or steps.
Fig. 1 illustrates a schematic diagram of an application scenario of an audio and video processing method for live broadcasting according to an embodiment of the present application.
Fig. 2 illustrates a flowchart of an audio and video processing method for live broadcasting according to an embodiment of the present application.
Fig. 3 illustrates a flowchart of an audio and video processing method for live game according to an embodiment of the present application.
Fig. 4 illustrates a schematic diagram of an overall scheme of a live game according to an embodiment of the present application.
Fig. 5 illustrates a block diagram of an audio-video processing device for live broadcasting according to an embodiment of the present application.
Fig. 6 illustrates a block diagram of an audio and video processing device for live game play according to an embodiment of the present application.
FIG. 7 illustrates a block diagram of an electronic device in accordance with an embodiment of the present application.
Detailed Description
Hereinafter, example embodiments according to the present application will be described in detail with reference to the accompanying drawings. It should be understood that the described embodiments are only some embodiments of the present application and not all embodiments of the present application, and that the present application is not limited by the example embodiments described herein.
Overview of a scene
Fig. 1 illustrates a schematic diagram of an application scenario of an audio and video processing method for live broadcasting according to an embodiment of the present application.
As shown in fig. 1, as the live user U performs live broadcast through a terminal device thereof, such as a smartphone T1, the live broadcast content may include game playing, singing, and the like, and in this embodiment of the present application, the live broadcast content includes an operation performed by the user on a live broadcast platform, for example, a server of C shown in fig. 1, for live broadcast. For example, the game host plays the game on the cloud server of the live platform C and performs live broadcasting. Or the performance anchor plays the songs on a cloud server of the live broadcast platform C for live broadcast of the songs and carries out live broadcast. That is, in the embodiment of the present application, the content live broadcast by the user on the live broadcast platform includes not only various operations performed by the user on the terminal device thereof, but also operations performed by the user in the server of the live broadcast platform.
In the process of operation for live broadcasting performed by a user on a server of the live broadcasting platform C, the audio and video processing method for live broadcasting according to the embodiment of the application can be applied to the live broadcasting platform C. Specifically, referring to fig. 1, in the live broadcast process, user audio related to the operation of the user is acquired from the terminal device T1 of the user U, for example, the user audio may be commentary audio of the user while playing a game or vocal audio of the user while singing a song. Also, in this process, the operation audio and the operation video related to the operation are acquired from the server of the live platform C itself, for example, the operation audio may be a game audio of the user when playing a game or a song accompaniment audio of the user when singing a song, and the operation video may be a game video of the user when playing a game or an MTV video of the user when singing a song.
Then, for the user audio obtained from the terminal device T and the operation audio obtained from the cloud server of the live broadcast platform C, audio and video mixing is performed at the cloud server, for example, first mixing is performed to obtain mixed audio, and then further mixing is performed with the operation video, and then, picture rendering of the operation video is performed to obtain multimedia data for live broadcast.
The multimedia data is used for live broadcast on a live broadcast platform, that is, the multimedia data can be further sent to a content delivery network server CDN through live broadcast streaming and further sent to terminal devices of other audiences, such as a smart phone T2 and a computer T3.
Therefore, live broadcast pictures are from videos of the cloud server instead of being from mobile phone screen recording, the live broadcast pictures of the anchor can be effectively guaranteed to be operation pictures, and privacy leakage risks caused by other operations of the anchor on the terminal equipment can be avoided.
Moreover, generally speaking, as the network of the cloud server is more stable than the home network of the anchor, and the bandwidth is also larger, the anchor does not need to care about the uplink bandwidth by acquiring the operation video from the cloud server, the anchor live broadcast threshold is greatly reduced, and the live broadcast stability and the watching effect are also improved.
In addition, the anchor is not required to be subjected to live broadcast while operating, so that screen recording, video acquisition coding and the like are not required, the system overhead of terminal equipment can be reduced, the fluency is improved, and the requirement on hardware configuration during live broadcast of the anchor is lowered.
Hereinafter, an exemplary audio and video processing method for live broadcasting according to an embodiment of the present application will be further described with reference to the accompanying drawings.
Exemplary method one
Fig. 2 illustrates a flowchart of an audio and video processing method for live broadcasting according to an embodiment of the present application.
As shown in fig. 2, an audio and video processing method for live broadcasting according to an embodiment of the present application includes the following steps.
S110, acquiring user audio related to live broadcast operation. As described above, the user audio refers to audio related to a live action of the user when the user plays the live broadcast, such as a commentary of a game anchor at the time of the game, a human voice of the game anchor at the time of the performance, and the like. Of course, those skilled in the art will appreciate that the user audio is not limited to the sound emitted by the anchor person, but may also include other audio that the anchor wants to play, such as a connection conversation with another person, or background music that wants to play.
In the embodiment of the application, the user audio is collected by the microphone of the terminal device in the live broadcasting process, and after the sound of the user is collected by the microphone on the terminal device, the user audio can be further encoded so as to be uploaded to the cloud server of the live broadcasting platform from the terminal device of the user. For example, the sound collected by the microphone is mainly in a PCM (Pulse Code Modulation) format, and the Audio data in the PCM format may be encoded by a codec provided in the terminal device, so as to obtain Audio data in an AAC (Advanced Audio Coding) format.
In addition, a user can use the audio and video processing method for live broadcasting according to the embodiment of the application by starting a specific application on the terminal device, for example, after the user opens the specific application, the application establishes connection with a cloud server of a live broadcasting platform to perform audio communication.
That is, in the audio and video processing method for live broadcasting according to the embodiment of the present application, acquiring the user audio related to the live broadcasting operation includes: acquiring encoded user audio related to the live operation from a terminal device, the encoded user audio being obtained by encoding a sound of the user acquired using a microphone.
In addition, those skilled in the art will appreciate that other processing may also be performed at the terminal device before the user audio is transmitted from the terminal device to the cloud server, such as noise reduction and echo cancellation processing on the PCM-formatted audio data before encoding, to improve sound quality.
And S120, acquiring operation audio and operation video related to the live operation. That is, since the live content includes an operation performed by the user on the cloud server for live broadcasting, part of the operation audio and the operation video may be directly obtained from the cloud server of the live broadcasting platform, instead of being obtained by recording a screen on the terminal device of the user.
It should be noted that, in the embodiment of the present application, in addition to acquiring the user audio related to the live broadcast operation from the terminal device, a user video related to the live broadcast operation may also be acquired from the terminal device, for example, a video of a user who is live broadcast is acquired through a camera of the terminal device, so that the user image during live broadcast is more vivid.
Therefore, in the audio and video processing method for live broadcasting according to the embodiment of the application, the method further includes: and collecting user videos related to the live broadcast operation from the terminal equipment.
In addition, in the embodiment of the present application, the operation video acquired from the cloud server may include other video content, such as barrage content of other viewers watching the live broadcast, in addition to the operation performed by the user on the cloud server for the live broadcast.
Therefore, in the audio and video processing method for live broadcasting according to the embodiment of the application, acquiring the operation audio and the operation video related to the live broadcasting operation includes: acquiring a barrage video related to the live broadcast operation as at least one part of the operation video.
In addition, the operation audio and the operation video related to the live operation may include existing operation audio and operation video prepared in advance, in addition to the real-time audio and video of the user at the time of live broadcasting. For example, when a user is singing a song live, the operation audio and the operation video may be background accompaniment audio of the song being performed and MTV picture video, which may be stored in the cloud server in advance. Accordingly, when the user performs live broadcasting, the existing operation audio and operation video prepared in advance can be directly inserted into the live broadcasting screen as the operation audio and operation video related to the live broadcasting operation.
That is, in the audio and video processing method for live broadcasting according to the embodiment of the present application, acquiring the operation audio and the operation video related to the live broadcasting operation includes: and acquiring the existing operation audio and operation video which are prepared in advance as the operation audio and operation video related to the live operation.
S130, mixing the user audio and the operation audio to obtain mixed audio. That is, since the user audio and the operation audio are both audio related to the live operation, in order to ensure synchronization of the user audio and the operation audio, the user audio and the operation audio are mixed.
In the embodiment of the present application, parameters of the audio and video that the user wants to specify may be further received from the terminal device of the user, for example, the video parameters may include resolution, frame rate, code rate, key frame time interval, and the like of the video, and the audio parameters may include a sampling rate, a channel, a bit rate, and the like of the audio.
In addition, as can be understood by those skilled in the art, the audio parameters and the video parameters may be received before the audio and video acquisition process, that is, the user specifies the desired audio and video parameters in advance, or may be received during the audio and video acquisition process, that is, the user changes the desired audio and video parameters during the live broadcast process.
Therefore, in the audio and video processing method for live broadcasting according to the embodiment of the application, the method further includes: and acquiring parameters of audio processing and/or video processing from the terminal equipment.
In the audio and video processing method for live broadcasting according to the embodiment of the application, the parameter of the audio processing includes at least one of a sampling rate, a channel and a bit rate of the audio; the parameters of the video processing comprise at least one of resolution, frame rate, code rate and key frame time interval of the video.
From the received parameters of the audio processing, the mixed audio may be obtained in accordance with the parameters of the audio processing. For example, the AAC-formatted user audio acquired from the terminal device may be first decoded into PCM format, mixed with the operation audio, and finally audio-encoded by the audio parameters set by the user through FFmpeg. Here, FFmpeg is an open source computer program that can be used to record, convert digital audio, video, and convert them into streams, which provides a complete solution to recording, converting, and streaming audio and video.
That is, in the audio and video processing method for live broadcasting according to the embodiment of the present application, mixing the user audio and the operation audio to obtain a mixed audio includes: mixing the user audio and the operation audio; and encoding the mixed audio according to the parameters of the audio processing to obtain mixed audio.
In addition, in the embodiment of the present application, the volume set for the user audio and the operation audio by the user of the terminal device of the user may also be set, for example, if the user wants the sound of the user to be a little louder, the volume of the user audio may be set to be larger. For example, when a game is played directly, the sound source of the game is from an audio stream of a cloud server for live game playing, and is not collected by a microphone of a mobile phone, so that the magnitudes of the anchor sound and the game sound in the live playing process can be controlled separately.
That is, in the audio and video processing method for live broadcasting according to the embodiment of the present application, mixing the user audio and the operation audio to obtain a mixed audio includes: receiving a first volume of the user audio and a second volume of the operation audio from the terminal device; and mixing the user audio and the operation audio according to the first volume and the second audio to obtain mixed audio.
And S140, mixing the mixed audio and the operation video, and performing picture rendering on the operation video to obtain multimedia data for live broadcasting. As described above, the user may specify desired video parameters, and thus, video stream encoding may be performed by FFmpeg on the cloud server according to the parameters of the video processing.
That is, in the audio and video processing method for live broadcasting according to the embodiment of the present application, mixing the mixed audio with the operation video and performing screen rendering of the operation video to obtain multimedia data for live broadcasting includes: and carrying out video stream coding on the operation video according to the parameters of the video processing.
In addition, the mixed audio and the operation video need to be synchronized, for example, the encoded audio data and video data may be Time-stamped according to the relative Time of the system, and then encapsulated into an RTMP (Real Time Messaging Protocol) data packet. And the RTMP data packet is sent to a content distribution network server for live broadcast service and further sent to terminal equipment of a user watching the live broadcast.
That is, in the audio and video processing method for live broadcasting according to the embodiment of the present application, mixing the mixed audio with the operation video and performing screen rendering of the operation video to obtain multimedia data for live broadcasting includes: synchronizing the mixed audio and the encoded video stream to obtain the multimedia data for live broadcast.
Moreover, in the audio and video processing method for live broadcasting according to the embodiment of the application, the method further includes: and sending the multimedia data for live broadcasting to a content distribution network server.
As described above, in the embodiment of the present application, it is also possible to capture a user video from a terminal device of a user, in which case, the operation video needs to be synchronized with the user video first, and then the picture rendering needs to be performed and synchronized with the mixed audio.
That is, in the audio and video processing method for live broadcasting according to the embodiment of the present application, mixing the mixed audio with the operation video and performing screen rendering of the operation video to obtain multimedia data for live broadcasting includes: synchronizing the operation video and the user video to obtain a mixed video; and performing picture rendering on the mixed video and synchronizing with the mixed audio to obtain multimedia data for live broadcasting.
Specifically, in the embodiments of the present application, synchronization between videos, and between audio and video may be performed in various ways. For example, the user video may be aligned in time with the operator video, or the video may be aligned in time with the audio, simply in an empirical way. Alternatively, the user video and the operation video may be synchronized per frame and the audio may be synchronized to each frame of the video according to respective time stamps of the video and the audio.
Alternatively, in the embodiment of the present application, video identification and audio identification may also be adopted to perform synchronization based on the content of the video and the audio, which may further enhance the synchronization effect between the videos and between the video and the audio.
Therefore, in the audio and video processing method for live broadcasting according to the embodiment of the application, synchronizing the operation video and the user video to obtain the mixed video includes: synchronizing the operation video and the user video according to experience; or performing video identification on the operation video and the user video and synchronizing the operation video and the user video based on the video identification result.
Furthermore, in the audio and video processing method for live broadcasting according to the embodiment of the application, performing screen rendering on the mixed video and synchronizing with the mixed audio to obtain multimedia data for live broadcasting includes: synchronizing the mixed video subjected to picture rendering with the mixed audio according to experience; or performing video recognition on the mixed video subjected to the picture rendering, performing audio recognition on the mixed audio, and synchronizing the mixed video subjected to the picture rendering with the mixed audio based on the results of the video recognition and the audio recognition.
In this embodiment of the present application, before obtaining the user audio related to the live broadcast operation, a message for prompting the user to send the user audio may be further sent to the terminal device of the user performing the live broadcast operation, so as to prompt the user performing the live broadcast operation to make a sound.
That is, in an audio and video processing method for live broadcasting according to an embodiment of the present application, before acquiring user audio related to a live broadcasting operation, the method further includes: and sending a message for prompting the user to send the user audio to the terminal equipment of the user carrying out the live broadcast operation.
The audio and video processing method for live broadcasting can be applied to various live broadcasting scenes, such as game live broadcasting, education live broadcasting, tour guide live broadcasting, telemedicine live broadcasting and the like. For example, in the case of a live education scene, the user audio may be audio of a teacher who is giving a lecture, audio collected from a classroom (including sound of multimedia courseware), or pre-recorded audio of a lecture of a teacher, etc., and a novel cloud education solution may be provided by combining with video for a live education.
In addition, the audio and video processing method for live broadcasting according to the embodiment of the application can be applied to scenes of multi-user live broadcasting. For example, a certain user as a host invites other users to participate in live broadcasting together, user audio can be collected from other users, and then the audio and video of the user in the cloud server are mixed through the audio and video processing method for live broadcasting according to the embodiment of the application, so as to obtain multimedia data for live broadcasting.
That is, in the audio and video processing method for live broadcasting according to the embodiment of the present application, the live broadcasting includes multi-person live broadcasting.
In addition, in the above audio and video processing method for live broadcasting, the user audio is an audio of a first person in the live broadcasting of multiple persons, and the operation audio and the operation video are an audio and a video of a second person in the live broadcasting of multiple persons.
In addition, in the above method for processing audio and video for live broadcasting, the first person is an invited party in the multi-person live broadcasting, and the second person is a host party in the multi-person live broadcasting.
Therefore, as the live broadcast picture is from the video stream of the cloud server instead of the screen recording on the mobile phone, the live broadcast picture quality can be improved, the risk of user privacy disclosure can be reduced, and meanwhile, the risk of illegal content live broadcast of the anchor broadcast on the live broadcast platform can be greatly reduced.
In addition, as the video stream is transmitted in the cloud server, the anchor does not need to care about the uplink bandwidth of the anchor, the configuration requirements of a mobile phone are lower, the anchor live broadcast threshold is greatly reduced, and the live broadcast stability and the watching effect are improved.
Exemplary method two
Fig. 3 illustrates a flowchart of an audio and video processing method for live game according to an embodiment of the present application.
As shown in fig. 3, an audio and video processing method for live game according to an embodiment of the present application includes: s210, acquiring a main broadcasting audio related to a game from a terminal device in the process that the game main broadcasts the game on a cloud server for live game broadcasting through the terminal device; s220, obtaining game audio and game video related to the game from the cloud server; s230, mixing the anchor audio and the game audio to obtain mixed audio; and S240, processing the mixed audio and the game video to obtain multimedia data for game live broadcast.
Fig. 4 illustrates a schematic diagram of an overall scheme of a live game according to an embodiment of the present application.
As shown in fig. 4, the scheme is used for live broadcasting of the cloud game at the mobile phone end, and the anchor can open a cloud game App and select live broadcasting before opening the game, that is, select live broadcasting on the cloud game App. By starting the game and selecting live broadcasting, the cloud game App establishes connection with the cloud game server to perform audio and video stream communication.
Before live broadcast, the anchor may set the video parameters and audio parameters of the live broadcast.
In the live broadcasting process, the cloud game App collects the sound of the anchor through a microphone, performs noise reduction and echo elimination processing to improve the sound quality, performs coding, and sends the coded sound of the anchor to the cloud game server. Similarly, the game sound collected by the microphone in the cloud game server can be subjected to noise reduction and echo cancellation processing.
Next, at the cloud game server, the main broadcast audio is decoded into PCM format, mixed with game audio, and then AAC audio encoded by FFmpeg.
On the other hand, the game pictures collected by the cloud game server can be subjected to format conversion (such as YUV format conversion) and video coding (such as H.265 video coding), then audio and video synchronization is carried out on the game pictures and audio, and then an RMTP data packet of the audio and video is pushed to a CDN server for live broadcast by a stream pusher. The CDN server can perform live transcoding service on the data packets acquired from the live streaming edge nodes and transmit the data packets to audience watching terminals through the live streaming edge nodes.
Therefore, the software of the audience watching end can obtain the transcoded audio and video stream from the CDN server to enjoy high-quality live game video.
Exemplary devices
Fig. 5 illustrates a block diagram of an audio-video processing device for live broadcasting according to an embodiment of the present application.
As shown in fig. 5, an audio/video processing apparatus 300 for live broadcasting according to an embodiment of the present application includes: a user acquisition unit 310 for acquiring user audio related to a live operation; a server acquisition unit 320, configured to acquire an operation audio and an operation video related to the live operation; an audio mixing unit 330, configured to mix the user audio and the operation audio to obtain a mixed audio; and an audio/video mixing unit 340, configured to mix the mixed audio with the operation video and perform picture rendering on the operation video to obtain multimedia data for live broadcasting.
In an example, in the above-mentioned audio-video processing apparatus 300 for live broadcasting, the user obtaining unit 310 is configured to: acquiring encoded user audio related to the live operation from a terminal device, the encoded user audio being obtained by encoding a sound of the user acquired using a microphone.
In an example, in the above-mentioned audio-video processing apparatus 300 for live broadcasting, the user obtaining unit 310 is further configured to: and acquiring parameters of audio processing and/or video processing from the terminal equipment.
In one example, in the above-mentioned audio-video processing apparatus 300 for live broadcasting, the parameter of the audio processing includes at least one of a sampling rate, a channel, and a bit rate of the audio; the parameters of the video processing comprise at least one of resolution, frame rate, code rate and key frame time interval of the video.
In an example, in the above-mentioned audio-video processing apparatus 300 for live broadcasting, the user obtaining unit 310 is further configured to: and collecting user videos related to the live broadcast operation from the terminal equipment.
In an example, in the above-mentioned audio/video processing apparatus 300 for live broadcasting, the server acquisition unit 320 is configured to: acquiring a barrage video related to the live broadcast operation as at least one part of the operation video.
In an example, in the above-mentioned audio/video processing apparatus 300 for live broadcasting, the server acquisition unit 320 is configured to: and acquiring the existing operation audio and operation video which are prepared in advance as the operation audio and operation video related to the live operation.
In an example, in the above-mentioned audio and video processing apparatus 300 for live broadcasting, the audio mixing unit 330 is configured to: mixing the user audio and the operation audio; and encoding the mixed audio according to the parameters of the audio processing to obtain mixed audio.
In an example, in the above-mentioned audio and video processing apparatus 300 for live broadcasting, the audio mixing unit 330 is configured to: receiving a first volume of the user audio and a second volume of the operation audio from the terminal device; and mixing the user audio and the operation audio according to the first volume and the second audio to obtain mixed audio.
In an example, in the above-mentioned audio and video processing apparatus 300 for live broadcasting, the audio and video mixing unit 340 is configured to: and carrying out video stream coding on the operation video according to the parameters of the video processing.
In an example, in the above-mentioned audio and video processing apparatus 300 for live broadcasting, the audio and video mixing unit 340 is configured to: synchronizing the mixed audio and the encoded video stream to obtain the multimedia data for live broadcast.
In an example, in the above-mentioned audio and video processing apparatus 300 for live broadcasting, the audio and video mixing unit 340 is configured to: synchronizing the operation video and the user video to obtain a mixed video; and performing picture rendering on the mixed video and synchronizing with the mixed audio to obtain multimedia data for live broadcasting.
In an example, in the above-mentioned audio and video processing apparatus 300 for live broadcasting, the audio and video mixing unit 340 is configured to: synchronizing the operation video and the user video according to experience; or performing video identification on the operation video and the user video and synchronizing the operation video and the user video based on the video identification result.
In an example, in the above-mentioned audio and video processing apparatus 300 for live broadcasting, the audio and video mixing unit 340 is configured to: synchronizing the mixed video subjected to picture rendering with the mixed audio according to experience; or performing video recognition on the mixed video subjected to the picture rendering, performing audio recognition on the mixed audio, and synchronizing the mixed video subjected to the picture rendering with the mixed audio based on the results of the video recognition and the audio recognition.
In an example, in the above-mentioned audio and video processing apparatus 300 for live broadcasting, further comprising: and the content push streaming unit is used for sending the multimedia data for live broadcasting to a content distribution network server.
In an example, in the above-mentioned audio and video processing apparatus 300 for live broadcasting, further comprising: and the prompting unit is used for further sending a message for prompting the user to send the user audio to the terminal equipment of the user carrying out the live broadcast operation before the user audio related to the live broadcast operation is acquired.
In one example, in the above-described audio/video processing apparatus 300 for live broadcasting, the live broadcasting includes at least one of a game live broadcasting, an education live broadcasting, a tour guide live broadcasting, and a telemedicine live broadcasting.
In one example, in the above-described audio video processing apparatus 300 for live broadcasting, the live broadcasting includes multi-person live broadcasting.
In one example, in the above-described audio-video processing apparatus 300 for live broadcasting, the user audio is audio of a first person in the live multi-person broadcasting, and the operation audio and the operation video are audio and video of a second person in the live multi-person broadcasting.
In one example, in the above-described audio-video processing apparatus 300 for a live broadcast, the first person is an invitee in the live multi-person broadcast, and the second person is a host in the live multi-person broadcast.
Fig. 6 illustrates a block diagram of an audio and video processing device for live game play according to an embodiment of the present application.
As shown in fig. 6, an audio/video processing apparatus 400 for live game according to an embodiment of the present application includes: an anchor audio acquiring unit 410, configured to acquire an anchor audio related to a game from a terminal device during a game being played by a game anchor on a cloud server for live game play through the terminal device; a game audio and video acquiring unit 420 for acquiring a game audio and a game video related to the game from the cloud server; an audio mixing unit 430 for mixing the anchor audio and the game audio to obtain mixed audio; and an audio and video mixing unit 440, configured to process the mixed audio and the game video to obtain multimedia data for live game play.
Here, it can be understood by those skilled in the art that the specific functions and operations of the respective units and modules in the intersection image processing apparatus 200 described above have been described in detail in the description of the intersection image processing method with reference to fig. 2 and 3, and thus, a repetitive description thereof will be omitted.
As described above, the audio/video processing apparatus 300 for live broadcast and the audio/video processing apparatus 400 for live game broadcast according to the embodiment of the present application may be implemented in various cloud server devices, such as a cloud server of a cloud game service provider. In one example, the audiovisual processing device 300 for live broadcast and the audiovisual processing device 400 for live game broadcast according to the embodiment of the present application may be integrated into a server device as one software module and/or hardware module. For example, it may be a software module in the operating system of the server device, or may be an application developed for the server device; of course, the audiovisual processing arrangement 300 for live broadcast and the audiovisual processing arrangement 400 for live game play may equally be one of many hardware modules of the server device.
Alternatively, in another example, the audiovisual processing device for live broadcast 300 and the audiovisual processing device for live game 400 and the server device may also be separate devices, and they may be connected to the server device through a wired and/or wireless network and transmit the interaction information according to an agreed data format.
Exemplary electronic device
Next, an electronic apparatus according to an embodiment of the present application is described with reference to fig. 7.
FIG. 7 illustrates a block diagram of an electronic device in accordance with an embodiment of the present application.
As shown in fig. 7, the electronic device 10 includes one or more processors 11 and memory 12.
The processor 11 may be a Central Processing Unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.
In one example, the electronic device 10 may further include: an input device 13 and an output device 14, which are interconnected by a bus system and/or other form of connection mechanism (not shown).
The input device 13 may include, for example, a keyboard, a mouse, and the like.
The output device 14 can output various information including multimedia data for live broadcasting and the like to the outside. The output devices 14 may include, for example, a display, speakers, a printer, and a communication network and its connected remote output devices, among others.
Of course, for simplicity, only some of the components of the electronic device 10 relevant to the present application are shown in fig. 7, and components such as buses, input/output interfaces, and the like are omitted. In addition, the electronic device 10 may include any other suitable components depending on the particular application.
Exemplary computer program product and computer-readable storage Medium
In addition to the above-described methods and apparatus, embodiments of the present application may also be a computer program product comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the audiovisual processing method for live broadcast and the audiovisual processing method for live game broadcast according to various embodiments of the present application described in the above-mentioned "exemplary methods" section of this specification.
The computer program product may be written with program code for performing the operations of embodiments of the present application in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the first user computing device, partly on the first user device, as a stand-alone software package, partly on the first user computing device and partly on a remote computing device, or entirely on the remote computing device or server.
Furthermore, embodiments of the present application may also be a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause the processor to perform the steps in the audiovisual processing method for live broadcast and the audiovisual processing method for live game broadcast according to various embodiments of the present application described in the above section of "exemplary methods" of this specification.
The computer-readable storage medium may take any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The foregoing describes the general principles of the present application in conjunction with specific embodiments, however, it is noted that the advantages, effects, etc. mentioned in the present application are merely examples and are not limiting, and they should not be considered essential to the various embodiments of the present application. Furthermore, the foregoing disclosure of specific details is for the purpose of illustration and description and is not intended to be limiting, since the foregoing disclosure is not intended to be exhaustive or to limit the disclosure to the precise details disclosed.
The block diagrams of devices, apparatuses, systems referred to in this application are only given as illustrative examples and are not intended to require or imply that the connections, arrangements, configurations, etc. must be made in the manner shown in the block diagrams. These devices, apparatuses, devices, systems may be connected, arranged, configured in any manner, as will be appreciated by those skilled in the art. Words such as "including," "comprising," "having," and the like are open-ended words that mean "including, but not limited to," and are used interchangeably therewith. The words "or" and "as used herein mean, and are used interchangeably with, the word" and/or, "unless the context clearly dictates otherwise. The word "such as" is used herein to mean, and is used interchangeably with, the phrase "such as but not limited to".
It should also be noted that in the devices, apparatuses, and methods of the present application, the components or steps may be decomposed and/or recombined. These decompositions and/or recombinations are to be considered as equivalents of the present application.
The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the application. Thus, the present application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing description has been presented for purposes of illustration and description. Furthermore, the description is not intended to limit embodiments of the application to the form disclosed herein. While a number of example aspects and embodiments have been discussed above, those of skill in the art will recognize certain variations, modifications, alterations, additions and sub-combinations thereof.
Claims (24)
1. An audio and video processing method for live broadcasting comprises the following steps:
acquiring user audio related to live broadcast operation;
acquiring operation audio and operation video related to the live operation;
mixing the user audio and the operation audio to obtain mixed audio; and
and mixing the mixed audio and the operation video and rendering the picture of the operation video to obtain multimedia data for live broadcasting.
2. The audio and video processing method for live broadcast according to claim 1, wherein acquiring user audio related to live broadcast operation comprises:
acquiring encoded user audio related to the live operation from a terminal device, the encoded user audio being obtained by encoding a sound of the user acquired using a microphone.
3. The audio-video processing method for live broadcast according to claim 1, further comprising:
and acquiring parameters of audio processing and/or video processing from the terminal equipment.
4. The audio-video processing method for a live broadcast according to claim 3, wherein,
the parameters of the audio processing comprise at least one of a sampling rate, a channel, and a bit rate of the audio;
the parameters of the video processing comprise at least one of resolution, frame rate, code rate and key frame time interval of the video.
5. The audio-video processing method for live broadcast according to claim 1, further comprising:
and collecting user videos related to the live broadcast operation from the terminal equipment.
6. The audio and video processing method for live broadcasting according to claim 1, wherein acquiring operation audio and operation video related to the live broadcasting operation includes:
acquiring a barrage video related to the live broadcast operation as at least one part of the operation video.
7. The audio and video processing method for live broadcasting according to claim 1, wherein acquiring operation audio and operation video related to the live broadcasting operation includes:
and acquiring the existing operation audio and operation video which are prepared in advance as the operation audio and operation video related to the live operation.
8. The audio and video processing method for live broadcasting of claim 4, wherein mixing the user audio and the operation audio to obtain mixed audio comprises:
mixing the user audio and the operation audio; and
and coding the audio after the audio mixing according to the parameters of the audio processing to obtain mixed audio.
9. The audio and video processing method for live broadcasting of claim 4, wherein mixing the user audio and the operation audio to obtain mixed audio comprises:
receiving a first volume of the user audio and a second volume of the operation audio from the terminal device; and
and mixing the user audio and the operation audio according to the first volume and the second audio to obtain mixed audio.
10. The audio and video processing method for live broadcasting according to claim 4, wherein mixing the mixed audio with the operation video and performing screen rendering of the operation video to obtain multimedia data for live broadcasting comprises:
and carrying out video stream coding on the operation video according to the parameters of the video processing.
11. The audio and video processing method for live broadcasting according to claim 10, wherein mixing the mixed audio with the operation video and performing screen rendering of the operation video to obtain multimedia data for live broadcasting comprises:
synchronizing the mixed audio and the encoded video stream to obtain the multimedia data for live broadcast.
12. The audio and video processing method for live broadcasting according to claim 5, wherein mixing the mixed audio with the operation video and performing screen rendering of the operation video to obtain multimedia data for live broadcasting comprises:
synchronizing the operation video and the user video to obtain a mixed video; and
and performing picture rendering on the mixed video and synchronizing with the mixed audio to obtain multimedia data for live broadcasting.
13. The audio-video processing method for a live broadcast of claim 12, wherein synchronizing the operational video with the user video to obtain a mixed video comprises:
synchronizing the operation video and the user video according to experience; or
And performing video identification on the operation video and the user video and synchronizing the operation video and the user video based on the video identification result.
14. The audio and video processing method for live broadcasting of claim 12, wherein the screen rendering and the synchronization of the mixed video with the mixed audio to obtain multimedia data for live broadcasting comprises:
synchronizing the mixed video subjected to picture rendering with the mixed audio according to experience; or
And performing video identification on the mixed video subjected to the picture rendering, performing audio identification on the mixed audio, and synchronizing the mixed video subjected to the picture rendering with the mixed audio based on the results of the video identification and the audio identification.
15. The audio-video processing method for live broadcast according to claim 1, further comprising:
and sending the multimedia data for live broadcasting to a content distribution network server.
16. The audio-video processing method for live broadcast according to claim 1, wherein prior to acquiring the user audio related to the live broadcast operation, further comprising:
and sending a message for prompting the user to send the user audio to the terminal equipment of the user carrying out the live broadcast operation.
17. The audio and video processing method for a live broadcast according to claim 1, wherein the live broadcast comprises at least one of a game live broadcast, an education live broadcast, a tour guide live broadcast, and a telemedicine live broadcast.
18. The audio and video processing method for a live broadcast according to claim 1, wherein the live broadcast comprises a multi-person live broadcast.
19. The audio-video processing method for a live broadcast according to claim 15, wherein the user audio is audio of a first person in the live multi-person broadcast, and the operation audio and the operation video are audio and video of a second person in the live multi-person broadcast.
20. A method of audiovisual processing for a live broadcast in accordance with claim 19, wherein the first person is an invitee in the multi-person live broadcast and the second person is a moderator in the multi-person live broadcast.
21. An audio and video processing method for live game includes:
in the process that a game anchor plays a game on a cloud server for live game play through terminal equipment, anchor audio related to the game is acquired from the terminal equipment;
obtaining game audio and game video related to the game from the cloud server;
mixing the anchor audio with the game audio to obtain mixed audio; and
and processing the mixed audio and the game video to obtain multimedia data for game live broadcast.
22. An audio and video processing apparatus for live broadcasting, comprising:
a user acquisition unit for acquiring a user audio related to a live operation;
the server acquisition unit is used for acquiring operation audio and operation video related to the live broadcast operation;
an audio mixing unit configured to mix the user audio and the operation audio to obtain a mixed audio; and
and the audio and video mixing unit is used for mixing the mixed audio and the operation video and performing picture rendering on the operation video to obtain multimedia data for live broadcasting.
23. An audio-video processing device for live game play, comprising:
the system comprises an anchor audio acquisition unit, a terminal device and a server, wherein the anchor audio acquisition unit is used for acquiring anchor audio related to a game from the terminal device in the process that the game anchor plays the game on the cloud server for live game through the terminal device;
a game audio and video acquisition unit for acquiring a game audio and a game video related to the game from the cloud server;
an audio mixing unit for mixing the anchor audio with the game audio to obtain a mixed audio; and
and the audio and video mixing unit is used for processing the mixed audio and the game video to obtain multimedia data for live game broadcast.
24. An electronic device, comprising:
a processor; and
a memory having stored therein computer program instructions which, when executed by the processor, cause the processor to perform an audiovisual processing method for a live broadcast as claimed in any of claims 1-20 or an audiovisual processing method for a live game as claimed in claim 21.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010753100.7A CN113301359A (en) | 2020-07-30 | 2020-07-30 | Audio and video processing method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010753100.7A CN113301359A (en) | 2020-07-30 | 2020-07-30 | Audio and video processing method and device and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113301359A true CN113301359A (en) | 2021-08-24 |
Family
ID=77318238
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010753100.7A Pending CN113301359A (en) | 2020-07-30 | 2020-07-30 | Audio and video processing method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113301359A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114363648A (en) * | 2021-12-30 | 2022-04-15 | 上海网达软件股份有限公司 | Method, equipment and storage medium for audio and video alignment in mixed flow process of live broadcast system |
| CN114979692A (en) * | 2022-05-23 | 2022-08-30 | 广州欢城文化传媒有限公司 | Switching method, device and system of audio and video stream pulling mode and storage medium |
| WO2023058330A1 (en) * | 2021-10-06 | 2023-04-13 | ソニーグループ株式会社 | Information processing device, information processing method, and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104967960A (en) * | 2015-03-25 | 2015-10-07 | 腾讯科技（深圳）有限公司 | Voice data processing method, and voice data processing method and system in game live broadcasting |
| CN110213601A (en) * | 2019-04-30 | 2019-09-06 | 大鱼互联科技(深圳)有限公司 | A kind of live broadcast system and live broadcasting method based on cloud game, living broadcast interactive method |
| CN111314724A (en) * | 2020-02-18 | 2020-06-19 | 华为技术有限公司 | Cloud game live broadcasting method and device |
-
2020
- 2020-07-30 CN CN202010753100.7A patent/CN113301359A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104967960A (en) * | 2015-03-25 | 2015-10-07 | 腾讯科技（深圳）有限公司 | Voice data processing method, and voice data processing method and system in game live broadcasting |
| CN110213601A (en) * | 2019-04-30 | 2019-09-06 | 大鱼互联科技(深圳)有限公司 | A kind of live broadcast system and live broadcasting method based on cloud game, living broadcast interactive method |
| CN111314724A (en) * | 2020-02-18 | 2020-06-19 | 华为技术有限公司 | Cloud game live broadcasting method and device |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023058330A1 (en) * | 2021-10-06 | 2023-04-13 | ソニーグループ株式会社 | Information processing device, information processing method, and storage medium |
| CN114363648A (en) * | 2021-12-30 | 2022-04-15 | 上海网达软件股份有限公司 | Method, equipment and storage medium for audio and video alignment in mixed flow process of live broadcast system |
| CN114979692A (en) * | 2022-05-23 | 2022-08-30 | 广州欢城文化传媒有限公司 | Switching method, device and system of audio and video stream pulling mode and storage medium |
| CN114979692B (en) * | 2022-05-23 | 2023-09-15 | 广州欢城文化传媒有限公司 | Audio and video streaming mode switching method, device and system and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10187668B2 (en) | Method, system and server for live streaming audio-video file | |
| WO2019205872A1 (en) | Video stream processing method and apparatus, computer device and storage medium | |
| EP3562163B1 (en) | Audio-video synthesis method and system | |
| US10244291B2 (en) | Authoring system for IPTV network | |
| CN113301359A (en) | Audio and video processing method and device and electronic equipment | |
| CN103327361B (en) | Acquisition methods, the apparatus and system of real-time video communication playback data stream | |
| WO2016150317A1 (en) | Method, apparatus and system for synthesizing live video | |
| CN112019927B (en) | Video live broadcast method, microphone connecting equipment, live broadcast system and storage medium | |
| US20080288990A1 (en) | Interactive Broadcasting System | |
| KR101550462B1 (en) | A method and an apparatus for embedding data in a media stream | |
| CN106803923B (en) | RTMP plug-flow real-time live broadcast system | |
| WO2011050690A1 (en) | Method and system for recording and replaying multimedia conference | |
| WO2015007137A1 (en) | Videoconference terminal, secondary-stream data accessing method, and computer storage medium | |
| CN112135155A (en) | Audio and video connecting and converging method and device, electronic equipment and storage medium | |
| CN109753259A (en) | A kind of throwing screen system and control method | |
| CN114554277A (en) | Multimedia processing method, device, server and computer readable storage medium | |
| US20140112636A1 (en) | Video Playback System and Related Method of Sharing Video from a Source Device on a Wireless Display | |
| CN111510720A (en) | Real-time streaming media data transmission method, electronic device and server | |
| CN105744347A (en) | Method of network media terminal for improving user audio-visual experience | |
| CN114363648A (en) | Method, equipment and storage medium for audio and video alignment in mixed flow process of live broadcast system | |
| CN106792117A (en) | The methods, devices and systems that the live transverse screen of mobile phone starts broadcasting | |
| WO2024087732A1 (en) | Livestreaming data processing method and system | |
| CN102438119A (en) | Audio/video communication system of digital television | |
| JP7153143B2 (en) | Video providing system and program | |
| CN110392276B (en) | Live broadcast recording and broadcasting method based on Real Time Messaging Protocol (RTMP) synchronous transmission MIDI |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20221111 Address after: China Hongkong Tongluowan 33 hysanavenue Lee Garden Phase 19 Building Room 1901 Applicant after: Lingxi Interactive Entertainment Holding Co.,Ltd. Address before: Box 847, four, Grand Cayman capital, Cayman Islands, UK Applicant before: ALIBABA GROUP HOLDING Ltd. |