WO2022237463A1

WO2022237463A1 - Livestreaming background sound processing method and apparatus, device, medium, and program product

Info

Publication number: WO2022237463A1
Application number: PCT/CN2022/087482
Authority: WO
Inventors: 陈映宜
Original assignee: 北京字节跳动网络技术有限公司
Priority date: 2021-05-13
Filing date: 2022-04-18
Publication date: 2022-11-17
Also published as: CN113132794A

Abstract

Embodiments of the present disclosure provide a livestreaming background sound processing method and apparatus, a device, a medium, and a program product. The method comprises: first displaying a background sound operation control on a display screen when a preset display condition is satisfied; then sending an audio processor a corresponding first audio code stream in a music playing application in response to an operation instruction of a user on the background sound operation control; and finally, the audio processor performing convergence and synthesis on a first audio data stream and a second audio code stream to determine an input audio code stream of the livestreaming application, the second audio code stream comprising a sound signal sent by a livestreamer and/or a sound signal in a livestreaming environment received by a client by means of an audio acquisition apparatus. The embodiments of the present disclosure solve the technical problem in the prior art that when facing a high-quality livestreaming requirement, the addition of background sound can only depend on a complex professional device and a professional technician team. The technical effect that background sound can be conveniently and quickly added by a livestreamer on a livestreaming terminal is achieved.

Description

Live broadcast background sound processing method, device, equipment, medium and program product

Related Application Cross Reference

This application claims the priority of the Chinese patent application submitted to the China Patent Office on May 13, 2021, with the application number 202110522604.2, and the title of the invention is "Live background sound processing method, device, equipment, medium and program product", the entire content of which Incorporated herein by reference.

technical field

Embodiments of the present disclosure relate to the technical field of broadcast television and Internet live broadcast, and in particular to a method, device, equipment, medium and program product for processing live background sound.

Background technique

With the continuous and in-depth development of social informatization, people's demand for the timeliness of information acquisition is also increasing. Live broadcast technology enables people to break through the limitations of space and obtain various information on the live broadcast site in real time. In recent years, webcasting has also become a platform for people to show themselves, and has gained great social popularity. Mobile live broadcasting lowers the technical threshold of live broadcasting, allowing ordinary people without professional skills to simply broadcast live broadcasting. As the audience's requirements for live broadcast content and quality continue to increase, a large number of peripheral auxiliary products have been derived around mobile live broadcast devices such as mobile phones.

At present, a high-quality mobile live broadcast requires hardware equipment including: desktop computers, sound cards, several intermediate converters/connectors, mobile phones, earphones, microphones, etc. Real-time addition of a live broadcast effect.

That is to say, the existing technology has the technical problem of only relying on complex professional equipment and a team of professional technicians to add background sound when facing high-quality live broadcast requirements. This undoubtedly raised the technical threshold for live broadcasting and increased the cost of live broadcasting for anchors.

Contents of the invention

Embodiments of the present disclosure provide a live broadcast background sound processing method, device, equipment, media, and program products to solve the existing technology that can only rely on complex professional equipment and professional technical personnel teams when faced with high-quality live broadcast requirements technical issues to add background sound.

In the first aspect, the embodiment of the present disclosure provides a live broadcast background sound processing method, which is applied to a user end, and the user end includes: a main controller and an audio processor, and the main controller is installed with a live broadcast application and a music playback application, The method includes:

When the preset display conditions are met, the background sound operation controls are displayed on the display screen;

Responding to the user's operation instruction on the background sound operation control, sending the corresponding first audio code stream in the music playing application to the audio processor;

The audio processor concatenates and synthesizes the first audio data stream and the second audio code stream to determine the input audio code stream of the live broadcast application, and the second audio code stream includes The device receives the sound signal from the anchor, and/or the sound signal in the live broadcast environment.

In a second aspect, an embodiment of the present disclosure provides a live broadcast background sound processing device, including: a main control module and an audio processing module, where a live broadcast application and a music playback application are installed in the main control module; wherein,

The main control module is used to display background sound operation controls on the display screen when preset display conditions are met;

The main control module is further configured to send the corresponding first audio code stream in the music playing application to the audio processing module in response to the user's operation instruction on the background sound operation control;

The audio processing module is configured to concatenate and synthesize the first audio data stream and the second audio code stream to determine the input audio code stream of the live application, and the second audio code stream includes Receive the sound signal from the anchor, and/or the sound signal in the live broadcast environment.

In a third aspect, an embodiment of the present disclosure provides an electronic device, including:

at least one processor and memory;

the memory stores a computer program;

The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the live broadcast background sound processing method described in the above first aspect and various possible designs of the first aspect.

In a fourth aspect, an embodiment of the present disclosure provides a live streaming all-in-one machine, including any possible electronic device provided in the third aspect.

In the fifth aspect, the embodiments of the present disclosure provide a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the processor executes the computer program, the above first aspect and each of the first aspect are realized. A possible design of the described live background sound processing method.

In a sixth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the live broadcast background sound processing method described in the above first aspect and various possible designs of the first aspect .

In the seventh aspect, the embodiments of the present disclosure provide a computer program. When the computer program is executed by a processor, it implements the live broadcast background sound processing method described in the first aspect and various possible designs of the first aspect.

The live broadcast background sound processing method, device, equipment, medium and program product provided by the embodiments of the present disclosure, the method first displays the background sound operation control on the display screen when the preset display condition is met, and then responds to the user's operation on the background sound The operation command of the control sends the corresponding first audio code stream in the music player application to the audio processor, and finally the audio processor merges and synthesizes the first audio data stream and the second audio code stream to determine the input of the live broadcast application Audio stream. The second audio code stream includes the sound signal sent by the anchor received by the client through the audio collection device, and/or the sound signal in the live broadcast environment. The embodiments of the present disclosure solve the technical problem in the prior art that the background sound can only be added by relying on complex professional equipment and a team of professional technicians in the face of high-quality live broadcast requirements. It achieves the technical effect that the host can conveniently and quickly add background sound to the live audio stream on the live broadcast terminal, reduces the cost and technical threshold of live broadcast, and improves the user experience.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description These are some embodiments of the present disclosure. Those skilled in the art can also obtain other drawings based on these drawings without any creative effort.

FIG. 1 is a schematic structural diagram of a live broadcast equipment kit used by an existing indoor anchor provided by an embodiment of the present disclosure;

FIG. 2 is a first schematic flow diagram of a method for processing live background sound provided by an embodiment of the present disclosure;

FIG. 3 is a schematic diagram of a background audio code stream processed by a client provided by an embodiment of the present disclosure;

FIG. 4 is a second schematic flow diagram of a method for processing live background sound provided by an embodiment of the present disclosure;

5a-5c are schematic diagrams of a user terminal display screen split-screen display background sound operation area provided by an embodiment of the present disclosure;

FIG. 6 is a third schematic flowchart of a method for processing live broadcast background sound provided by an embodiment of the present disclosure;

7a-7e are schematic diagrams of displaying background sound operation controls in the form of a floating window according to an embodiment of the present disclosure;

FIG. 8 is a structural block diagram of a live broadcast background sound processing device provided by an embodiment of the present disclosure;

FIG. 9 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments It is a part of the embodiments of the present disclosure, but not all of them. Based on the embodiments in the present disclosure, all other embodiments obtained by persons of ordinary skill in the art without creative work, including but not limited to combinations of multiple embodiments, fall within the protection scope of the present disclosure.

The terms "first", "second", "third", "fourth", etc. (if any) in the description and claims of the present disclosure and the above drawings are used to distinguish similar objects, and not necessarily Used to describe a specific sequence or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the disclosure described herein are, for example, capable of practice in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having", as well as any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device comprising a sequence of steps or elements is not necessarily limited to the expressly listed instead, may include other steps or elements not explicitly listed or inherent to the process, method, product or apparatus.

With the continuous development of mobile Internet technology, people's daily life has become highly informatized, and live broadcast technology is an efficient means for people to receive live broadcast information in different geographical spaces. The live webcast that has emerged in recent years has added a lot of fun to people's entertainment life and has been sought after by people. A large number of anchors have joined the ranks of the webcast. Live broadcasting through mobile devices is favored by many anchors for its simplicity. However, as the audience's requirements for live broadcast quality continue to improve, the anchor needs to add live background music when performing live broadcast.

Faced with this demand, for most anchors, it is only possible to purchase a set of professional live broadcast equipment, including: desktop computers, professional sound cards, several intermediate converters/connectors and other professional live broadcast auxiliary equipment to realize live broadcast. Live background sound processing requirements.

FIG. 1 is a schematic structural diagram of a live broadcast equipment kit used by an existing indoor anchor provided by an embodiment of the present disclosure. As shown in FIG. 1 , the live broadcast equipment kit includes: a mobile terminal 101 , an intermediate adapter 102 , an intermediate converter 103 , a sound card 104 , and a desktop computer 105 .

Wherein, the live application is installed in the mobile terminal 101 .

The intermediate adapter 102 is used to solve the problem that the charging interface of the mobile terminal 101 cannot be charged because the charging interface is occupied by the data line when the mobile terminal 101 is live broadcasting for a long time.

The intermediate converter 103 is used to transfer the special effect data in the desktop computer 105 to the mobile terminal 101, and after connecting the earphones, it can provide the customer with an ear return function, and it can also solve the problem that the sound card 104 can only output a single channel and cannot receive live broadcast in reverse A problem with the audio data output by the app.

The sound card 104 is connected with the microphone to convert the audio analog signal into a digital signal, and input it into the desktop computer 105 for various sound effect processing.

However, the method shown in Figure 1 increases the cost of live broadcasting and requires the configuration of multiple auxiliary equipment, and most anchors do not have professional skills and cannot configure this set of equipment by themselves, so a dedicated team of technical personnel is also required To build, operate and maintain this set of professional live broadcast equipment, these have raised the live broadcast threshold that has been lowered through mobile terminals such as mobile phones.

Moreover, due to the connection of multiple devices, the stability of the entire set of live broadcast equipment is easily affected by various factors. If there is a problem with the connector socket of any device, or the line is broken for some reason, the system will not work.

Based on the above-mentioned technical problems, the inventive concept of the present application aims at:

Through a client, the anchor can quickly and easily add the required audio to the live audio stream through the screen during the live broadcast, that is, set the live background music. The live broadcast does not require multiple devices, and only one small device can complete the background sound processing of the live broadcast, and does not require a team of professional and technical personnel to participate in operation and maintenance. The anchor can select and add audio such as background music in real time during the live broadcast.

The live broadcast background sound processing method provided by the present application will be described in detail below with reference to the accompanying drawings.

Referring to FIG. 2 , FIG. 2 is a first schematic flowchart of a method for processing live background sound provided by an embodiment of the present disclosure. The method of this embodiment is applied to the user end, and the implementation form of the user end includes a device with a live broadcast function. The device with a live broadcast function includes but is not limited to: a live broadcast integrated terminal, a mobile terminal, and a computer device. Users can use the live broadcast integrated The live broadcast can be directly completed by a mobile terminal or a mobile terminal, without using other supporting equipment to perform additional processing on the live audio or video.

The live broadcast background sound processing method includes:

S201: When the preset display condition is met, display the background sound operation control on the display screen.

In this embodiment, the client includes: a main controller and an audio processor, and a live broadcast application and a music playing application are installed on the main controller.

In this step, the preset display conditions include: detection of an instruction to turn on the camera device, detection of a live broadcast terminal moving in a preset motion mode (such as shaking, circular motion, flipping motion, etc.), receiving a preset voice command and At least one of an opening instruction corresponding to a preset button or a switch control is received.

It should be noted that the type of the display screen can be a touch screen or a non-touch screen.

Specifically, for example, the user clicks the shortcut control displayed on the live broadcast interface to display the background sound operation control.

In a possible design, the display mode of the background sound operation control includes: split screen, and/or, floating window display.

In this embodiment, the user terminal includes at least one display screen. In a possible design, the user terminal can be connected to multiple external display screens through an extension interface, and the extension interface includes a wireless interface and a wired interface.

In a possible design, when the number of display screens on the user terminal is greater than one, the background sound operation control is displayed on the display screen where the non-live broadcast interface is located.

S202: Send the corresponding first audio code stream in the music playing application to the audio processor in response to the user's operation instruction on the background sound operation control.

In this step, the operation instructions for the background sound operation control include: pause, play, previous song, next song, playlist, list or playlist loop playback, sequential play, music search, lyrics search, list search Wait for at least one of the operations.

Specifically, the background sound operation control can be used as a display medium of the music playing application, and the operation interface of the music playing application is directly copied and projected onto the preset display area of the background sound operation control. In this way, the anchor can directly search for the background music needed, play, pause, etc., like operating a music player application, or select a playlist in the preset song list, and then choose a playback method, such as: play in order of playlist, random Play, list loop playback, single loop playback, etc.

It should be noted that, in the prior art, when multiple applications are installed on a terminal, and if there are multiple applications that need to use the same interface or external device when they are opened at the same time, these applications will conflict. For example, during a live broadcast on a mobile phone, the live broadcast application needs to occupy the headphone jack or the speaker. At this time, if you want to open the audio playback application on the mobile terminal, since the audio playback application also needs to use the speaker or the headphone jack, this creates a conflict between the two applications. , if the user insists on opening the audio playback application, the live broadcast application will be forced to close.

This problem in the prior art makes it impossible for the anchor to directly open the audio playback application on the mobile phone or other mobile terminals to play the background music. To solve this problem, anchors usually need another music playback device to play music in the live broadcast environment, so that the microphone can collect the music sound in the live broadcast environment, and then add it to the input audio stream of the live broadcast application.

Or as shown in Figure 1, through the form of an external sound card 104 and an intermediate converter 103, a desktop computer 105 is used to start a music player application, and the audio stream generated by it is input to an external independent sound card 104, and the microphone is not directly connected to the external sound card 104. The mobile phone or mobile terminal 101 is connected to the sound card 104 instead, so that the sound card 104 synthesizes the audio code stream generated by the music player application and the sound code stream collected by the microphone.

Apparently, in the prior art mentioned above, the contradiction of resource occupation when the two applications are started at the same time can be solved by separating the live broadcast application and the music playback application into two different devices, and then adding an intermediate connection device. But this will greatly increase the cost of live broadcast and the threshold of live broadcast technology.

This embodiment, on the other hand, breaks the technical barrier that two applications cannot be installed on the same device, and installs the live broadcast application and the music player application on the same terminal, but the opening and operation methods of the music player application during the live broadcast , but in a different way than when not live. Call the music playback application through the background sound operation control, so that the original music playback application needs to occupy peripheral resources such as headphone jacks and speakers when it is started, and instead occupies the audio processor resources in the terminal device corresponding to the background sound operation control. Processor resources can be physical hardware devices such as IO pin interfaces corresponding to audio processing chips, or virtual resources such as virtual sound cards, virtual headphones, and virtual speakers.

It should be noted that the virtual resources need to be implemented by the main controller through a special interface program or application module, which can be selected by those skilled in the art according to the actual situation, which is not limited in this application.

Through the combination of visual background sound operation controls and audio processors, external auxiliary live broadcast equipment such as desktop computers, external independent sound cards, intermediate converters, intermediate adapters, etc. are eliminated, making it easy to add background music during live broadcasting. It is fast and reduces the cost of equipment, and does not need to rely on the support of a professional technical team. The anchor can easily live broadcast by one person.

S203: The audio processor concatenates and synthesizes the first audio data stream and the second audio code stream, so as to determine the input audio code stream of the live application.

In this step, the second audio code stream includes the sound signal sent by the anchor received by the client through the audio collection device, and/or the sound signal in the live broadcast environment.

In this embodiment, a preset synthesis algorithm is used to synthesize the first audio data stream and the second audio code stream into an input audio code stream;

The live application receives the input audio code stream and encodes the input audio code stream to determine the output audio code stream;

The output audio code stream is sent from the network interface to the network server, so that the network server sends the output audio code stream to the live viewing terminal used by each audience.

That is, the live broadcast application encodes the input audio stream, transmits it to the live broadcast platform server through the network, and then sends it to each viewer by the live broadcast platform service.

FIG. 3 is a schematic diagram of processing a background audio code stream at a user terminal according to an embodiment of the present disclosure. As shown in FIG. 3 , the client terminal 300 includes a main controller 31 and an audio processor 32 , wherein a live broadcast application 311 and a music playing application 312 are installed in the main controller 31 . After opening the live broadcast application, the user opens the background sound operation control by clicking the shortcut key, thereby calling the music playback application 312, and the user selects the audio played as the background music from the music playback application 312 by operating the background sound operation control, The music player application 312 generates a continuous audio code stream, that is, the first audio data stream, and inputs it into the audio processor 32 .

Audio processor 32 is also connected (as USB (Universal Serial Bus, Universal Serial Bus) audio input interface, 3.5mm interface microphone audio input interface, Canon head interface microphone audio input interface) with each audio collection interface of client 300, with The audio of the anchor received by each audio collection device or the audio of the environment where the anchor is located is the second audio stream.

The audio processor 32 inputs the received first audio code stream and the second audio code stream into a preset synthesis algorithm model to determine a synthesized audio code stream as an input audio code stream of the live application 311 .

It should be noted that those skilled in the art can select a specific preset synthesis algorithm model according to actual needs, which is not limited in this application.

In the live broadcast background sound processing method provided by the embodiments of the present disclosure, firstly, when the preset display conditions are met, the background sound operation control is displayed on the display screen, and then in response to the user's operation instruction on the background sound operation control, the corresponding The first audio code stream is sent to the audio processor, and finally the audio processor merges and synthesizes the first audio data stream and the second audio code stream to determine the input audio code stream of the live application. The second audio code stream includes the sound signal sent by the anchor received by the client through the audio collection device, and/or the sound signal in the live broadcast environment. The embodiment of the present disclosure solves the technical problem in the prior art that it can only rely on complex professional equipment and a team of professional technicians to add background sound when facing high-quality live broadcast requirements. It achieves the technical effect that the host can conveniently and quickly add background sound to the live audio stream on the live broadcast terminal, reduces the cost and technical threshold of live broadcast, and improves the user experience.

In order to further illustrate the split-screen display in S201 and the background sound operation controls displayed in the floating window, the following two specific embodiments are used to describe them respectively.

Referring to FIG. 4 , FIG. 4 is a second schematic flowchart of a method for processing live background sound provided by an embodiment of the present disclosure. The method of this embodiment is applied to the user end, and the implementation form of the user end includes a device with a live broadcast function. The device with a live broadcast function includes but is not limited to: a live broadcast integrated terminal, a mobile terminal, and a computer device. Users can use the live broadcast integrated The live broadcast can be directly completed by a mobile terminal or a mobile terminal, without using other supporting equipment to perform additional processing on the live audio or video.

The live broadcast background sound processing method includes:

S401: Obtain an opening instruction issued by the user.

In this step, the enabling instruction includes: clicking the shortcut key button or control for enabling the background sound on the live broadcast interface.

In this embodiment, after the user starts the live broadcast application on the user terminal, the user clicks to start live broadcast, and the live broadcast application starts the camera on the user terminal. And when the user terminal detects that the camera is turned on, the split-screen control is displayed on the edge of the display screen.

In a possible design, the background sound activation control includes a floating window control for split-screen display, that is, a split-screen control, and the main controller at the user end can detect the activation instruction when the user clicks the split-screen control.

S402: Simultaneously display background sound operation controls on the live broadcast interface according to the preset display mode indicated by the activation instruction.

In this embodiment, the preset display mode includes: a split-screen display mode.

Specifically, at least one display screen of the user terminal is divided into a live interface area and a background sound operation area.

It should be noted that the boundaries of the live interface area and the background sound operation area include: straight line boundaries and/or curved boundaries.

5a-5c are schematic diagrams of a display screen of a user terminal for displaying a background sound operation area in split screens according to an embodiment of the present disclosure. As shown in FIG. 5 a , after the user clicks “split screen control” on the display screen of the user terminal 500, the display screen will be divided into a live interface area 510 and a background sound operation area 520.

It should be noted that the content displayed in the background sound operation control, that is, the background sound operation area 520 includes: all or part of the content of the operation interface of the music playing application.

The user terminal 500 maps and corresponds each control in the background sound operation area 520 to the operation interface of the music playing application through the background control algorithm.

As shown in Figure 5a, the background sound operation area 520 includes:

The song list sub-control 521 is used to switch the song list;

Search sub-control 522 for searching songs;

The menu sub-control 523 is used to display more functional options of the music player application;

The operation sub-control 524 is used to implement operations such as play, pause, stop, previous song, and next song.

Of course, it can be understood that the background sound operation area 520 may also include information controls such as playback progress, volume control, song title, singer, lyricist/musician, and the like. It should be noted that those skilled in the art can select the layout and implementation style of the background sound operation area 520 according to the actual situation, which is not limited in this application.

In a possible design, the split screen can also be a left and right split screen, as shown in Figure 5b, after the split screen, the ratio of the live broadcast interface area 510 and the background sound operation area 520 to the display screen can be adjusted according to actual needs. For example: the background sound operation area 520 occupies 30%-50%.

It should be noted that the boundary between the live broadcast interface area 510 and the background sound operation area 520 after screen splitting can also be any geometric shape surrounded by straight lines and curves. As shown in FIG. 5 c , there may be multiple background sound operation areas 520 , and each sub-control is distributed in multiple background sound operation areas 520 .

S403: Obtain the user's operation instruction on the background sound operation control.

In this embodiment, the user's operation instructions on the background sound operation controls include: various operation instructions of the user in the background sound operation area 520 , and operation instructions on the boundary of the background sound operation area 520 .

The user's various operating instructions in the background sound operation area 520, including: pause, play, previous song, next song, playlist, list or playlist cycle play, sequential play, music search, lyrics search, ranking list At least one of the search. The specific function of these operation instructions is to select the audio data or audio code stream as the background sound of the live broadcast in the music playing application.

The operation instructions on the border of the background sound operation area 520 include: the user changes the live interface area and the border adjacent to the background sound operation area in a preset manner to change the display state of the background sound operation control, and the display state includes: on or off , expand or shrink the background sound operation area.

Specifically, in the background sound operation area 520 shown in Figures 5a and 5b, the user can close the background sound operation area 520 or change the background sound operation by clicking at least one point on the boundary or clicking at least one straight line segment on the boundary. The proportional size of the area 520.

In a possible design, the user presses and holds at least one point on the boundary, and then slides up and down or left and right to change the proportion of the background sound operation area 520 . As shown in FIG. 5 a , the user presses a point on the boundary, and then slides up and down or left and right to change the proportion of the background sound operation area 520 .

What needs to be explained here is that when sliding left and right, it can also be understood that the user's finger slides along the path indicated by the boundary. When sliding from left to right, it means expanding the proportion of the background sound operation area 520; from right to left When sliding, it means reducing the proportion of the background sound operation area 520 .

As shown in Figure 5c, this kind of boundary has both straight lines and curves. Different operation methods can be set. For example, when the user presses the straight line segment of the boundary and slides in the direction perpendicular to the straight line segment, the Expand or shrink the background sound operation area 520 in the direction of .

When the user presses the boundary of the curve and slides in the direction of the tangent of the pressed point on the curve, or in the vertical direction of the tangent, the background sound operation area 520 is expanded or reduced according to the corresponding direction.

When the user slides along the curved boundary or the straight boundary, the proportion of the background sound operation area 520 can also be changed.

S404: Send the corresponding first audio code stream in the music playing application to the audio processor.

When the user selects and starts to play a certain audio in the background sound operation area 520, the corresponding first audio code stream generated when the audio data is played in the music playing application is sent to the audio processor.

S405: The audio processor concatenates and synthesizes the first audio data stream and the second audio code stream, so as to determine the input audio code stream of the live application.

Please refer to S203 for the specific principle and terminology explanation of this step, which will not be repeated here.

In the live broadcast background sound processing method provided by the embodiments of the present disclosure, first, when the preset display conditions are met, the background sound operation control is displayed on the display screen in a split-screen form, and then in response to the user's operation instruction on the background sound operation control, the The corresponding first audio code stream in the music playing application is sent to the audio processor, and finally the audio processor merges and synthesizes the first audio data stream and the second audio code stream to determine the input audio code stream of the live application. The second audio code stream includes the sound signal sent by the anchor received by the client through the audio collection device, and/or the sound signal in the live broadcast environment. The embodiments of the present disclosure solve the technical problem in the prior art that the background sound can only be added by relying on complex professional equipment and a team of professional technicians in the face of high-quality live broadcast requirements. It achieves the technical effect that the host can easily and quickly add background sound to the live audio stream on the live broadcast terminal, reduces the cost and technical threshold of live broadcast, and improves the user experience.

FIG. 6 is a third schematic flowchart of a method for processing live broadcast background sound provided by an embodiment of the present disclosure. The method of this embodiment is applied to the user end, and the implementation form of the user end includes a device with a live broadcast function. The device with a live broadcast function includes but is not limited to: a live broadcast integrated terminal, a mobile terminal, and a computer device. Users can use the live broadcast integrated The live broadcast can be directly completed by a mobile terminal or a mobile terminal, without using other supporting equipment to perform additional processing on the live audio or video.

The live broadcast background sound processing method includes:

S601: Obtain an opening instruction issued by the user.

Please refer to S401 for the specific principles and terminology explanation of this step, which will not be repeated here.

S602: According to the opening instruction, simultaneously display the background sound operation controls on the live broadcast interface in the preset display mode of the floating window.

In this embodiment, the background sound operation control is superimposed and displayed on the live broadcast interface in the form of a floating window.

7a-7e are schematic diagrams of displaying background sound operation controls in the form of a floating window according to an embodiment of the present disclosure. As shown in FIG. 7 a , after the user clicks the shortcut key of “Background Music” on the client terminal 700 , the floating window 710 , which is the background sound operation control, is superimposed and displayed on the live broadcast interface.

The floating window 710 includes: a song icon 711 , a search control 712 , a menu control 713 , a play mode control 714 and a zoom control 715 .

In a possible design, the song icon 711 is used to display the cover image of the last played song.

The search control 712 is used to search for audio-related information such as songs, lyrics, singers, albums, playlists, rankings, etc., so that the user can filter out the background music that needs to be played during the live broadcast.

S603: Send the corresponding first audio code stream in the music playing application to the audio processor in response to the user's operation instruction on the background sound operation control.

In this embodiment, the operation instruction refers to the user's operation on each sub-control on the floating window, specifically, as shown in Figures 7a-7d:

When the user clicks the zoom control 715 on the floating window 710 shown in FIG. 7a, the floating window 710 only displays the song icon 711, so as to reduce the visual impact of the floating window 710 on the live interface.

When the user clicks the menu control 713 on the floating window 710 shown in Fig. 7a, the floating window 710 will change to the form shown in Fig. Such as song list 1, then begin to play the song in this song list, as shown in Figure 7c, this song can be selected from the song list at random, also can be the song of preset position in this song list, as the first The song can also be the song played last time in the playlist.

At this time, if the user clicks the menu control 713 again, the play list will be collapsed, as shown in FIG. 7d.

When the user enters the search keyword/word on the search control 712 and instructs to perform the search (such as pressing Enter, or clicking the "magnifying glass button"), as shown in Figure 7e, the search will be launched below the floating window 710 In the result list, if the user clicks on any search result, the song corresponding to the search result will be switched and played.

After the floating window 710 starts to play the song, the main controller of the user terminal sends the corresponding first audio data stream in the music playing application to the audio processor.

It should also be noted that the floating window can also be designed to display the same operation interface as the music player application, so that the host does not need to change the usage habits of the music player application, making it easier to use and improving user experience.

S604. The audio processor concatenates and synthesizes the first audio data stream and the second audio code stream, so as to determine an input audio code stream of the live application.

In the live broadcast background sound processing method provided by the embodiments of the present disclosure, first, when the preset display conditions are met, the background sound operation control is displayed on the display screen in the form of a floating window, and then in response to the user's operation instruction on the background sound operation control, the The corresponding first audio code stream in the music playing application is sent to the audio processor, and finally the audio processor merges and synthesizes the first audio data stream and the second audio code stream to determine the input audio code stream of the live application. The second audio code stream includes the sound signal sent by the anchor received by the client through the audio collection device, and/or the sound signal in the live broadcast environment. The embodiments of the present disclosure solve the technical problem in the prior art that the background sound can only be added by relying on complex professional equipment and a team of professional technicians in the face of high-quality live broadcast requirements. It achieves the technical effect that the host can conveniently and quickly add background sound to the live audio stream on the live broadcast terminal, reduces the cost and technical threshold of live broadcast, and improves the user experience.

Corresponding to the method for processing live background sound in the above embodiments, FIG. 8 is a structural block diagram of an apparatus 800 for processing live background sound provided in an embodiment of the present disclosure. For ease of description, only the parts related to the embodiments of the present disclosure are shown. Referring to Figure 8, the device includes:

Main control module 801 and audio processing module 802, live application and music playing application are installed in the main control module 801;

Wherein, the main control module 801 is used to display background sound operation controls on the display screen when the preset display conditions are met;

The main control module 801 is also configured to send the corresponding first audio code stream in the music playing application to the audio processing module 802 in response to the user's operation instruction on the background sound operation control;

The audio processing module 802 is configured to concatenate and synthesize the first audio data stream and the second audio code stream to determine the input audio code stream of the live broadcast application. The second audio code stream includes the sound from the host received through the audio collection device signal, and/or, sound signal in a live environment.

In one embodiment of the present disclosure, the main control module 801 is configured to:

Obtain the opening instruction issued by the user;

According to the preset display mode indicated by the activation instruction, the background sound operation controls are simultaneously displayed on the live broadcast interface.

In an embodiment of the present disclosure, the preset display mode includes: at least one mode of split screen and floating window.

In an embodiment of the present disclosure, the content displayed by the background sound operation control includes: all or part of the content of the operation interface of the music playing application.

In one embodiment of the present disclosure, when the preset display mode is split screen, the main control module 801 is configured to divide at least one display screen of the user terminal into a live interface area and a background sound operation area;

The main control module 801 is also used to obtain the user to change the adjacent boundary of the live interface area and the background sound operation area in a preset manner, so as to change the display state of the background sound operation control. The display state includes: open or closed state, enlargement or reduction Background sound operation area.

In an embodiment of the present disclosure, the preset method includes: clicking at least one point on the boundary, clicking at least one straight line or curved segment on the boundary, and sliding the boundary along a preset direction and/or a preset path.

In one embodiment of the present disclosure, the boundary includes a straight line boundary and/or a curved boundary.

In an embodiment of the present disclosure, the operation instructions include: pause, play, previous song, next song, playlist, list or playlist loop play, sequential play, music search, lyrics search, list search at least one.

In one embodiment of the present disclosure, the audio processing module 802 is configured to use a preset synthesis algorithm to synthesize the first audio data stream and the second audio code stream into an input audio code stream;

The main control module 801 is also used to receive the input audio code stream through the live broadcast application, and encode the input audio code stream to determine the output audio code stream; send the output audio code stream from the network interface to the network server, so that The network server sends the output audio code stream to the live viewing terminal used by each audience.

The device provided in this embodiment can be used to execute any one of the above method embodiments, and its implementation principles and technical effects are similar, so this embodiment will not repeat them here.

Referring to FIG. 9 , it shows a schematic structural diagram of an electronic device suitable for implementing the embodiments of the present disclosure. The electronic device 900 may be a terminal device or a server. Among them, the terminal equipment may include but not limited to mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (Personal Digital Assistant, PDA for short), tablet computers (Portable Android Device, PAD for short), portable multimedia players (Portable Media Player, PMP for short), mobile terminals such as vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and fixed terminals such as digital TV (Television), desktop computers, etc. The electronic device shown in FIG. 9 is only an example, and should not limit the functions and application scope of the embodiments of the present disclosure.

As shown in Figure 9, an electronic device 900 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 908 loads the programs in the random access memory (Random Access Memory, RAM for short) 903 to execute various appropriate actions and processes. In the RAM 903, various programs and data necessary for the operation of the electronic device 900 are also stored. The processing device 901, ROM 902, and RAM 903 are connected to each other through a bus 904. An input/output (Input/Output, I/O for short) interface 905 is also connected to the bus 904 .

Generally, the following devices can be connected to the I/O interface 905: an input device 909 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; ), a speaker, a vibrator, etc.; a storage device 908 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 909. The communication means 909 may allow the electronic device 900 to perform wireless or wired communication with other devices to exchange data. While FIG. 9 shows electronic device 900 having various means, it is to be understood that implementing or having all of the means shown is not a requirement. More or fewer means may alternatively be implemented or provided.

In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, where the computer program includes program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network via communication means 909, or from storage means 908, or from ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are executed.

It should be noted that the above-mentioned computer-readable medium in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. A computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable Read Only Memory (Erasable Programmable Read Only Memory, referred to as EPROM or flash memory), optical fiber, portable compact disk read only memory (Compact Disc Read Only Memory, referred to as CD-ROM), optical storage device, magnetic storage device, or any of the above the right combination. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave carrying computer-readable program code therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device . The program code contained on the computer readable medium can be transmitted by any appropriate medium, including but not limited to: electric wire, optical cable, RF (Radio Frequency, radio frequency), etc., or any suitable combination of the above.

The above-mentioned computer-readable medium may be included in the above-mentioned electronic device, or may exist independently without being incorporated into the electronic device.

The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device is made to execute the methods shown in the above-mentioned embodiments.

Computer program code for carrying out the operations of the present disclosure can be written in one or more programming languages, or combinations thereof, including object-oriented programming languages—such as Java, Smalltalk, C++, and conventional Procedural Programming Language - such as "C" or a similar programming language. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or it can be connected to an external A computer (connected via the Internet, eg, using an Internet service provider).

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or portion of code that contains one or more logical functions for implementing specified executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified functions or operations , or may be implemented by a combination of dedicated hardware and computer instructions.

The units involved in the embodiments described in the present disclosure may be implemented by software or by hardware. Wherein, the name of the unit does not constitute a limitation of the unit itself under certain circumstances, for example, the first obtaining unit may also be described as "a unit for obtaining at least two Internet Protocol addresses".

The functions described herein above may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: Field Programmable Gate Array (Field Programmable Gate Array, FPGA for short), Application Specific Integrated Circuit (ASIC for short), Application Specific Standard Products ( Application Specific Standard Parts (ASSP for short), System on Chip (SOC for short), Complex Programmable Logic Device (CPLD for short), etc.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer discs, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

Embodiments of the present disclosure also provide a computer program product, including a computer program, which implements the methods in the foregoing embodiments when the computer program is executed by a processor.

An embodiment of the present disclosure also provides a live broadcast integrated machine or a live broadcast integrated device, including the electronic device corresponding to FIG. 9 . It should also be noted that the control circuit of the live broadcast all-in-one machine includes: a main control module and an audio processing module, wherein the main control module is installed with a live broadcast application and an audio playback application, and the audio processing module is used for the audio played in the audio playback application. The data code stream is synthesized into the host audio collected by the audio collection device (such as a microphone) to form the input audio of the live application.

In the first aspect, according to one or more embodiments of the present disclosure, there is provided a live broadcast background sound processing method, which is applied to a user end, and the user end includes: a main controller and an audio processor, and the main controller installs There are live broadcast applications and music playback applications, and the methods include:

According to one or more embodiments of the present disclosure, when the preset display condition is met, displaying the background sound operation control on the display screen includes:

Obtain an opening instruction issued by the user;

According to the preset display mode indicated by the opening instruction, the background sound operation control is simultaneously displayed on the live broadcast interface.

According to one or more embodiments of the present disclosure, the preset display manner includes: at least one of split screen and floating window.

According to one or more embodiments of the present disclosure, the content displayed by the background sound operation control includes: all or part of the content of the operation interface of the music playing application.

According to one or more embodiments of the present disclosure, when the preset display mode is the split screen, the simultaneously displaying the background sound operation control on the live broadcast interface includes:

Divide at least one display screen of the user terminal into a live interface area and a background sound operation area;

The operating instructions include:

The user changes the live broadcast interface area and the border adjacent to the background sound operation area in a preset manner to change the display state of the background sound operation control, and the display state includes: open or closed state, enlarged or Reduce the background sound operation area.

According to one or more embodiments of the present disclosure, the preset method includes: clicking at least one point on the boundary, clicking at least one straight line or curve segment on the boundary, sliding all stated boundary.

According to one or more embodiments of the present disclosure, the boundary includes a straight line boundary and/or a curved boundary.

According to one or more embodiments of the present disclosure, the operation instructions include: pause, play, previous song, next song, song list, list or song list loop play, sequential play, music search, lyrics search, ranking list At least one of the single searches.

According to one or more embodiments of the present disclosure, the audio processor concatenates the first audio data stream and the second audio code stream to determine the input audio code stream of the live application, including:

Synthesizing the first audio data stream and the second audio code stream into the input audio code stream by using a preset synthesis algorithm;

The live application receives the input audio code stream, and encodes the input audio code stream to determine the output audio code stream;

In the second aspect, according to one or more embodiments of the present disclosure, a live broadcast background sound processing device is provided, including:

The main control module and the audio processing module, the live application and the music playing application are installed in the main control module;

Wherein, the main control module is used to display background sound operation controls on the display screen when the preset display conditions are met;

The main control module is also used to send the corresponding first audio code stream in the music playing application to the audio processing module in response to the user's operation instruction on the background sound operation control;

The audio processing module is used to concatenate and synthesize the first audio data stream and the second audio code stream to determine the input audio code stream of the live broadcast application. The second audio code stream includes the sound signal sent by the anchor received through the audio collection device , and/or, the sound signal in the live environment.

According to one or more embodiments of the present disclosure, the main control module is configured to:

Obtain the opening instruction issued by the user;

According to one or more embodiments of the present disclosure, when the preset display mode is split screen, the main control module is configured to divide at least one display screen of the client into a live interface area and a background sound operation area;

The main control module is also used to obtain the user to change the adjacent boundary of the live interface area and the background sound operation area according to the preset method, so as to change the display state of the background sound operation control. The display state includes: open or close state, expand or reduce the background sound operation area.

According to one or more embodiments of the present disclosure, the preset method includes: clicking at least one point on the boundary, clicking at least one straight line or curved segment on the boundary, and sliding the boundary along a preset direction and/or a preset path.

According to one or more embodiments of the present disclosure, the operation instructions include: pause, play, previous song, next song, playlist, list or playlist loop play, sequential play, music search, lyrics search, list search at least one of the

According to one or more embodiments of the present disclosure, the audio processing module is configured to use a preset synthesis algorithm to synthesize the first audio data stream and the second audio code stream into an input audio code stream;

The main control module is also used to receive the input audio code stream through the live broadcast application, and encode the input audio code stream to determine the output audio code stream; send the output audio code stream from the network interface to the network server, so that the network The server sends the output audio code stream to the live viewing terminals used by each viewer.

In a third aspect, according to one or more embodiments of the present disclosure, an electronic device is provided, including:

at least one processor and memory;

the memory stores the computer program;

At least one processor executes the computer program stored in the memory, so that the at least one processor executes the live broadcast background sound processing method of the above first aspect and various possible designs of the first aspect.

In a fourth aspect, according to one or more embodiments of the present disclosure, there is provided a live streaming all-in-one machine, including: electronic devices of various possible designs in the third aspect.

In the fifth aspect, according to one or more embodiments of the present disclosure, there is provided a computer-readable storage medium, in which a computer program is stored, and when the processor executes the computer program, the above first aspect and The first aspect is various possible designs of live background sound processing methods.

In the sixth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program. When the processor executes the computer program, the live background sound processing of various possible designs of the above first aspect can be realized. method.

In the seventh aspect, according to one or more embodiments of the present disclosure, a computer program is provided. When the processor executes the computer program, various possible designs of the live background sound processing method in the above first aspect are realized.

The above description is only a preferred embodiment of the present disclosure and an illustration of the applied technical principles. Those skilled in the art should understand that the disclosure scope involved in this disclosure is not limited to the technical solution formed by the specific combination of the above-mentioned technical features, but also covers the technical solutions formed by the above-mentioned technical features or Other technical solutions formed by any combination of equivalent features. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

In addition, while operations are depicted in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or performed in sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while the above discussion contains several specific implementation details, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

A live broadcast background sound processing method, characterized in that it is applied to a user end, the user end includes: a main controller and an audio processor, the main controller is installed with a live broadcast application and a music playback application, and the method includes:

When the preset display conditions are met, the background sound operation controls are displayed on the display screen;

Responding to the user's operation instruction on the background sound operation control, sending the corresponding first audio code stream in the music playing application to the audio processor;

The audio processor concatenates and synthesizes the first audio data stream and the second audio code stream to determine the input audio code stream of the live broadcast application, and the second audio code stream includes The device receives the sound signal from the anchor, and/or the sound signal in the live broadcast environment.
The method for processing live background sound according to claim 1, wherein the displaying the background sound operation control on the display screen when the preset display condition is satisfied comprises:

Obtain an opening instruction issued by the user;

According to the preset display mode indicated by the opening instruction, the background sound operation control is simultaneously displayed on the live broadcast interface.
The method for processing live background sound according to claim 2, wherein the preset display mode includes at least one of split screen and floating window.
The method for processing live background sound according to any one of claims 1-3, wherein the content displayed by the background sound operation control includes: all or part of the content of the operation interface of the music playing application.
The live broadcast background sound processing method according to claim 3, wherein when the preset display mode is the split screen, the simultaneous display of the background sound operation controls on the live broadcast interface includes:

Divide at least one display screen of the user terminal into a live interface area and a background sound operation area;

The operating instructions include:

The user changes the live broadcast interface area and the border adjacent to the background sound operation area in a preset manner to change the display state of the background sound operation control, and the display state includes: open or closed state, enlarged or Reduce the background sound operation area.
The live broadcast background sound processing method according to claim 5, wherein the preset method includes: clicking at least one point on the boundary, clicking at least one straight line or curve segment on the boundary, pressing a preset direction and/or Or slide the boundary along a preset path.
The live broadcast background sound processing method according to claim 5 or 6, wherein the boundary includes a straight line boundary and/or a curved boundary.
The live broadcast background sound processing method according to any one of claims 1-7, wherein the operation instructions include: pause, play, previous song, next song, song list, list or song list loop playback , sequence play, music search, lyrics search, and list search at least one.
The live broadcast background sound processing method according to any one of claims 1-8, wherein the audio processor combines and synthesizes the first audio data stream and the second audio code stream to determine the The input audio code stream of the live application, including:

Synthesizing the first audio data stream and the second audio code stream into the input audio code stream by using a preset synthesis algorithm;

The live application receives the input audio code stream, and encodes the input audio code stream to determine the output audio code stream;

The output audio code stream is sent from the network interface to the network server, so that the network server sends the output audio code stream to the live viewing terminal used by each audience.
A live broadcast background sound processing device, characterized in that it includes: a main control module and an audio processing module, and a live broadcast application and a music playback application are installed in the main control module; wherein,

The main control module is used to display background sound operation controls on the display screen when preset display conditions are met;

The main control module is further configured to send the corresponding first audio code stream in the music playing application to the audio processing module in response to the user's operation instruction on the background sound operation control;

The audio processing module is configured to concatenate and synthesize the first audio data stream and the second audio code stream to determine the input audio code stream of the live application, and the second audio code stream includes Receive the sound signal from the anchor, and/or the sound signal in the live broadcast environment.
An electronic device, characterized in that it comprises:

at least one processor and memory;

The memory is used to store computer programs;

The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the live broadcast background sound processing method according to any one of claims 1-9.
An all-in-one live broadcasting machine, comprising: the electronic device according to claim 11.
A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, and when the processor executes the computer program, the live broadcast background according to any one of claims 1-9 is realized sound processing method.
A computer program product, characterized in that it includes a computer program, and is characterized in that, when the computer program is executed by a processor, the method for processing live background sound according to any one of claims 1-9 is realized.
A computer program, characterized in that, when the computer program is executed by a processor, the method for processing live background sound according to any one of claims 1-9 is implemented.