WO2022017007A1 - 音频数据处理方法、服务器及存储介质 - Google Patents
音频数据处理方法、服务器及存储介质 Download PDFInfo
- Publication number
- WO2022017007A1 WO2022017007A1 PCT/CN2021/097794 CN2021097794W WO2022017007A1 WO 2022017007 A1 WO2022017007 A1 WO 2022017007A1 CN 2021097794 W CN2021097794 W CN 2021097794W WO 2022017007 A1 WO2022017007 A1 WO 2022017007A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- audio data
- program
- audio
- detection
- module
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/16—Sound input; Sound output
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/16—Sound input; Sound output
- G06F3/162—Interface to dedicated audio devices, e.g. audio drivers, interface to CODECs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/16—Sound input; Sound output
- G06F3/165—Management of the audio stream, e.g. setting of volume, audio stream path
Definitions
- the present application relates to the field of computer technology, in particular to audio data processing.
- Cloud application refers to the application running on the server.
- the server runs the cloud application, generates corresponding audio data, and sends the audio data to the local application of the terminal for playback.
- the local application of the terminal only needs to use the audio data can be played.
- the server provides an AudioRecord (audio recording) interface for the audio capture program.
- AudioRecord audio recording
- the server will record the audio data through the recording thread, and the audio acquisition program can call the AudioRecord interface, read the recorded audio data from the recording thread, and send it to the local application of the terminal.
- an embodiment of the present application provides an audio data processing method, the method is applied to a server, and the server includes a cloud application program, a system framework, a transfer program, and an audio collection program, and the method includes: The first audio data of the application is input into the system framework; the first audio data is processed by the system framework to obtain second audio data, and the second audio data is sent to the transfer program; The transfer program, according to the communication connection between the transfer program and the audio collection program, sends the second audio data to the audio collection program, and the audio collection program is used to The data is sent to the application local to the terminal.
- an embodiment of the present application provides a delayed acquisition method, the method is applied to a server, and the server includes a detection application program, a system framework, a transfer program, and an audio collection program, and the method includes: The first detection audio data of the detection application is input into the system framework, and the sending time of the first detection audio data is recorded; the first detection audio data is processed by the system framework to obtain the second detection audio data , and send the second detection audio data to the relay program; through the relay program, according to the communication connection between the relay program and the audio collection program, the second detection audio data is sent to the relay program.
- the audio collection program records the first reception time when the audio collection program receives the second detection audio data, and the audio collection program is used to send the second detection audio data to the local application program of the terminal; A first time difference between the sending time and the first receiving time, where the first time difference represents a delay in the transmission of the detection audio data from the detection application program to the audio collection program.
- an embodiment of the present application provides a server, the server includes an application running module, a framework running module, a transfer module and a collection module; the application running module is used to input the first audio data of the cloud application program to the framework operation module; the framework operation module is used to process the first audio data, obtain second audio data, and send the second audio data to the transfer module; the transfer module, It is used to send the second audio data to the collection module according to the communication connection between the relay module and the collection module, and the collection module is used to send the second audio data to the local terminal of the terminal. application.
- an embodiment of the present application provides a server, the server includes an application running module, a framework running module, a transfer module, a collection module, a recording module and an acquisition module, the application running module is used to detect the The first detection audio data is input into the framework operation module; the recording module is used to record the sending time of the first detection audio data; the framework operation module is used to process the first detection audio data , obtain the second detection audio data, and send the second detection audio data to the relay module; the relay module is used for transferring the The second detection audio data is sent to the collection module, and the collection module is used for sending the second detection audio data to the local application of the terminal; the recording module is also used for recording the collection module receiving the first 2. Detect the first receiving time of the audio data; the obtaining module is used to obtain a first time difference between the sending time and the first receiving time, where the first time difference indicates that the detected audio data is transmitted from the application running module Delay to the acquisition module.
- an embodiment of the present application provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and the computer program is used to execute the audio data processing method described in the above aspect; or , which is used to execute the delayed acquisition method described in the above aspect.
- embodiments of the present application provide a computer program product or computer program, where the computer program product or computer program includes computer program code, and the computer program code is stored in a computer-readable storage medium.
- a processor of a computer device reads the computer program code from the computer-readable storage medium, the processor executes the computer program code, so that the computer device implements the audio data processing method as described in the above aspect; or , to implement the delayed acquisition method described in the above aspects.
- an embodiment of the present application provides a server, where the server includes:
- processors communication interfaces, memories and communication buses;
- the processor, the communication interface and the memory communicate with each other through the communication bus;
- the communication interface is an interface of a communication module;
- the memory for storing program codes and transmitting the program codes to the processor
- the processor is configured to invoke the instructions of the program code in the memory to execute the audio data processing method described in the above aspect; or, to execute the delay acquisition method described in the above aspect.
- FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application.
- FIG. 2 is an optional schematic structural diagram of the distributed system provided by the embodiment of the present application applied to the blockchain system;
- FIG. 3 is a flowchart of an audio data processing method provided by an embodiment of the present application.
- FIG. 4 is a flowchart of audio data transmission in a process of delivering audio data to a terminal by a server provided by an embodiment of the present application;
- FIG. 5 is a flowchart of an audio data processing method provided by an embodiment of the present application.
- FIG. 6 is a flowchart of a hardware abstraction layer provided by an embodiment of the present application sending audio data to an audio acquisition program
- FIG. 7 is a flow chart of audio data transmission in a process of delivering audio data to a terminal by a server provided by an embodiment of the present application;
- FIG. 8 is a flowchart of an audio data processing method provided by an embodiment of the present application.
- FIG. 9 is a flowchart of a method for obtaining a delay provided by an embodiment of the present application.
- FIG. 10 is a flowchart of a method for obtaining a delay provided by an embodiment of the present application.
- FIG. 11 is a schematic diagram of multiple audio data output by a detection application provided by an embodiment of the present application.
- FIG. 12 is a schematic diagram of a delay in acquiring audio data by multiple programs in a server provided by an embodiment of the present application
- FIG. 13 is a flowchart of a method for obtaining a delay provided by an embodiment of the present application.
- FIG. 14 is a schematic diagram of a delay in acquiring audio data by multiple programs in a server provided by an embodiment of the present application.
- 15 is a schematic diagram of a delay in acquiring audio data by multiple programs in a server provided by an embodiment of the present application.
- 16 is a schematic structural diagram of an audio data processing apparatus provided by an embodiment of the present application.
- FIG. 17 is a schematic structural diagram of another audio data processing apparatus provided by an embodiment of the present application.
- FIG. 18 is a schematic structural diagram of another audio data processing apparatus provided by an embodiment of the present application.
- 19 is a schematic structural diagram of a delay acquisition device provided by an embodiment of the present application.
- FIG. 20 is a structural block diagram of a terminal provided by an embodiment of the present application.
- FIG. 21 is a schematic structural diagram of a server provided by an embodiment of the present application.
- first audio data may be referred to as second audio data
- second audio data may be referred to as first audio data
- Cloud application an application running in a server, optionally, the cloud application is a game application or an audio processing application.
- Container The container encapsulates the relevant details necessary to run the application, such as the operating system, etc.
- a server can run multiple containers, and each container can run cloud applications and operating systems, where the operating system is any one.
- Operating system such as Android operating system, iOS (iPhone Operation System, Apple operating system), etc.
- the hardware abstraction layer between the system framework and the hardware driver, it is responsible for receiving the audio data issued by the system framework, and outputting the audio data to the hardware through the hardware driver.
- System framework a framework provided in the operating system, optionally, an audio processing framework (AudioFlinger) in the operating system.
- AudioFlinger an audio processing framework
- the resampling program (RemoteSubmix): a module in the operating system, which is used to mix the audio in the operating system and send it to the remote end through the network.
- Audio collection program a program used to collect audio data from the operating system of the server, which can send the collected audio data to the encoding module (WebrtcProxy), and the encoding module encodes the audio data and sends it to the application of the terminal
- the program optionally, when the cloud application is a cloud game program, the audio collection program is the CloudGame cloud game backend.
- Audio recording interface (AudioRecord): the interface for audio data collection in the operating system, the source of audio data is microphone, RemoteSubmix, etc.
- MixerThread The thread responsible for mixing in the system framework.
- Recording thread The thread responsible for recording in the system framework.
- FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application.
- the implementation environment includes: a terminal 101 and a server 102.
- the terminal 101 and the server 102 can be directly or indirectly connected through wired or wireless communication. This application is not limited here.
- the terminal 101 is a device such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto.
- the server 102 is an independent physical server; optionally, the server 102 is a server cluster or a distributed system composed of multiple physical servers; Cloud servers for basic cloud computing services such as functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.
- CDN Content Delivery Network
- the server 102 runs a cloud application. During the running of the cloud application, the cloud application generates audio data, and the server 102 sends the audio data to the terminal 101, so that the terminal 101 can play the audio without running the application. Application-generated audio data.
- the terminal 101 is installed with a local application program
- the user can send a control instruction to the server 102 through the local application program
- the cloud application program in the server 102 runs according to the control instruction, and generates audio data corresponding to the control instruction
- the server 102 will The audio data is delivered to the terminal 101 so that the user can play the audio data through the local application on the terminal 101 .
- the terminal and the server involved in the embodiments of the present application are connected to form a distributed system.
- the distributed system as the blockchain system as an example, refer to FIG. 2 , which is an optional schematic structural diagram of the distributed system 200 provided in the embodiment of the present application applied to the blockchain system.
- Any form of computing device (such as server, terminal) and client 202 entering the network is formed, and a peer-to-peer (P2P, Peer To Peer) network is formed between nodes.
- P2P protocol is a transmission control protocol (TCP, Transmission).
- TCP Transmission
- Application layer protocol on top of Control Protocol Application layer protocol on top of Control Protocol
- the involved functions include:
- Routing a basic function that a node has to support communication between nodes.
- a node can also have the following functions:
- each server is a node in the blockchain, and data obtained by running the cloud applications on the multiple servers is synchronized.
- the user controls the operation of the cloud game through the terminal, and the audio data processing method provided in the embodiment of the present application is used to send the audio data generated during the operation of the cloud game to the terminal, and the terminal plays the audio data, so that the user can play the audio data during the game. Listen to audio data.
- the server can send the audio data to the terminal faster, which reduces the delay of the audio data, and enables the user to listen to the audio data faster.
- the embodiments of the present application can also be applied to other scenarios where the server runs the cloud application, and the embodiments of the present application do not limit the application scenarios.
- FIG. 3 is a flowchart of an audio data processing method provided by an embodiment of the present application.
- the execution body of the embodiment of the present application is a server. Referring to FIG. 3 , the method includes the following steps.
- the cloud application is any application running in the server.
- the cloud application is a game application, or the cloud application is an audio processing application. This embodiment of the present application does not limit the type of the cloud application.
- the first audio data is audio data generated during the running of the cloud application.
- 302. Process the first audio data through the system framework to obtain second audio data, and send the second audio data to the transfer program.
- the system framework is a framework in the operating system of the server, and is used for processing audio data.
- the transfer program is a program between the system framework and the audio collection program, and is used to transmit the audio data processed by the system framework to the audio collection program.
- the transfer program has the function of forwarding audio data.
- the transfer program can also have other functions. , which is not limited in the embodiments of the present application.
- a communication connection is established between the transfer program and the audio collection program, and the transfer program can directly send the second audio data to the audio collection program through the communication connection.
- a local application program is installed on the terminal, and the local application program is an application program that supports the interaction between the terminal and the server.
- the audio collection program After receiving the second audio data, the audio collection program sends the second audio data to the local application program of the terminal, so that the The terminal plays the second audio data, where the local application is the terminal local application in step 303 .
- a relay program is set between the system framework and the audio acquisition program, and a communication connection between the relay program and the audio acquisition program is established, and the system framework can be directly processed through the communication connection.
- the resulting audio data is sent to the audio capture program.
- the above-mentioned method of directly sending the audio data through the communication connection reduces the transmission link of the audio data and shortens the time required for the audio capture program to obtain the audio data. The length of time reduces the delay for the server to deliver audio data.
- the transfer program in the above steps 302 and 303 is a hardware abstraction layer; or an original resampling program in the operating system; or other programs, which are not limited in this embodiment of the present application.
- the embodiment of the present application describes the server by taking the transfer program as the hardware abstraction layer as an example.
- the server 400 includes a cloud application program 401 , a system framework 402 , a hardware abstraction layer 403 and an audio collection program 404 .
- the cloud application 401 can call the interface of the system framework 402 to write audio data into the system framework 402 through, for example, a mixing thread, and the system framework 402 can call the interface of the hardware abstraction layer 403 to write audio data into the hardware abstraction layer 403 middle.
- a communication connection is established between the hardware abstraction layer 403 and the audio collection program 404 , and audio data can be sent to the audio collection program 404 .
- cloud application program 401 system framework 402 , hardware abstraction layer 403 and audio collection program 404 all run in the operating system container of the server 400 .
- the server 400 also includes an encoding program 405, the audio collection program 404 sends the audio data to the encoding program 405, the encoding program 405 encodes the audio data, and sends the encoded audio data to the local application of the terminal. program.
- FIG. 5 is a flowchart of an audio data processing method provided by an embodiment of the present application.
- the execution body is the server shown in FIG. 4 .
- the method includes the following steps.
- the cloud application is an application running in the server
- the local application is an application installed on the terminal
- the local application is an application that supports the interaction between the terminal and the server
- the server can store the data generated during the running of the cloud application. It is sent to the local application of the terminal, so that the terminal can display the data, so the terminal can obtain the data generated by the cloud application without running the cloud application.
- the user can also send an instruction to the server through the local application of the terminal, and the server runs the cloud application according to the instruction, and sends the data generated by the cloud application to the local application of the terminal, so that the terminal can control the cloud in the server.
- the application runs, and the terminal can also obtain the data generated after the cloud application runs, so the terminal can use the cloud application without installing and running the cloud application.
- the user triggers an operation for the virtual character A to release the skill a in the local application of the terminal, and the local application of the terminal responds to the operation and sends a skill release instruction to the cloud application in the server, and the skill release instruction carries the virtual character A.
- the virtual identifier and the skill identifier corresponding to skill a after receiving the skill release instruction, the cloud application renders the video data of the virtual character A releasing skill a according to the skill release instruction, and sends the video data to the local application of the terminal. , the video data is displayed by the local application of the terminal, so that the user can watch the picture of the virtual character A releasing the skill a.
- the operation of releasing the skill a of the virtual character A is realized by the mutual cooperation between the cloud application program in the server and the local application program of the terminal.
- the cloud application will generate audio data, and the server can send the audio data to the local application of the terminal, so that the terminal can play the audio data or store the audio data.
- the cloud application acquires the first audio data according to the virtual identifier and the skill identifier in the skill release instruction, and sends the first audio data to the local application of the terminal, where the first audio data is the virtual character A releasing the skill a The corresponding skill release sound effect.
- the local application of the terminal plays the first audio data, so that the user can hear the corresponding skill release sound effect when the virtual character A releases the skill a.
- the cloud application stores multiple types of audio data, and the multiple types of audio data include the following types.
- the background music is audio data played with the running of the cloud application, optionally, the cloud application stores a background music, and the background music is played in a loop with the running of the cloud application; optionally , the cloud application stores multiple background music, and the multiple background music is played cyclically with the running of the cloud application, or different background music is suitable for different running stages, and the cloud application selects from multiple In the background music, the background music corresponding to the running stage is selected to be played in a loop.
- the cloud application can also render video data during the running process, and according to the rendered video data, the cloud application selects background music corresponding to the video data from a plurality of background music for loop playback.
- the audio system notification is an audio notification message sent to the terminal during the running of the cloud application.
- the audio system notification is "The enemy still has XX seconds to reach the station. Field”, "Our teammate XXX is besieged”, etc., after receiving the audio system notification, the terminal will play the audio system notification.
- Operation sound effect is the audio data played along with the operation, so that the user has an immersive experience. For example, if the user operates the virtual character A to release the skill, the sound effect of releasing the skill is played, so that the user can clearly perceive that he has performed the operation of releasing the skill, thereby making the user feel immersed in the situation.
- the cloud application can select audio data corresponding to the current running state from various types of audio data according to the current running state and send it to the terminal, where the first audio data is the audio data corresponding to the current running state.
- the running state of the cloud application includes: a startup state of the cloud application, a state in which the cloud application executes an operation instruction, or a loading scene state of the cloud application, and the like.
- the cloud application selects audio data corresponding to the startup state from multiple types of audio data, and the audio data is the first audio data.
- the startup process of the cloud application refers to: the cloud application has been started, but has not been started yet. At this time, the cloud application can implement some functions, such as acquiring audio data and delivering audio data.
- the audio data corresponding to the startup state is audio data of background music.
- the cloud application is a game application.
- the startup process will take a certain amount of time. Therefore, during the startup process of the cloud application, audio data is sent to the terminal, and the terminal plays the audio. data to avoid the boredom of users waiting for the process.
- an operation instruction sent by the local application of the terminal is received, and the cloud application, in response to the operation instruction, executes the operation corresponding to the operation instruction, and selects the operation instruction from various types of audio data.
- the audio data corresponding to the operation instruction is selected, and the audio data is the first audio data.
- the cloud application is a game application.
- the cloud application receives a skill release instruction sent by the terminal.
- the skill release instruction carries the virtual character identifier and the skill identifier, and the cloud application responds to the skill release instruction.
- control the corresponding virtual character to release the corresponding skill and select the skill from various types of audio data to release the corresponding audio data.
- one or more audio sources are included in the cloud application, and the multiple types of audio data are stored in the one or more audio sources.
- each audio source stores one type of audio data, and different audio sources store different types of audio data.
- the cloud application can select the first audio data corresponding to the current operation state from multiple types of audio data according to the current operation state and send it to the terminal, including: the cloud application program reads from any audio source.
- the first audio data corresponding to the current operating state is sent to the terminal; or, the cloud application program determines the target audio source according to the current operating state, reads the first audio data corresponding to the current operating state from the target audio source, and sends to the terminal.
- the cloud application will first input the first audio data into the system framework for processing.
- the system framework is a framework in an operating system, and the operating system is an Android system or an IOS (iPhone Operation System, Apple operating system), etc., optionally, the system framework is an audio processing framework (AudioFlinger).
- the first audio data includes multiple channels of audio data, and the first audio data is mixed into one channel of audio data, and the third audio data obtained by the mixing process is one channel of audio data. data.
- the first audio data includes audio data corresponding to background music and audio data corresponding to operation sound effects, that is, the first audio data includes two channels of audio data.
- the background The audio data corresponding to the music and the audio data corresponding to the operation sound effects are mixed into one channel of audio data to obtain the third audio data, so that the third audio data heard by the subsequent user is smoother and the user's hearing effect is guaranteed.
- the first audio data includes multiple channels of audio data
- the first audio data includes audio data corresponding to background music and audio data corresponding to operation sound effects.
- the background music is the audio data that is played along with the running of the cloud application
- the operation sound effect is the audio data that is played along with the user operation. Therefore, the user may pay more attention to the audio data corresponding to the operation sound effect. Therefore, the first audio data includes multiple channels of audio data
- performing mixing processing on the first audio data to obtain third audio data includes: determining the weight of each channel of audio data in the first audio data, according to the weight of each channel of audio data. , and mix the multiple channels of audio data into one channel of audio data to obtain third audio data.
- the weight of each channel of audio data is determined according to the type of the audio data, for example, the weight of the system notification is the largest, the weight of the operation sound effect is second, and the weight of the background music is the smallest; or, the weight of the operation sound effect is the largest, the system notification The weight of the music is second, and the weight of the background music is the smallest.
- the system framework includes a processing thread
- performing sound mixing processing on the first audio data, and obtaining the third audio data includes: performing sound mixing processing on the first audio data through the processing thread to obtain the third audio data.
- the processing thread is a mixing thread.
- the embodiment of the present application customizes the hardware abstraction layer.
- the hardware abstraction layer is different from the hardware abstraction layer on the terminal.
- the hardware abstraction layer on the terminal is used to call the interface of hardware such as speakers, and input audio data into the hardware for playback.
- the hardware abstraction layer in the embodiment of the present application is not connected with the hardware, but establishes a communication connection with the audio collection program, and sends the audio data to the audio collection program.
- the audio collection program is configured with an audio parameter, and the audio parameter indicates that the audio data received by the audio collection program needs to meet the audio parameter.
- the audio parameter is 24KHz (kilohertz) dual-channel, indicating that the audio collection program is configured. To receive 24KHz dual-channel audio data.
- the hardware abstraction layer stores audio parameters, and the audio parameters are based on the audio The requirements of the acquisition program are set, so that the system framework can obtain audio parameters from the hardware abstraction layer and generate audio data that meets the audio parameters, so that the hardware abstraction layer can successfully send the audio data to the audio acquisition program.
- the audio collection program receives 24KHz (kilohertz) audio data, and the audio parameters include: a sampling rate of 24KHz.
- the audio parameter includes at least one of a target sampling rate, a target number of channels, or a target sampling depth.
- this step 503 is performed before step 502 , or this step 503 is performed simultaneously with step 502 , or this step 503 is performed after step 502 .
- this step 503 is performed only once, or the system framework needs to perform this step 503 every time the audio data is processed, which is not limited in this embodiment of the present application.
- the third audio data can be processed according to the audio parameters in the hardware abstraction layer to obtain the second audio data, so that the audio data of the second audio data can be processed.
- the parameters are consistent with the audio parameters in the hardware abstraction layer, so that the audio parameters of the second audio data meet the requirements of the audio collection program. That is, through the system framework, the third audio data is processed according to the audio parameters to obtain the second audio data, which is equivalent to adjusting the audio parameters of the audio data.
- the audio parameter includes at least one of the target sampling rate, the target channel number or the target sampling depth; through the system framework, the third audio data is processed according to the audio parameter to obtain the second audio data, including the following (1 ) to at least one of (3).
- the audio parameters include the target sampling rate, and through the system framework, the third audio data is resampled according to the target sampling rate to obtain the second audio data.
- the third audio data is resampled to obtain the second audio data with a sampling rate of 24KHz.
- the audio parameter includes the target channel number, and through the system framework, the channel number conversion processing is performed on the third audio data according to the target channel number to obtain the second audio data.
- the channel number conversion process is performed on the third audio data to obtain two-channel second audio data.
- the audio parameter includes the target sampling depth.
- the third audio data is resampled according to the target sampling depth to obtain the second audio data.
- the third audio data is re-sampled to obtain second audio data with a sampling depth of 8 bits.
- the system framework includes a processing thread, and through the system framework, processing the third audio data according to the audio parameters, and obtaining the second audio data includes: processing the third audio data according to the audio parameters through the processing thread, and obtaining the first audio data.
- Audio data In the system framework, the mixing processing of the first audio data and the processing of the third audio data according to the audio parameters are all completed by the same thread, and there is no need for multiple threads to process separately, which reduces the processing time of the audio data. In the process of transmission, the processing speed of audio data is accelerated.
- the processing thread is a mixing thread.
- the second audio data is sent to the hardware abstraction layer, and the hardware abstraction layer will send the second audio data to the audio acquisition program, but if the audio acquisition program has not been started, or the hardware abstraction layer and the audio acquisition program have not been established Communication connection, even if the second audio data is sent to the hardware abstraction layer, the hardware abstraction layer cannot send the second audio data to the audio acquisition program. Therefore, through the system framework, a communication connection is successfully established between the hardware abstraction layer and the audio acquisition program. case, the second audio data is sent to the hardware abstraction layer.
- sending the second audio data to the hardware abstraction layer through the system framework includes: if a communication connection has been established between the hardware abstraction layer and the audio acquisition program, sending the second audio data to the hardware through the system framework Abstraction layer; if the hardware abstraction layer has not established a communication connection with the audio acquisition program, control the hardware abstraction layer to establish a communication connection with the audio acquisition program.
- the second audio data is sent to the hardware abstraction layer.
- controlling the hardware abstraction layer to establish a communication connection with the audio acquisition program includes: controlling the hardware abstraction layer to send a communication connection establishment request to the audio acquisition program, and if the audio acquisition program detects the communication connection establishment request, establishing the hardware abstraction layer and the audio Communication link between acquisition programs.
- the hardware abstraction layer and the audio acquisition program fail to establish a communication connection successfully, and the system framework discards the second audio data and no longer uses the second audio data. Audio data is sent to the hardware abstraction layer.
- the audio acquisition program does not detect the communication connection establishment request sent by the hardware abstraction layer, which may be because the audio acquisition program has not been successfully started.
- the audio collection program is not only used to send the audio data generated by the cloud application to the local application of the terminal, but also used to send the video data generated by the cloud application to the local application of the terminal. If the audio collection program has not been successfully started, the audio collection program will not send the video data generated by the cloud application to the local application of the terminal, so that the terminal cannot render the screen of the cloud application according to the video data. The second audio data of the application will not affect the user.
- the hardware abstraction layer includes a writing interface
- sending the second audio data to the hardware abstraction layer includes: calling the writing interface of the hardware abstraction layer through the system framework to write the second audio data into the hardware abstraction layer.
- the system framework will periodically call the writing interface of the hardware abstraction layer, and in the writing interface, it is determined whether the hardware abstraction layer has established a communication connection with the audio acquisition program, and if a communication connection has been established, the second audio data Write to the hardware abstraction layer, if the communication connection is not established, control the hardware abstraction layer to try to establish a communication connection with the audio acquisition program, and the second audio data is written into the hardware abstraction layer if the communication connection is established successfully, if the communication connection fails, Then the second audio data is discarded.
- a communication connection is established between the hardware abstraction layer and the audio acquisition program, and the communication connection can be any form of communication connection.
- the communication connection between the hardware abstraction layer and the audio collection program is a socket (socket) connection.
- the hardware abstraction layer 601 is used as the client of the socket, and the audio collection program 602 is used as the server of the socket.
- Listen the socket's accept (receive) function call is a blocking call, and will wait until a socket client is connected.
- the audio acquisition program 602 will call the read of the socket. (read) function, the read function is configured as a blocking function, and will always wait for the hardware abstraction layer 601 to send audio data.
- sending the second audio data to the audio collection program 602 through the hardware abstraction layer 601 is equivalent to sending the second audio data locally, and the delay is at the microsecond level.
- the transmission time of the second audio data is greatly reduced, and the delay for the server to obtain the audio data is shortened.
- the communication connection between the hardware abstraction layer and the audio acquisition program is: a shared memory connection.
- the shared memory connection means: program A and program B share a memory, program A stores data in the memory, and program B can read data from the memory to realize The connection between program A and program B is realized, and the effect of program A sending data to program B is also realized.
- sending the second audio data to the hardware abstraction layer through the system framework includes: sending the second audio data to a target memory of the hardware abstraction layer through the system framework, where the target memory is the hardware abstraction layer Shared memory for the layer and the audio capture program. Therefore, sending the second audio data to the audio collection program through the hardware abstraction layer according to the communication connection between the hardware abstraction layer and the audio collection program includes: the audio collection program reads the second audio data from the target memory.
- any kind of communication connection can be established between the hardware abstraction layer and the audio acquisition program, the embodiment of the present application does not limit the communication connection mode between the two, and the embodiment of the present application only uses socket connection and sharing.
- the memory is exemplified and does not limit how the two can be connected to communicate.
- the audio collection program sends the second audio data to the encoding program, the encoding program encodes the second audio data, and then the encoding program encodes the encoded second audio data. Sent to the application local to the terminal.
- the encoding program and the terminal can establish a communication connection, and according to the communication connection, the encoded second audio data is sent to a local application program of the terminal, and the local application program of the terminal decodes and plays the data.
- the communication connection is a webrtc peer-to-peer connection.
- the embodiment of the present application only takes the cloud application program outputting the first audio data and the audio collection program acquiring the second audio data as an example, and the processing process and transmission of the audio data among multiple programs in the server are taken as an example.
- the process is exemplified.
- the cloud application can generate audio data all the time, or generate audio data multiple times, and each time the audio data is transmitted from the cloud application to the audio collection
- the process of the program is similar to the process of the above-mentioned steps 501 to 506 , which is not repeated in this embodiment of the present application.
- the cloud application continuously outputs audio data
- the cloud application periodically outputs audio data of the target size.
- the target size of the audio data depends on the size of the terminal audio data buffer.
- the target size of the audio data depends on the system framework, the hardware abstraction layer, or the buffer size in the audio collection program.
- the audio data is audio data with a playback duration of 10ms.
- a relay program is set between the system framework and the audio acquisition program, and a communication connection between the relay program and the audio acquisition program is established, and the system framework can be directly processed through the communication connection.
- the resulting audio data is sent to the audio capture program.
- the above method of directly sending audio data through a communication connection reduces the transmission link of audio data and shortens the time required for the audio acquisition program to obtain audio data. The length of time reduces the delay for the server to deliver audio data.
- the thread for mixing processing in the system framework and the thread for processing according to audio parameters are both processing threads, one thread can perform two processing, which reduces the transmission of audio data, thereby shortening the acquisition of audio data by the hardware abstraction layer. time, which further reduces the delay for the server to deliver audio data.
- the hardware abstraction layer cannot send the second audio data to the audio acquisition program.
- the system framework sends the second audio data to the hardware abstraction layer, it will determine the hardware Whether a communication connection is established between the abstraction layer and the audio acquisition program, for example, as shown in Figure 6, if the hardware abstraction layer and the audio acquisition program have not established a communication connection, the control hardware abstraction layer attempts to establish a communication connection with the audio acquisition program. In the case of success, the second audio data will be sent to the hardware abstraction layer. If the communication connection fails to be established, the second audio data will be discarded, which reduces the sending of useless data and reduces the burden on the server.
- the transfer program is a resampling program.
- the server 700 includes a cloud application program 701 , a system framework 702 , a resampling program 703 and an audio collection program 704 .
- the cloud application 701 can call the interface of the system framework 702 to write audio data into the system framework 702 , and the system framework 702 sends the obtained audio data to the resampling program 703 after processing the audio data.
- a communication connection is established between the resampling program 703 and the audio collection program 704 , and the audio data can be directly sent to the audio collection program 704 .
- cloud application program 701 system framework 702 , resampling program 703 and audio collection program 704 all run in the operating system container of the server 700 .
- the server 700 also includes an encoding program 705, the audio collection program 704 sends the audio data to the encoding program 705, the encoding program 705 encodes the audio data, and sends the encoded audio data to the local application of the terminal. program.
- FIG. 8 is a flowchart of an audio data processing method provided by an embodiment of the present application.
- the execution body is the server shown in FIG. 7 . Referring to FIG. 8 , the method includes the following steps.
- This step 801 is similar to the above-mentioned step 501 and will not be repeated here.
- This step 802 is similar to the above-mentioned step 502, and details are not repeated here.
- the resampling program is configured with an audio parameter, and the audio parameter indicates that the audio data received by the resampling program needs to satisfy the audio parameter.
- the audio parameter is 48KHz dual-channel, indicating that the resampling program is configured to receive 48KHz dual-channel audio data. Therefore, the system framework takes the audio data from the resampling program to generate audio data that meets the needs of the resampling program.
- This step 804 is similar to the above-mentioned step 504, and details are not repeated here.
- the second audio data is sent to the resampling program, and the resampling program will send the second audio data to the audio collection program, but if the audio collection program has not been started, or the resampling program and the audio collection program If the communication connection is not established, even if the second audio data is sent to the resampling program, the resampling program cannot send the second audio data to the audio collection program.
- the resampling program and the audio collection program succeed. With the communication connection established, the second audio data is sent to the resampling program.
- sending the second audio data to the resampling program through the system framework includes: if a communication connection has been established between the resampling program and the audio acquisition program, sending the second audio data to the resampling program through the system framework Sampling program; if the resampling program has not established a communication connection with the audio collection program, control the resampling program to establish a communication connection with the audio collection program.
- the second audio data is sent to the resampling program.
- controlling the resampling program to establish a communication connection with the audio collection program includes: controlling the resampling program to send a communication connection establishment request to the audio collection program, and if the audio collection program detects the communication connection establishment request, establishing the resampling program and the audio Communication link between acquisition programs.
- the system framework discards the second audio data, and will no longer use the second audio data.
- the audio data is sent to the resampling program.
- the audio collection program does not detect the communication connection establishment request sent by the resampling program, possibly because the audio collection program has not been successfully started.
- the audio collection program is not only used to send the audio data generated by the cloud application to the local application of the terminal, but also used to send the video data generated by the cloud application to the local application of the terminal. If the audio collection program has not been successfully started, the audio collection program will not send the video data generated by the cloud application to the local application of the terminal, so that the terminal cannot render the screen of the cloud application according to the video data. The second audio data of the application will not affect the user.
- the embodiment of the present application only takes as an example that the system framework sends the second audio data to the resampling program under the condition that the resampling program and the audio collection program successfully establish a communication connection.
- the transmission process is exemplified, and in another embodiment, regardless of whether the resampling program establishes a communication connection with the audio acquisition program, the system framework will send the second audio data to the resampling program.
- the resampling program includes a receiving thread
- sending the second audio data to the hardware abstraction layer through the system framework includes: sending the second audio data to the receiving thread of the resampling program through the system framework.
- the system framework processes the first audio data through a processing thread to obtain the second audio data. Therefore, in a possible implementation manner, the second audio data is sent to the resampling program through the system framework.
- the receiving thread includes: sending the second audio data to the receiving thread of the resampling program through the processing thread.
- a communication connection is established between the resampling program and the audio collection program, and the communication connection is any form of communication connection.
- the communication connection between the resampling program and the audio acquisition program is a socket connection, wherein the resampling program acts as a client of the socket, and the audio acquisition program acts as a server of the socket.
- the second audio data is sent to the audio collection program.
- the hardware abstraction layer according to the hardware abstraction layer and the audio frequency
- the socket connection between the collection programs is similar to the manner in which the second audio data is sent to the audio collection program, which will not be repeated here.
- the communication connection between the resampling program and the audio acquisition program is a shared memory connection.
- the second audio data is sent to the audio collection program.
- the hardware abstraction layer according to the hardware abstraction layer and The shared memory connection between the audio collection programs is similar to the manner in which the second audio data is sent to the audio collection program, which will not be repeated here.
- the resampling program includes a receiving thread
- the communication connection between the resampling program and the audio collection program is: the communication connection between the receiving thread and the audio collection program; or, the resampling program includes the receiving thread and the first sending A thread, wherein the receiving thread is used to receive the second audio data sent by the system framework, and the first sending thread is used to send the second audio data received by the receiving thread to the audio collection program.
- the communication connection between the resampling program and the audio collection program is: the communication connection between the first sending thread and the audio collection program.
- the audio parameters of the second audio data meet the requirements of the resampling program. If the audio parameters of the second audio data also meet the requirements of the audio collection program, the resampling program can directly convert the second audio data Sent to the audio collection program, if the audio parameters of the second audio data do not meet the requirements of the audio collection program, the resampling program needs to perform resampling processing on the second audio data, so that the processed second audio data conforms to the audio collection program. requirements of the program, and then send the processed second audio data to the audio collection program.
- the audio parameter configured by the resampling program is 48KHz dual-channel. If the audio parameter of the audio acquisition program is 48KHz dual-channel, the resampling program does not need to resample the second audio data, and directly sends the second audio data to the audio Acquisition program; if the audio parameter of the audio acquisition program is 16KHz dual-channel, the resampling program needs to perform resampling processing on the second audio data, so that the sampling rate of the processed second audio data is 16KHz.
- the resampling program Since the audio parameters configured by the resampling program are the same as the audio parameters configured by the audio acquisition program, the resampling program does not need to perform resampling processing. Therefore, the resampling program can be configured according to the audio parameters configured by the audio acquisition program to make the resampling program The audio parameters configured by the program are the same as those configured by the audio capture program.
- the system framework also includes a recording thread and a detection thread.
- the detection thread in the system framework will detect whether there are other programs currently reading the data in the recording thread. Record the data in the thread, the system framework will no longer send data to the resampling program. The original intention of the detection thread is to save unnecessary operations and reduce power consumption.
- the server also needs to perform the following steps 807 to 810. If the system framework does not include a recording thread, after acquiring the second audio data, the audio collection program sends the second audio data to the local application of the terminal.
- the second audio data is sent to the recording thread through the resampling program, and the recording thread will record the received second audio data, because the recording thread records the second audio data while receiving the second audio data, and the recording
- the process will take a certain amount of time, and it will also take a certain amount of time for the resampling program to send the second audio data to the recording thread.
- the resampling program includes a receiving thread and a second sending thread, wherein the receiving thread is used to receive the second audio data from the system framework, and when the second sending thread has an available buffer, the second audio data is sent to the in the second sending thread.
- the second sending thread After receiving the second audio data, the second sending thread determines whether to perform resampling processing on the second audio data according to the audio parameters configured in the recording thread.
- the audio parameters configured by the thread perform resampling processing on the second audio data, obtain the processed second audio data, and send the processed second audio data to the recording thread; if it is not necessary to perform resampling processing on the second audio data , the second audio data is directly sent to the recording thread.
- the existence of an available buffer in the second sending thread means that: the second sending thread sends all the audio data received by the resampling program last time to the recording thread.
- the resampling program directly sends the second audio data to the recording thread, and the recording thread can record the second audio data. If the audio parameters of the audio data are different from the audio parameters configured by the recording thread, the resampling program directly sends the second audio data to the recording thread, and the recording thread may not be able to receive the second audio data in sequence.
- the second sending thread determining whether to perform resampling processing on the second audio data according to the audio parameters configured in the recording thread includes: the second sending thread determining whether the audio parameters of the second audio data are the same as the audio parameters configured by the recording thread , if the audio parameters of the second audio data are the same as the audio parameters configured by the recording thread, then it is determined that there is no need to perform resampling processing on the second audio data; if the audio parameters of the second audio data are different from the audio parameters configured by the recording thread, then determine The second audio data needs to be resampled.
- the system framework also includes a buffer corresponding to the recording thread, and recording the second audio data through the recording thread to obtain the third audio data, including: copying the second audio data to the corresponding buffer through the recording thread to obtain the third audio data,
- the data content of the third audio data is the same as the data content of the second audio data.
- the recording thread copies the third audio data into the corresponding cache, and calls the audio recording interface to read the third audio data from the recording thread through the audio collection program, including: using the audio collection program, invoking the audio recording interface from the recording thread
- the third audio data is read from the corresponding buffer.
- the audio recording interface includes a read (reading) function, and through the audio collection program, the audio recording interface is called to read the third audio data from the recording thread, including: the audio collection program calls the read function of the audio recording interface, from recording.
- the third audio data is read in the cache corresponding to the thread. If the third audio data does not exist in the cache corresponding to the recording thread, the audio acquisition program will wait until the recording thread copies the third audio data to the cache, and then reads Pick.
- the data content of the second audio data is the same as the data content of the third audio data, but the second audio data is directly sent by the resampling program to the audio acquisition program, and the third audio data is sent by the resampling program to the recording thread, It is then read from the recording thread by the audio acquisition program. Therefore, the second audio data can reach the audio acquisition program faster than the third audio data. The second audio data is sent to the local application of the terminal, and the third audio data is discarded.
- a communication connection is established between the audio collection program and the resampling program, the second audio data is the audio data obtained according to the communication connection, and the third audio data is the audio data obtained by calling the audio recording interface through the audio collection program, so , the acquisition methods of the second audio data and the third audio data are different.
- the second audio data and the third audio data are distinguished according to the acquisition methods, and the second audio data is sent to the local application of the terminal.
- the audio collection program includes a first collection thread and a second collection thread
- the first collection thread is used to collect the second audio data
- a communication connection is established between the first collection thread and the resampling program
- the resampling program is based on the resampling program.
- the communication connection between the program and the first collection program sends the second audio data to the first collection thread; the second collection thread is used to collect the third audio data, and the second collection thread calls the audio recording interface from the recording thread.
- the third audio data is read.
- the server sends the audio data collected by the first collection thread to the local application of the terminal, and discards the audio data collected by the second collection thread.
- the resampling program in the embodiment of the present application is a program in the operating system, that is to say, the resampling program is a self-contained program of the operating system, and the present application improves the original program in the operating system.
- the above audio data processing method is implemented.
- the embodiment of the present application only takes the cloud application program outputting the first audio data and the audio collection program acquiring the second audio data as an example, and the processing process and transmission of the audio data among multiple programs in the server are taken as an example.
- the process is exemplified.
- the cloud application can generate audio data all the time, or generate audio data multiple times, and each time the audio data is transmitted from the cloud application to the audio capture
- the process of the program is similar to the process of the above-mentioned steps 801 to 810 , and details are not repeated in this embodiment of the present application.
- the cloud application continuously outputs audio data
- the cloud application periodically outputs audio data of the target size.
- the target size of the audio data depends on the size of the terminal audio data buffer.
- the target size of the audio data depends on the size of the buffer in the system framework, the resampling program, or the audio collection program.
- the audio data is audio data with a playback duration of 10ms.
- the audio data processing method provided by the embodiment of the present application improves the resampling program in the operating system, and establishes a communication connection between the resampling program and the audio acquisition program, so that the resampling program can directly Send the second audio data to the audio capture program.
- the above method of directly sending audio data through a communication connection reduces the transmission link of audio data and shortens the time required for the audio acquisition program to obtain audio data. The length of time reduces the delay for the server to deliver audio data.
- the resampling program will also send audio data to the recording thread, and the audio acquisition program reads the audio data from the recording thread to ensure that the system framework continues to send audio data to the resampling program, ensuring the continuous processing and sending of audio data.
- the audio acquisition program will send the audio data sent by the resampling program, and discard the audio data read from the recording thread, which ensures that the delay in sending the audio data is small.
- FIG. 9 is a flowchart of a delayed acquisition method provided by an embodiment of the present application.
- the execution body of the embodiment of the present application is a server. Referring to FIG. 9 , the method includes the following steps.
- the detection application is an application that runs in the server and is used to detect the delay of audio data delivered by the server.
- the detection application can output the detection audio data, and then obtain the time consumed by the transmission of the detection audio data in other programs by acquiring the time when other programs in the server receive the detection audio data, wherein the other programs in the server are detection applications program outside of the program.
- the first detection audio data is any detection audio data output by the detection application program.
- the detection application program can continuously output audio data, and in addition to outputting the detection audio data, other audio data can also be output, wherein, The detection audio data is different from other audio data, so as to distinguish the detection audio data from other audio data, so as to obtain the time when the program receives the detection audio data.
- the system framework is a framework in the operating system and is used for processing audio data.
- the transfer program is a program between the system framework and the audio collection program, and is used to transmit the audio data processed by the system framework to the audio collection program.
- the transfer program has the function of forwarding audio data.
- the transfer program also has other functions. This embodiment of the present application does not limit this.
- the second detection audio data is the audio data after the first detection audio data is processed by the system framework, but the second detection audio data and the first detection audio data are both audio data that can be distinguished from other audio data. Therefore, Even if the first detected audio data is processed to obtain the second detected audio data, the second detected audio data can be distinguished from other audio data, thereby obtaining the time when the program receives the second detected audio data.
- the audio collection program is a program in the server for collecting audio data and sending the audio data to the terminal.
- a communication connection is established between the relay program and the audio collection program, and the relay program directly transfers the second audio data through the communication connection. Send to audio capture program.
- the sending time is the time when the detection application program outputs the detected audio data
- the first receiving time is the time when the audio collection program receives the detected audio data
- the audio collection program is used in the server to collect audio data, and the audio data is collected
- the time when the audio collection program receives the detected audio data can be considered as the time when the server obtains the audio data
- the first time difference between the sending time and the first receiving time also means that the server obtains The time it takes to reach the audio data, that is, the delay for the server to deliver the audio data.
- the detection audio data is sent out by the detection application, and the reception time of the audio acquisition program receiving the detection audio data is acquired, and according to the time difference between the transmission time and the reception time, the audio data can be accurately obtained from the detection
- the time taken by the application program to transmit to the audio collection program that is, the time required to obtain the audio data from the server, which can represent the delay in sending the audio data by the server, and then determine whether the delay in obtaining the audio data from the server can be determined according to the time length. It will affect the playback effect of the audio data, and whether it will affect the hearing effect of the end user, and then determine whether to continue to improve the server, which provides a better basis for improvement for developers.
- the transfer program in the above steps 902 and 903 is a hardware abstraction layer; or an original resampling program in the operating system; or other programs, which are not limited in this embodiment of the present application.
- the transfer program is a hardware abstraction layer.
- the server 400 includes a cloud application program 401 , a system framework 402 , a hardware abstraction layer 403 and an audio collection program 404 .
- an embodiment of the present application further provides a delay acquisition method, which can be used to detect the delay of the audio data delivered by the server shown in FIG. 4 .
- FIG. 10 is a flowchart of a delayed acquisition method provided by an embodiment of the present application.
- the execution body of the embodiment of the present application is the server shown in FIG. 4 . Referring to FIG. 10 , the method includes the following steps.
- the detection application is an application that runs in the server and is used to detect the delay of audio data delivered by the server.
- the detection application can output detection data, and then obtain the time consumed by the transmission of detection data in other programs by obtaining the time when other programs in the server receive the detection data, wherein other programs in the server are outside the detection application. program of.
- the detection data output by the detection application program is the detection audio data. Since the detection audio data is audio data, the detection audio data is output to the After other programs such as the system framework, the other program can simulate the real audio data processing process, so that it will be more accurate to determine the delay by obtaining the time when other programs receive and detect the audio data.
- the detection application is different from the cloud application in the above step 501, the cloud application outputs audio data according to the received operation instruction, and the detection application outputs audio data according to the configured detection logic.
- the configured detection logic is to send the detection audio data every first time period.
- the first duration may be any duration such as 4 seconds or 5 seconds.
- the first detection audio data is any detection audio data output by the detection application program.
- the detection application program can continuously output audio data, and in addition to outputting the detection audio data, other audio data can also be output, wherein, The detection audio data is different from other audio data, so as to distinguish the detection audio data from other audio data, so as to obtain the time when the program receives the detection audio data.
- the first detected audio data is audio data carrying a tag, and it can be subsequently determined whether the first detected audio data is received according to the tag carried by the first detected audio data.
- the first detection audio data is audio data of a fixed value
- the first detection audio data is different from other audio data output by the detection application program.
- the value of the first detected audio data is 0xffff (0x represents a hexadecimal value, and ffff represents a hexadecimal value)
- the value of other audio data output by the detection application is 0, as shown in Figure 11, the The detection application program outputs audio data whose value is 0, and periodically outputs detection audio data 1101 whose value is 0xffff.
- the server further includes a recording program, which records the current time when the detection application program inputs the first detection audio data into the system framework, where the current time is the sending time of the first detection audio data.
- a recording program which records the current time when the detection application program inputs the first detection audio data into the system framework, where the current time is the sending time of the first detection audio data.
- the detection application program when it inputs the first detection audio data into the system framework, it will send a message to the recording program, and the message instructs the detection application program to input the first detection audio data into the system framework, and the recording program records the reception.
- the time of the message is used as the sending time of the first detected audio data.
- the recording program is a program other than the detection application, or a program with a recording function in the detection application.
- this recording program also has the function of detecting other programs, and this recording program can detect the data in the system frame, and when detecting that the system frame includes detection audio data, record the current time, and this current time is the first detection audio. The time when the data was sent.
- the method of processing the first detected audio data through the system framework is similar to the method of processing the first audio data through the system framework in the above step 502, and the second detected audio data is sent to the hardware through the system framework
- the manner of the abstraction layer is similar to the manner of sending the second audio data to the hardware abstraction layer through the system framework in the above step 505, and details are not repeated here.
- the obtained second detection audio data is similar to the first detection audio data, and both are audio data that can be distinguished from other audio data.
- the second detected audio data also carries the tag. If the value of the first detected audio data is 0xfff, and the value of other audio data is 0, then the value of the second detected audio data is a non-zero value, and the value of other audio data is still 0 after processing, that is, it will not Because the detected audio data is processed, the detection function of the detected audio data becomes invalid.
- the recording program is further configured to record the second reception time when the hardware abstraction layer receives the second detection audio data, and before recording the second reception time, it is determined that the hardware abstraction layer receives the second detection audio data.
- the hardware abstraction layer reports a message to the recording program, informing the recording program that the second detection audio data has been received, and when the recording program receives the reported message, records the current time, the current time.
- the time is the second receiving time when the hardware abstraction layer receives the second detection audio data.
- the recording program also has the function of detecting other programs.
- the recording program detects whether the second detection audio data is included in the code of the hardware abstraction layer, and when the second detection audio data is detected, the current time is recorded, the The current time is the second receiving time when the hardware abstraction layer receives the second detection audio data.
- the second detection audio data is sent to the audio collection program, and in the above step 506 through the hardware abstraction layer, according to the hardware abstraction layer and audio collection program
- the communication connection between the programs is similar to sending the second audio data to the audio collection program, and details are not repeated here.
- the recording program is also used to record the first receiving time when the audio collection program receives the second detection audio data. Before recording the first reception time, it is necessary to determine that the audio collection program receives the second detection audio data.
- the audio collection program reports a message to the recording program, informing the recording program that the second detection audio data has been received, and when the recording program receives the reported message, records the current time, the current time The time is the first receiving time when the audio collection program receives the second detected audio data.
- the recording program also has the function of detecting other programs, for example, whether the recording program detects the second detection audio data in the code of the audio collection program, when the second detection audio data is detected, the current time is recorded, The current time is the first receiving time when the audio collection program receives the second detected audio data.
- the embodiments of the present application only take the time duration consumed by the detection application outputting the first detection audio data and acquiring the first detection audio data transmitted between multiple programs in the server as an example, and issuing the detection server to the acquisition server.
- the delay of audio data is exemplified.
- the detection application can always output audio data, and output the detection audio data every certain period of time.
- Each of the detected audio data can obtain the delay of the audio data delivered by a server.
- statistical processing is performed on multiple delays to obtain the target delay of the audio data delivered by the server. Because the target delay considers many Therefore, the target delay is more accurate.
- the statistical process is an averaging process.
- the detection application sends the detection audio data every certain period of time, and can subsequently obtain multiple first time differences and second time differences.
- the obtained time differences can more accurately represent the detection audio data.
- the delay from the detection application program to the audio acquisition program; by performing statistical processing on multiple second time differences, the obtained time differences can more accurately represent the delay of the detection audio data transmission from the detection application program to the hardware abstraction layer.
- a larger time interval may be set, such as 4 seconds, 5 seconds, and the like.
- the sending time for the detection application 1201 to send the first detection audio data is t0
- the second receiving time for the hardware abstraction layer 1202 to receive the second detection audio data is t1.
- the delay of detecting audio data from the detecting application program 1201 to the hardware abstraction layer 1202 is about 40ms (milliseconds)
- the time for the audio collection program 1203 to receive the second detecting audio data is t2.
- the first The delay of the second detection audio data from the hardware abstraction layer 1202 to the audio acquisition program 1203 receiving the second detection audio data is about 0 ms. Therefore, the time-consuming for the server to obtain audio data from the operating system is controlled to be about 40ms, which greatly shortens the time-consuming for the server to obtain the audio data.
- the server not only sends the audio data generated by the cloud application to the terminal, but also sends the video data generated by the cloud application to the terminal.
- Delay detection it is found that the audio and video are not synchronized, and the delay from playing the video to playing the audio corresponding to the video is about 0.37 seconds.
- the delay between video and audio is higher than 0.3 seconds, the human ear can feel a relatively obvious delay, which affects the user's experience. If the audio data processing method provided by the embodiment of the present application is sampled, the delay in sending audio data from the server can be reduced, and the delay from video to audio can be reduced to about 0.242 seconds, so that the human ear cannot feel the obvious delay, and the increase is improved. user experience.
- the detection audio data is sent out by the detection application, and the reception time of the audio acquisition program receiving the detection audio data is acquired, and according to the time difference between the transmission time and the reception time, the audio data can be accurately obtained from the detection application.
- the time consumed by the transmission to the audio collection program that is, the time required to obtain the audio data from the server. This time can represent the delay in sending the audio data by the server. Based on this time, it can be determined whether the delay in obtaining the audio data by the server will affect the Whether the playback effect of the audio data will affect the hearing effect of the end user, and then determine whether to continue to improve the server, provides a better basis for improvement for developers.
- the hardware abstraction layer receives the detection audio data, and further obtain the time consumed by the detection audio data transmitted from the detection application to the hardware abstraction layer, and the detection audio data transmitted from the hardware abstraction layer to the audio acquisition program.
- the duration of consumption so as to accurately obtain the duration of audio data consumption in each transmission stage, so that subsequent developers can improve the server in a targeted manner.
- the transfer program in the server is a resampling program.
- the server 700 includes a cloud application program 701 , a system framework 702 , a resampling program 703 and an audio collection program 704 .
- an embodiment of the present application further provides a delay acquisition method, which can be used to detect the delay of the audio data delivered by the server shown in FIG. 7 .
- FIG. 13 is a flowchart of a delayed acquisition method provided by an embodiment of the present application.
- the execution body of the embodiment of the present application is the server shown in FIG. 7 . Referring to FIG. 13 , the method includes the following steps.
- the resampling program through the resampling program, according to the communication connection between the resampling program and the audio collection program, send the second detection audio data to the audio collection program, record the audio collection program to receive the first reception time of the second detection audio data, audio
- the collection program is used for sending the second detection audio data to a local application program of the terminal.
- steps 1301 to 1305 are similar to the above steps 1101 to 1105, the only difference is that the second detection audio data is sent to the hardware abstraction layer in the above steps 1101 to 1105, while the above steps 1301 to 1105 1305 is to send the second detected audio data to the resampling program.
- the sending time for the detection application 1401 to send the first detection audio data is t0
- the second receiving time for the resampling program 1402 to receive the second detection audio data is t1
- the audio The delay of the data from the detection application program 1401 to the resampling program 1402 is about 40ms (milliseconds)
- the first receiving time when the audio acquisition program 1403 receives the second detection audio data is t2
- the audio data is from the resampling program 1402 to the audio acquisition program.
- 1403 latency is about 0ms. Therefore, the time-consuming for the server to obtain audio data from the operating system is controlled to be about 40ms, which greatly shortens the time-consuming for the server to obtain the audio data.
- the resampling program is further configured to send the second detected audio data to the recording thread.
- the recording thread records the second detection audio data to obtain the third detection audio data, and the audio collection program reads the third detection audio data from the recording thread.
- the recording program is further configured to record the third receiving time when the audio collection program receives the third audio data, the server obtains the third time difference between the sending time and the third receiving time, and the third time difference represents the detected audio The latency of data transfer from the detection application, resampling and recording threads to the audio capture program.
- the audio data is obtained from the detection application program output, and the audio reaches the audio through the resampling program and the recording thread. Delay in acquisition procedure. In this way, the audio collection program only receives the third detection audio data, but cannot receive the second detection audio data, so the obtained third reception time is more accurate.
- the sending time for the detection application 1501 to send the first detection audio data is t0
- the second reception time for the resampling program 1502 to receive the second detection audio data is t1
- the audio The delay of the data from the detection application program 1501 to the resampling program 1502 is about 40ms (milliseconds)
- the third receiving time when the audio acquisition program 1503 receives the second detection audio data is t2
- the audio data is from the resampling program 1502 to the audio acquisition program.
- 1503 latency is about 90ms.
- the server not only sends the audio data generated by the cloud application to the terminal, but also sends the video data generated by the cloud application to the terminal.
- Delay detection it is found that the audio and video are not synchronized, and the delay from playing the video to playing the audio corresponding to the video is about 0.37 seconds.
- the delay between video and audio is higher than 0.3 seconds, the human ear can feel a relatively obvious delay, which affects the user's experience. If the audio data processing method provided by the embodiment of the present application is sampled, the delay in sending audio data from the server can be reduced, and the delay from video to audio can be reduced to about 0.242 seconds, so that the human ear cannot feel the obvious delay, and the increase is improved. user experience.
- the detection audio data is sent out by the detection application, the reception time of the audio acquisition program receiving the detection audio data is acquired, and according to the time difference between the transmission time and the reception time, the audio data from the detection application can be accurately obtained.
- the time consumed by the transmission to the audio collection program that is, the time required to obtain the audio data from the server. This time can represent the delay in sending the audio data by the server. Then, according to this time, it can be determined whether the delay in obtaining the audio data by the server will affect the Whether the playback effect of the audio data will affect the hearing effect of the end user, and then determine whether to continue to improve the server, provides a better basis for improvement for developers.
- the receiving time when the resampling program receives the detection audio data and further obtain the time consumed by the detection audio data transmitted from the detection application program to the resampling program, and the consumption of the detection audio data transmitted from the resampling program to the audio acquisition program.
- the duration of the audio data in each transmission stage can be accurately obtained, so that subsequent developers can improve the server in a targeted manner.
- Table 1 uses any audio data processing methods provided by the embodiments of the present application. method, the obtained video-to-audio delay, and the obtained video-to-audio delay using other audio data processing methods in the related art, as shown in Table 1:
- product A adopts the audio data processing method provided in the embodiment of the present application
- product B and product C adopt other audio data processing methods.
- product A, product B, and product C are provided with at least one type of game, and the application of the at least one type of game runs in the server, that is, the server runs at least one type of cloud application .
- FIG. 16 is a schematic structural diagram of a server provided by an embodiment of the present application.
- the server includes: an application running module 1601, a framework running module 1602, a transfer module 1603, and a collection module 1604;
- the application running module 1601 for inputting the first audio data of the cloud application to the framework running module 1602;
- This framework operation module 1602 is used to process the first audio data, obtain the second audio data, and send the second audio data to this relay module 1603;
- the relay module 1603 is configured to send the second audio data to the collection module 1604 according to the communication connection between the relay module 1603 and the collection module 1604, and the collection module 1604 is used to send the second audio data to Terminal-native applications.
- the framework operation module 1602 is configured to send the second audio data to the relay module 1603 if the relay module 1603 and the acquisition module 1604 have established a communication connection;
- the framework operation module 1602 is used to control the relay module 1603 to establish a communication connection with the acquisition module 1604 if the relay module 1603 has not established a communication connection with the acquisition module 1604, and the relay module 1603 and the acquisition module 1604 are successful When the communication connection is established, the second audio data is sent to the transfer module 1603 .
- the framework operation module 1602 is configured to perform mixing processing on the first audio data to obtain third audio data, and process the third audio data according to audio parameters to obtain the second audio data.
- the framework operation module 1602 is configured to execute at least one of the following:
- the audio parameter includes a target sampling rate, and the third audio data is resampled according to the target sampling rate to obtain the second audio data;
- the audio parameter includes the target channel number, and the third audio data is subjected to channel number conversion processing according to the target channel number to obtain the second audio data;
- the audio parameter includes a target sampling depth, and the third audio data is resampled according to the target sampling depth to obtain the second audio data.
- the framework operation module 1602 includes a processing unit 1612, and the processing unit 1612 is configured to perform mixing processing on the first audio data to obtain the third audio data;
- the processing unit 1612 is configured to process the third audio data according to the audio parameters to obtain the second audio data.
- the relay module 1603 is a hardware abstraction layer operation module 1613
- the framework operation module 1602 is used to obtain the audio parameters from the hardware abstraction layer operation module 1613
- the hardware abstraction layer operation module 1613 stores the audio parameters.
- the relay module 1603 is a hardware abstraction layer operation module 1613
- the framework operation module 1602 is used to call the writing interface of the hardware abstraction layer operation module 1613, and write the second audio data into the hardware abstraction layer operation. Module 1613.
- the relay module 1603 is a resampling module 1623
- the framework operation module 1602 is further configured to obtain the audio parameter from the resampling module 1623, which is configured with the audio parameter.
- the relay module 1603 is a resampling module 1623;
- the resampling module 1623 is further configured to perform resampling processing on the second audio data to obtain processed second audio data;
- the resampling module 1623 is configured to send the processed second audio data to the gathering module 1604 according to the communication connection between the resampling module 1623 and the gathering module 1604 .
- the framework running module 1602 includes a recording unit 1622;
- the resampling module 1623 for sending the second audio data to the recording unit 1622;
- the recording unit 1622 is used to record the second audio data to obtain the third audio data
- the acquisition module 1604 is configured to call the audio recording interface to read the third audio data from the recording unit 1622 .
- the collection module 1604 is configured to discard the third audio data and send the second audio data to a local application of the terminal.
- FIG. 19 is a schematic structural diagram of a server provided by an embodiment of the present application.
- the server includes: an application running module 1901, a framework running module 1902, a transfer module 1903, a collection module 1904, a recording module 1905, and an acquisition module 1906;
- the application running module 1901 is configured to input the first detection audio data of the detection application to the framework running module 1902;
- the recording module 1905 is used to record the sending time of the first detected audio data
- the framework operation module 1902 is used to process the first detection audio data to obtain the second detection audio data, and send the second detection audio data to the relay module 1903;
- the relay module 1903 is configured to send the second detection audio data to the acquisition module 1904 according to the communication connection between the relay module 1903 and the acquisition module 1904, and the acquisition module 1904 is used for the second detection audio data sent to the local application of the terminal;
- the recording module 1905 is also used to record the first receiving time when the acquisition module 1904 receives the second detection audio data
- the obtaining module 1906 is configured to obtain a first time difference between the sending time and the first receiving time, where the first time difference represents the delay in transmitting the detected audio data from the application running module 1901 to the collecting module 1904 .
- the recording module 1905 is also used to record the second receiving time when the relay module receives the second detection audio data
- the obtaining module 1906 is configured to obtain a second time difference between the sending time and the second receiving time, where the second time difference represents the delay in transmitting the detected audio data from the application running module 1901 to the transfer module 1903 .
- the relay module 1903 is a hardware abstraction layer running module; or, the relay module 903 is a resampling module.
- FIG. 20 is a structural block diagram of a terminal provided by an embodiment of the present application.
- the terminal 2000 is used to perform the steps performed by the terminal in the above-mentioned embodiments.
- the terminal 2000 is a portable mobile terminal, such as a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III, a moving image expert Compression standard audio layer 3), MP4 (Moving Picture Experts Group Audio Layer IV, moving picture expert compression standard audio layer 4) player, laptop or desktop computer.
- Terminal 2000 may also be called user equipment, portable terminal, laptop terminal, desktop terminal, and the like by other names.
- the terminal 2000 includes: a processor 2001 and a memory 2002 .
- the processor 2001 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like.
- the processor 2001 can use at least one hardware form among DSP (Digital Signal Processing, digital signal processing), FPGA (Field-Programmable Gate Array, field programmable gate array), PLA (Programmable Logic Array, programmable logic array) accomplish.
- the processor 2001 may also include a main processor and a coprocessor.
- the main processor is a processor used to process data in the wake-up state, also called CPU (Central Processing Unit, central processing unit); the coprocessor is A low-power processor for processing data in a standby state.
- the processor 2001 may be integrated with a GPU (Graphics Processing Unit, image processor), and the GPU is used for rendering and drawing the content that needs to be displayed on the display screen.
- the processor 2001 may further include an AI (Artificial Intelligence, artificial intelligence) processor, where the AI processor is used to process computing operations related to machine learning.
- AI Artificial Intelligence, artificial intelligence
- Memory 2002 may include one or more computer-readable storage media, which may be non-transitory. Memory 2002 may also include high-speed random access memory, as well as non-volatile memory, such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 2002 is used to store at least one program code, and the at least one program code is used to be executed by the processor 2001 to implement the methods provided by the method embodiments in this application. Audio data processing method, or delay acquisition method.
- the terminal 2000 may optionally further include: a peripheral device interface 2003 and at least one peripheral device.
- the processor 2001, the memory 2002 and the peripheral device interface 2003 may be connected through a bus or a signal line.
- Each peripheral device can be connected to the peripheral device interface 2003 through a bus, a signal line or a circuit board.
- the peripheral device includes: at least one of a radio frequency circuit 2004 , a display screen 2005 , a camera assembly 2006 , an audio circuit 2007 , a positioning assembly 2008 and a power supply 2009 .
- FIG. 20 does not constitute a limitation on the terminal 2000, and may include more or less components than the one shown, or combine some components, or adopt different component arrangements.
- the server 2100 may vary greatly due to different configurations or performance, and may include one or more processors (Central Processing Units, CPU) 2101 and one Or more than one memory 2102, wherein the memory 2102 stores at least one piece of program code, and the at least one piece of program code is loaded and executed by the processor 2101 to implement the methods provided by the above method embodiments.
- the server may also have components such as a wired or wireless network interface, a keyboard, and an input/output interface for input and output, and the server may also include other components for implementing device functions, which will not be described here.
- the server 2100 may be configured to execute the steps executed by the server in the above-mentioned audio data processing method; or, used to execute the steps executed by the server in the above-mentioned delay acquisition method.
- An embodiment of the present application further provides a computer device, the computer device includes a processor and a memory, the memory stores at least one piece of program code, and the at least one piece of program code is loaded and executed by the processor to implement the above-mentioned embodiments.
- an embodiment of the present application further provides a storage medium, where the storage medium is used to store a computer program, and the computer program is used to execute the method provided by the foregoing embodiment.
- the embodiments of the present application also provide a computer program product including instructions, which, when executed on a computer, cause the computer to execute the methods provided by the above embodiments.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Information Transfer Between Computers (AREA)
- Telephonic Communication Services (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
Description
Claims (17)
- 一种音频数据处理方法,所述方法应用于服务器,所述服务器包括云应用程序、系统框架、中转程序和音频采集程序,所述方法包括:将所述云应用程序的第一音频数据输入至所述系统框架;通过所述系统框架对所述第一音频数据进行处理,得到第二音频数据,将所述第二音频数据发送至所述中转程序;通过所述中转程序,根据所述中转程序与所述音频采集程序之间的通信连接,将所述第二音频数据发送至所述音频采集程序,所述音频采集程序用于将所述第二音频数据发送至终端本地的应用程序。
- 根据权利要求1所述的方法,所述将所述第二音频数据发送至所述中转程序,包括:若所述中转程序与所述音频采集程序已建立通信连接,则将所述第二音频数据发送至所述中转程序;若所述中转程序还未与所述音频采集程序建立通信连接,则控制所述中转程序与所述音频采集程序建立通信连接,在所述中转程序与所述音频采集程序成功建立通信连接的情况下,将所述第二音频数据发送至所述中转程序。
- 根据权利要求1所述的方法,所述通过所述系统框架对所述第一音频数据进行处理,得到第二音频数据,包括:通过所述系统框架,对所述第一音频数据进行混音处理,得到第三音频数据;通过所述系统框架,按照音频参数对所述第三音频数据进行处理,得到所述第二音频数据。
- 根据权利要求3所述的方法,所述通过所述系统框架,按照音频参数对所述第三音频数据进行处理,得到所述第二音频数据,包括以下至少一项:所述音频参数包括目标采样率,通过所述系统框架,按照所述目标采样率对所述第三音频数据进行重采样处理,得到所述第二音频数据;所述音频参数包括目标通道数,通过所述系统框架,按照所述目标通道数对所述第三音频数据进行通道数转换处理,得到所述第二音频数据;所述音频参数包括目标采样深度,通过所述系统框架,按照所述目标采样深度对所述第三音频数据进行重采样处理,得到所述第二音频数据。
- 根据权利要求3所述的方法,所述系统框架包括处理线程,所述通过所述系统框架,对所述第一音频数据进行混音处理,得到第三音频数据,包括:通过所述处理线程对所述第一音频数据进行混音处理,得到所述第三音频数据;所述通过所述系统框架,按照音频参数对所述第三音频数据进行处理,得到所述第二音频数据,包括;通过所述处理线程,按照所述音频参数对所述第三音频数据进行处理,得到所述第二音频数据。
- 根据权利要求3所述的方法,所述中转程序为硬件抽象层,所述通过所述系统框架,按照音频参数对所述第三音频数据进行处理,得到所述第二音频数据之前,所述方法还包括:通过所述系统框架从所述硬件抽象层获取所述音频参数,所述硬件抽象层存储有所述音频参数。
- 根据权利要求1所述的方法,所述中转程序为硬件抽象层,所述将所述第二音频数据发送至所述中转程序,包括:通过所述系统框架调用所述硬件抽象层的写入接口,将所述第二音频数据写入所述硬件抽象层。
- 根据权利要求1所述的方法,所述中转程序为重采样程序,所述通过所述中转程序,根据所述中转程序与所述音频采集程序之间的通信连接,将所述第二音频数据发送至所述音频采集程序,包括:通过所述重采样程序,对所述第二音频数据进行重采样处理,得到处理后的第二音频数据;通过所述重采样程序,根据所述重采样程序与所述音频采集程序之间的通信连接,将所述处理后的第二音频数据发送至所述音频采集程序。
- 根据权利要求1所述的方法,所述中转程序为重采样程序,所述系统框架包括录制线程,所述将所述第二音频数据发送至所述中转程序之后,所述方法还包括:通过所述重采样程序,将所述第二音频数据发送至所述录制线程;通过所述录制线程对所述第二音频数据进行录制,得到第三音频数据;通过所述音频采集程序,调用音频录制接口从所述录制线程中读取所述第三音频数据。
- 根据权利要求9所述的方法,所述方法还包括:通过所述音频采集程序,丢弃所述第三音频数据,将所述第二音频数据发送至所述终端本地的应用程序。
- 一种延时获取方法,所述方法应用于服务器,所述服务器包括检测应用程序、系统框架、中转程序和音频采集程序,所述方法包括:将所述检测应用程序的第一检测音频数据输入至所述系统框架,记录所述第一检测音频数据的发送时间;通过所述系统框架对所述第一检测音频数据进行处理,得到第二检测音频数据,将所述第二检测音频数据发送至所述中转程序;通过所述中转程序,根据所述中转程序与所述音频采集程序之间的通信连接,将所述第二检测音频数据发送至所述音频采集程序,记录所述音频采集程序接收所述第二检测音频数据的第一接收时间,所述音频采集程序用于将所述第二检测音频数据发送至终端本地的应用程序;获取所述发送时间和所述第一接收时间之间的第一时间差,所述第一时间差表示检测音频数据从所述检测应用程序传输至所述音频采集程序的延时。
- 根据权利要求11所述的方法,所述将所述第二检测音频数据发送至所述中转程序之后,所述方法还包括:记录所述中转程序接收所述第二检测音频数据的第二接收时间;获取所述发送时间与所述第二接收时间之间的第二时间差,所述第二时间差表示检测音频数据从所述检测应用程序传输至所述中转程序的延时。
- 一种服务器,所述服务器包括应用运行模块、框架运行模块、中转模块和采集模块;所述应用运行模块,用于将云应用程序的第一音频数据输入至所述框架运行模块;所述框架运行模块,用于对所述第一音频数据进行处理,得到第二音频数据,将所述第二音频数据发送至所述中转模块;所述中转模块,用于根据所述中转模块与所述采集模块之间的通信连接,将所述第二音频数据发送至所述采集模块,所述采集模块用于将所述第二音频数据发送至终端本地的应用程序。
- 一种服务器,所述服务器包括应用运行模块、框架运行模块、中转模块、采集模块、记录模块和获取模块,所述应用运行模块,用于将检测应用程序的第一检测音频数据输入至所述框架运行模块;所述记录模块,用于记录所述第一检测音频数据的发送时间;所述框架运行模块,用于对所述第一检测音频数据进行处理,得到第二检测音频数据,将所述第二检测音频数据发送至所述中转模块;所述中转模块,用于根据所述中转模块与所述采集模块之间的通信连接,将所述第二检测音频数据发送至所述采集模块,所述采集模块用于将所述第二检测音频数据发送至终端本地的应用程序;所述记录模块,还用于记录所述采集模块接收所述第二检测音频数据的第一接收时间;所述获取模块,用于获取所述发送时间和所述第一接收时间的第一时间差,所述第一时间差表示检测音频数据从所述应用运行模块传输至所述采集模块的延时。
- 一种计算机可读存储介质,所述计算机可读存储介质中存储有计算机程序,所述计算机程序用于执行如权利要求1至10任一项所述的音频数据处理方法;或者,用于执行如权利要求11或12所述的延时获取方法。
- 一种服务器,所述服务器包括:处理器、通信接口、存储器和通信总线;其中,所述处理器、所述通信接口和所述存储器通过所述通信总线完成相互间的通信;所述通信接口为通信模块的接口;所述存储器,用于存储程序代码,并将所述程序代码传输给所述处理器;所述处理器,用于调用存储器中程序代码的指令执行如权利要求1至10任一项所述的音频数据处理方法;或者,执行如权利要求11或12所述的延时获取方法。
- 一种包括指令的计算机程序产品,当其在计算机上运行时,使得所述计算机执行如权利要求1至10任一项所述的音频数据处理方法;或者,执行如权利要求11或12所述的延时获取方法。
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP21846422.0A EP4047471A4 (en) | 2020-07-23 | 2021-06-02 | AUDIO DATA PROCESSING METHOD, SERVER AND STORAGE MEDIA |
KR1020227017498A KR20220080198A (ko) | 2020-07-23 | 2021-06-02 | 오디오 데이터 프로세싱 방법, 서버, 및 저장 매체 |
JP2022548829A JP7476327B2 (ja) | 2020-07-23 | 2021-06-02 | オーディオデータ処理方法、遅延時間取得方法、サーバ、及びコンピュータプログラム |
US17/737,886 US20220261217A1 (en) | 2020-07-23 | 2022-05-05 | Audio data processing method, server, and storage medium |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010716978.3 | 2020-07-23 | ||
CN202010716978.3A CN111596885B (zh) | 2020-07-23 | 2020-07-23 | 音频数据处理方法、服务器及存储介质 |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/737,886 Continuation US20220261217A1 (en) | 2020-07-23 | 2022-05-05 | Audio data processing method, server, and storage medium |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2022017007A1 true WO2022017007A1 (zh) | 2022-01-27 |
Family
ID=72186622
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2021/097794 WO2022017007A1 (zh) | 2020-07-23 | 2021-06-02 | 音频数据处理方法、服务器及存储介质 |
Country Status (6)
Country | Link |
---|---|
US (1) | US20220261217A1 (zh) |
EP (1) | EP4047471A4 (zh) |
JP (1) | JP7476327B2 (zh) |
KR (1) | KR20220080198A (zh) |
CN (1) | CN111596885B (zh) |
WO (1) | WO2022017007A1 (zh) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114879930A (zh) * | 2022-07-07 | 2022-08-09 | 北京麟卓信息科技有限公司 | 一种安卓兼容环境的音频输出优化方法 |
CN116132413A (zh) * | 2023-01-13 | 2023-05-16 | 深圳市瑞云科技股份有限公司 | 一种基于WebRTC实时语音透传改进方法 |
Families Citing this family (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111596885B (zh) * | 2020-07-23 | 2020-11-17 | 腾讯科技(深圳)有限公司 | 音频数据处理方法、服务器及存储介质 |
CN112206520B (zh) * | 2020-10-21 | 2022-09-02 | 深圳市欢太科技有限公司 | 实时音频采集方法、系统、服务端、客户端及存储介质 |
CN114338621A (zh) * | 2021-11-30 | 2022-04-12 | 北京金山云网络技术有限公司 | 一种云应用程序运行方法、系统及装置 |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2012086917A2 (ko) * | 2010-12-24 | 2012-06-28 | (주)케이티 | 클라우드 컴퓨팅 환경에서 게임 서비스 제공 방법, 클라우드 컴퓨팅 서버, 및 클라우드 컴퓨팅 시스템 |
CN105491021A (zh) * | 2015-11-24 | 2016-04-13 | 华东师范大学 | 一种Android云应用服务器及Android云应用服务器系统 |
CN106961421A (zh) * | 2017-02-17 | 2017-07-18 | 浙江大学 | 一种Android系统服务端、远程桌面音频重定向方法及系统 |
CN110034828A (zh) * | 2018-01-12 | 2019-07-19 | 网宿科技股份有限公司 | 云应用的音频采集方法及服务器 |
CN110694267A (zh) * | 2019-11-14 | 2020-01-17 | 珠海金山网络游戏科技有限公司 | 一种云游戏实现方法及装置 |
CN110841278A (zh) * | 2019-11-14 | 2020-02-28 | 珠海金山网络游戏科技有限公司 | 一种云游戏实现方法及装置 |
CN111596885A (zh) * | 2020-07-23 | 2020-08-28 | 腾讯科技(深圳)有限公司 | 音频数据处理方法、服务器及存储介质 |
Family Cites Families (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP1113361A1 (en) * | 2000-01-03 | 2001-07-04 | Wimba.Com S.A. | Process of communication between an applet and a local agent using a socket communication channel |
US8446823B2 (en) * | 2009-06-23 | 2013-05-21 | Magor Communications Corporation | Method of managing the flow of time-sensitive data over packet networks |
CN103023872B (zh) * | 2012-11-16 | 2016-01-06 | 杭州顺网科技股份有限公司 | 一种云游戏服务平台 |
CN104093046B (zh) * | 2013-04-01 | 2019-02-15 | 天津米游科技有限公司 | 一种基于云游戏的视频插播系统及方法 |
CN104637488B (zh) | 2013-11-07 | 2018-12-25 | 华为终端(东莞)有限公司 | 声音处理的方法和终端设备 |
CN105280212A (zh) * | 2014-07-25 | 2016-01-27 | 中兴通讯股份有限公司 | 混音播放方法及装置 |
CN111324576B (zh) | 2018-12-14 | 2023-08-08 | 深圳市优必选科技有限公司 | 一种录音数据保存的方法、装置、存储介质及终端设备 |
CN109947387B (zh) * | 2019-03-28 | 2022-10-21 | 阿波罗智联(北京)科技有限公司 | 音频采集方法、音频播放方法、系统、设备及存储介质 |
CN111294438B (zh) * | 2020-01-22 | 2021-06-01 | 华为技术有限公司 | 实现立体声输出的方法及终端 |
-
2020
- 2020-07-23 CN CN202010716978.3A patent/CN111596885B/zh active Active
-
2021
- 2021-06-02 EP EP21846422.0A patent/EP4047471A4/en active Pending
- 2021-06-02 WO PCT/CN2021/097794 patent/WO2022017007A1/zh unknown
- 2021-06-02 JP JP2022548829A patent/JP7476327B2/ja active Active
- 2021-06-02 KR KR1020227017498A patent/KR20220080198A/ko not_active Application Discontinuation
-
2022
- 2022-05-05 US US17/737,886 patent/US20220261217A1/en active Pending
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2012086917A2 (ko) * | 2010-12-24 | 2012-06-28 | (주)케이티 | 클라우드 컴퓨팅 환경에서 게임 서비스 제공 방법, 클라우드 컴퓨팅 서버, 및 클라우드 컴퓨팅 시스템 |
CN105491021A (zh) * | 2015-11-24 | 2016-04-13 | 华东师范大学 | 一种Android云应用服务器及Android云应用服务器系统 |
CN106961421A (zh) * | 2017-02-17 | 2017-07-18 | 浙江大学 | 一种Android系统服务端、远程桌面音频重定向方法及系统 |
CN110034828A (zh) * | 2018-01-12 | 2019-07-19 | 网宿科技股份有限公司 | 云应用的音频采集方法及服务器 |
CN110694267A (zh) * | 2019-11-14 | 2020-01-17 | 珠海金山网络游戏科技有限公司 | 一种云游戏实现方法及装置 |
CN110841278A (zh) * | 2019-11-14 | 2020-02-28 | 珠海金山网络游戏科技有限公司 | 一种云游戏实现方法及装置 |
CN111596885A (zh) * | 2020-07-23 | 2020-08-28 | 腾讯科技(深圳)有限公司 | 音频数据处理方法、服务器及存储介质 |
Non-Patent Citations (2)
Title |
---|
JIANG PENGFEI: "The Design and Implementation of Multimedia Redirection Oriented for Android Mobile Desktop Cloud", MASTER THESIS, TIANJIN POLYTECHNIC UNIVERSITY, CN, no. 12, 15 January 2019 (2019-01-15), CN , XP055889780, ISSN: 1674-0246 * |
See also references of EP4047471A4 * |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114879930A (zh) * | 2022-07-07 | 2022-08-09 | 北京麟卓信息科技有限公司 | 一种安卓兼容环境的音频输出优化方法 |
CN114879930B (zh) * | 2022-07-07 | 2022-09-06 | 北京麟卓信息科技有限公司 | 一种安卓兼容环境的音频输出优化方法 |
CN116132413A (zh) * | 2023-01-13 | 2023-05-16 | 深圳市瑞云科技股份有限公司 | 一种基于WebRTC实时语音透传改进方法 |
Also Published As
Publication number | Publication date |
---|---|
KR20220080198A (ko) | 2022-06-14 |
EP4047471A4 (en) | 2023-01-25 |
JP7476327B2 (ja) | 2024-04-30 |
JP2023516905A (ja) | 2023-04-21 |
EP4047471A1 (en) | 2022-08-24 |
CN111596885A (zh) | 2020-08-28 |
CN111596885B (zh) | 2020-11-17 |
US20220261217A1 (en) | 2022-08-18 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2022017007A1 (zh) | 音频数据处理方法、服务器及存储介质 | |
WO2019174472A1 (zh) | 画质参数调节方法、装置、终端及存储介质 | |
EP3547715B1 (en) | Method and apparatus for reducing continuous-wakeup delay of bluetooth loudspeaker, and bluetooth loudspeaker | |
US20220241686A1 (en) | Information processing method, system, apparatus, device, and storage medium | |
WO2022100304A1 (zh) | 应用内容跨设备流转方法与装置、电子设备 | |
CN109194972B (zh) | 直播流获取方法、装置、计算机设备及存储介质 | |
WO2022037261A1 (zh) | 音频播放、设备管理方法及装置 | |
WO2019201340A1 (zh) | 处理器核心调度方法、装置、终端及存储介质 | |
JP7500574B2 (ja) | スタンドアロンプログラムの実行方法、装置、デバイス及びコンピュータプログラム | |
CN107707972A (zh) | 用于屏幕共享的数据处理方法、装置、系统及电子设备 | |
US20200112768A1 (en) | Network-based media device upgrading system | |
CN112565876B (zh) | 投屏方法、装置、设备、系统及存储介质 | |
WO2023165320A1 (zh) | 播放参数配置方法及装置 | |
WO2023116311A1 (zh) | 数据交互方法、装置、设备及存储介质 | |
US11190842B2 (en) | USB-based media device upgrading system | |
WO2023185589A1 (zh) | 音量控制方法及电子设备 | |
CN107395493B (zh) | 一种基于意图Intent分享消息的方法及装置 | |
US20230297324A1 (en) | Audio Control Method, System, and Electronic Device | |
WO2022252928A1 (zh) | 投屏方法、装置、无线终端、投屏设备及存储介质 | |
US11516586B2 (en) | Contextual latency configuration for isochronous audio transport | |
US20230114327A1 (en) | Method and system for generating media content | |
US12058186B2 (en) | Private audio communication in a conference call | |
WO2024093922A1 (zh) | 音频控制方法、存储介质、程序产品及电子设备 | |
WO2024045877A1 (zh) | 设备的连接方法、装置、电子设备以及存储介质 | |
WO2023169202A1 (zh) | 视频流数据获取方法、装置、电子设备和计算机可读介质 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21846422 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 20227017498 Country of ref document: KR Kind code of ref document: A |
|
ENP | Entry into the national phase |
Ref document number: 2021846422 Country of ref document: EP Effective date: 20220517 |
|
ENP | Entry into the national phase |
Ref document number: 2022548829 Country of ref document: JP Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |