WO2025213356A1 - 音频交互的处理方法、系统、服务端、客户端和电子设备 - Google Patents

音频交互的处理方法、系统、服务端、客户端和电子设备

Info

Publication number
WO2025213356A1
WO2025213356A1 PCT/CN2024/086741 CN2024086741W WO2025213356A1 WO 2025213356 A1 WO2025213356 A1 WO 2025213356A1 CN 2024086741 W CN2024086741 W CN 2024086741W WO 2025213356 A1 WO2025213356 A1 WO 2025213356A1
Authority
WO
WIPO (PCT)
Prior art keywords
audio
text
processing server
reply
stream
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/CN2024/086741
Other languages
English (en)
French (fr)
Inventor
张凡
茅志祥
丁卫兵
吴克强
尹树成
吕大千
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Zitiao Network Technology Co Ltd
Original Assignee
Beijing Zitiao Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Zitiao Network Technology Co Ltd filed Critical Beijing Zitiao Network Technology Co Ltd
Priority to PCT/CN2024/086741 priority Critical patent/WO2025213356A1/zh
Priority to CN202480002169.2A priority patent/CN119096295A/zh
Publication of WO2025213356A1 publication Critical patent/WO2025213356A1/zh
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L13/00—Speech synthesis; Text to speech systems
    • G10L13/08—Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/26—Speech to text systems
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
    • H04L51/02—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/10—Architectures or entities
    • H04L65/1063—Application servers providing network services

Definitions

  • the present disclosure relates to the field of computer technology, and in particular to a method, system, server, client, and electronic device for processing audio interaction.
  • a method for processing audio interaction is provided, which is executed by an audio processing server, including: converting the input audio sent by the client into text to be replied; sending the text to be replied to the text processing server to generate a reply text stream; receiving a request from the client for a reply audio stream corresponding to the reply text stream, wherein the request includes a message identifier corresponding to the reply text stream, and the message identifier is sent by the text processing server to the client; obtaining the reply text stream from the text processing server according to the message identifier; generating a reply audio stream according to the reply text stream; and sending the reply audio stream to the client for playback.
  • a method for processing audio interaction is provided, which is executed by a client and includes: sending input audio to an audio processing server to convert it into text to be replied, wherein the text to be replied is sent to a text processing server to generate a reply text stream; receiving a message identifier corresponding to the reply text stream sent by the text processing server; sending a request for a reply audio stream corresponding to the reply text stream to the audio processing server, wherein the request includes the message identifier; receiving the reply audio stream sent by the audio processing server; and playing the reply audio stream.
  • an audio processing server comprising: a conversion module, Configured to convert the input audio sent by the client into text to be replied; the first sending module is configured to send the text to be replied to the text processing server to generate a reply text stream; the receiving module is configured to receive the client's request for the reply audio stream corresponding to the reply text stream, wherein the request includes a message identifier corresponding to the reply text stream, and the message identifier is sent by the text processing server to the client; the obtaining module is configured to obtain the reply text stream from the text processing server according to the message identifier; the generating module is configured to generate a reply audio stream according to the reply text stream; the second sending module is configured to send the reply audio stream to the client for playback.
  • a conversion module Configured to convert the input audio sent by the client into text to be replied
  • the first sending module is configured to send the text to be replied to the text processing server to generate a reply text stream
  • the receiving module is configured to receive the client's request for the reply audio stream corresponding to the reply text
  • a client including: a first sending module, configured to send input audio to an audio processing server to convert it into text to be replied, wherein the text to be replied is sent to the text processing server to generate a reply text stream; a first receiving module, configured to receive a message identifier corresponding to the reply text stream sent by the text processing server; a second sending module, configured to send a request for a reply audio stream corresponding to the reply text stream to the audio processing server, wherein the request includes the message identifier; a second receiving module, configured to receive the reply audio stream sent by the audio processing server; and a playing module, configured to play the reply audio stream.
  • a system for processing audio interaction comprising: an audio processing server according to any embodiment of the present disclosure and a client according to any embodiment of the present disclosure.
  • an electronic device comprising: a processor; and a memory coupled to the processor, for storing instructions, which, when executed by the processor, causes the processor to execute the audio interaction processing method of any embodiment of the present disclosure.
  • a computer-readable storage medium on which a computer program is stored.
  • the program is executed by a processor, the method for processing audio interaction of any embodiment of the present disclosure is performed.
  • a computer program product comprising: instructions, which, when executed by a processor, implement the audio interaction processing method of any embodiment of the present disclosure.
  • a computer program comprising: instructions, which, when executed by a processor, implement the audio interaction processing method of any embodiment of the present disclosure.
  • FIG1 is a schematic flow chart showing a method for processing audio interaction according to some embodiments of the present disclosure
  • FIG2 is a schematic flow chart showing a method for processing audio interaction according to other embodiments of the present disclosure.
  • FIG3 is a schematic diagram showing an audio processing server performing audio data packet retransmission according to some embodiments of the present disclosure
  • FIG4 is a schematic flow chart showing a method for processing audio interaction according to yet other embodiments of the present disclosure.
  • FIG5 is a schematic diagram showing a client performing audio data packet retransmission according to some embodiments of the present disclosure
  • FIG6A is a schematic diagram showing an information interaction process in an ASR message link according to some embodiments of the present disclosure
  • FIG6B is a schematic diagram showing an information interaction process in a TTS message link according to some embodiments of the present disclosure
  • FIG6C is a schematic diagram showing the information interaction process of the audio interaction processing method according to some embodiments of the present disclosure.
  • FIG7 shows a schematic structural diagram of an audio processing server in some embodiments of the present disclosure
  • FIG8 is a schematic diagram showing the structure of a client in some embodiments of the present disclosure.
  • FIG9 is a schematic structural diagram of an electronic device according to some embodiments of the present disclosure.
  • FIG10 is a schematic diagram showing the structure of a computer system according to some embodiments of the present disclosure.
  • FIG11 shows a schematic structural diagram of an audio interaction processing system according to some embodiments of the present disclosure.
  • the term “include” and its variations are intended to be open-ended terms that include at least the following elements/features but do not exclude other elements/features, i.e., “including but not limited to.”
  • the term “comprise” and its variations are intended to be open-ended terms that include at least the following elements/features but do not exclude other elements/features, i.e., “including but not limited to.” Therefore, “include” and “include” are synonymous.
  • the term “based on” means “based, at least in part, on.”
  • references throughout this specification to "one embodiment,” “some embodiments,” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention.
  • the term “one embodiment” means “at least one embodiment”
  • the term “another embodiment” means “at least one additional embodiment”
  • the term “some embodiments” means “at least some embodiments.”
  • the appearances of the phrases “in one embodiment,” “in some embodiments,” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment, but may.
  • Users can interact with robots through text, voice, or a combination of the two.
  • user voice (audio) input can be converted into text in real time for display, and robots can respond with both text and voice, enhancing the user's interactive experience.
  • the robot needs to configure a text processing server that can process text and an audio processing server that can process audio.
  • the degree of progress directly affects the user experience.
  • the present disclosure provides a method for processing audio interaction to improve the efficiency and timeliness of text and voice responses to input audio.
  • FIG. 1 is a block diagram of some embodiments of the audio interaction processing method disclosed herein. As shown in Figure 1 , the method of this embodiment includes steps S102 to S112. The method of this embodiment can be performed by an audio processing server.
  • the audio processing server can be implemented in hardware or software, such as a server or application capable of audio processing.
  • step S102 the input audio sent by the client is converted into text to be replied.
  • the client can be implemented by hardware or software, for example, it can be implemented as a terminal or an application set in the terminal.
  • the user can input audio through the client, and the client can transcode the audio file into multiple audio segments, and then compress the multiple audio segments and use a queue for local storage, and then use streaming to send them to the audio processing server for audio recognition and conversion into text to be replied.
  • the audio processing server can be implemented based on a machine learning model.
  • a machine learning model can realize two-way conversion between speech and text.
  • the audio processing server is a server with ASR (Automatic Speech Recognition) and TTS (Text To Speech) functions.
  • step S104 the text to be replied is sent to the text processing server to generate a reply text stream.
  • the audio processing server sends the text to be replied directly to the text processing server. Compared with the method in which the text to be replied is sent to the client and then sent to the text processing server by the client, this method improves the efficiency of the text processing server in obtaining the text to be replied, thereby improving the overall processing efficiency.
  • the text processing server is responsible for generating a response text stream based on the reply text.
  • This can be implemented using machine learning models, such as a Large Language Model (LLM) or a Foundation Model.
  • LLM Large Language Model
  • the text processing server can stream out real-time response text fragments, generating a response text stream. This improves the timeliness and efficiency of response text generation, eliminating the need to wait until the entire response text is generated before providing feedback, thus enhancing the user experience.
  • step S106 a request from the client for a reply audio stream corresponding to the reply text stream is received.
  • the request includes a message identifier corresponding to the reply text stream, and the message identifier is sent by the text processing server to the client.
  • the text processing server receives the text to be replied, it can generate a corresponding message identifier and send it to the client.
  • the message identifier is used to identify the reply text stream generated for the text to be replied.
  • a request for replying the audio stream is generated, which is used to request the audio processing server to reply the audio stream.
  • step S108 a reply text stream is obtained from the text processing server according to the message identifier.
  • the audio processing server can directly obtain the reply text stream from the text processing server based on the message identifier. Compared with the text server sending the reply text stream to the client, and then the client sending the reply text stream to the audio processing server, this improves the efficiency of the audio processing server in obtaining the reply text stream, and thus improves the efficiency of the audio processing server in subsequently generating reply audio.
  • the audio processing server obtains the reply text stream from the text processing server. That is, the text processing server uses streaming transmission to transmit the generated reply text fragments to the audio processing server in real time, instead of generating the entire reply text and then sending it to the audio processing server. This improves the transmission efficiency and the subsequent processing efficiency of the audio processing server.
  • step S110 a reply audio stream is generated according to the reply text stream.
  • the audio processing server converts each reply text segment in the received reply text stream into audio in real time and streams the reply audio stream.
  • step S112 the reply audio stream is sent to the client for playback.
  • the audio processing server streams out the real-time generated reply audio segments to form a reply audio stream and transmits it to the client for real-time playback. There is no need to generate all the reply audio and then return it to the client for playback.
  • the method of the above embodiment provides an overall implementation scheme for audio interaction.
  • the audio processing server receives the audio input by the client and converts it into text to be replied, and then directly interacts with the text processing server.
  • the text to be replied is directly sent to the text processing server to generate a reply text stream.
  • the reply text stream is directly obtained from the text processing server to generate a reply audio stream, and then returned to the client for playback.
  • the direct communication between the audio processing server and the text processing server improves the transmission efficiency of the text to be replied and the reply text stream compared to the method of forwarding the text to be replied and the reply text stream through the client, so that the text processing server can obtain the text to be replied in time, and the audio processing server can obtain the reply audio stream in time, thereby improving the generation efficiency of the reply text stream and the reply audio stream.
  • the reply text and reply audio are generated and transmitted in a streaming manner throughout the process, which can provide timely feedback on the input audio, thereby improving the efficiency and timeliness of text and voice replies to the input audio.
  • the audio processing server obtains the reply audio stream from the text processing server in real time and generates a reply audio stream, it can improve the synchronization effect of text and voice replies on the client and enhance the user experience.
  • the reply text stream includes multiple text data packets
  • the audio processing server processes the text data packets according to the message tags. Identification, periodically pulls the current text data packet generated by the text processing server from the text processing server, or receives the current text data packet generated by the text processing server, wherein each of the multiple text data packets includes a message identifier.
  • the text processing server generates text data packets in a streaming manner and uses streaming transmission to form a reply text stream.
  • the entire reply text generated by the text processing server is "Hello, it's nice to chat with you.”
  • streaming By using streaming to generate text data packets, multiple data packets can be generated for this sentence, for example, "Hello,” “Nice to chat with you,” and “Chatting” are each used as the content of different data packets.
  • the streaming generation and transmission method makes the reply more timely and efficient, and the user experience better.
  • Each text data packet includes a message identifier (message_id), so that the audio processing server can accurately identify each text data packet that is a reply to the input audio.
  • the audio processing server can use the message identifier to poll to obtain each text data packet, or the text processing server can push each text data packet to the audio processing server after generating it.
  • Pulling the reply audio stream by the audio processing server can reduce the intrusion into the message link and improve the stability of the overall link.
  • the audio processing server relatively isolates the ASR and TTS links. If the reply text stream is pushed through the text processing server, it is necessary to cooperate with the audio processing server to provide active push capabilities, which has a high cost for modifying the existing link.
  • the audio processing server obtains a first preset amount of text before the reply text stream from the text processing server, generates a current audio segment corresponding to the current text data packet based on the text segment in the current text data packet, the text segment in one or more text data packets before the current text data packet, and the first preset amount of text, and generates the current audio data packet in the reply audio stream.
  • Each round of interaction involves client inputting audio, converting it into text to be replied to, generating a reply text stream, and generating a reply audio stream.
  • the audio processing server can obtain a first preset amount of text from one or more previous rounds of conversation, combine it with the text segments in the current text data packet, and the text segments before the current text data packet, to generate the current audio segment corresponding to the current text data packet.
  • the audio processing server can generate the current audio segment based on the context of multiple rounds of conversation, enabling a better understanding of semantics and context, improving the accuracy of audio generation, and improving the accuracy of reading aloud.
  • the reply text is generally stored on the text processing server
  • the text of one or more rounds of historical interactions can be obtained from the text processing server.
  • the audio processing server can store the text to be replied and the reply text in each round of historical interactions, it can be stored after obtaining the reply text stream each time, without having to repeatedly obtain it from the text processing server.
  • the acquisition delay of the reply text stream and the text of historical interactions can be reduced, the efficiency can be improved, and the generation of reply texts by the audio processing server can be improved. Efficiency of multiplexing audio streams.
  • the audio processing server determines whether to perform sentence segmentation after the text segment in the current text data packet based on the text segment in the current text data packet and the text segment in one or more text data packets before the current text data packet; in response to performing sentence segmentation after the text segment in the current text data packet, a current audio segment corresponding to the current text data packet is generated based on the text segment in the current text data packet, the text segment in the text data packet before the current text data packet, and a first preset number of texts.
  • the audio processing server makes sentence segmentation decisions based on the text segments in the current text data packet and the text segments in one or more text data packets before the current text data packet. If sentence segmentation is required, the current audio segment is generated. If sentence segmentation is not required, the server continues to receive text data packets and generates an audio segment when sentence segmentation is required.
  • the audio processing server makes sentence segmentation decisions based on the context, which improves the fluency of the generated reply audio and the fluency of the reading. In addition, compared to the method in which the text processing server sends the reply text stream to the client, and the client performs sentence segmentation and then sends it to the audio processing server for audio generation, the efficiency of audio generation is improved.
  • the text processing server generates a response text stream and sends it to the client for display.
  • the audio processing server generates a response audio stream and sends it to the client for playback. Furthermore, after the client inputs audio, the audio processing server can also generate a response text stream and send it to the client for display.
  • all content that needs to be sent to the client for display or playback is generated and transmitted in a streaming manner, reducing the latency of client display and playback and improving the timeliness and efficiency of client display and playback.
  • the audio processing server performs speech recognition on the input audio and converts it into a text stream of the text to be replied; and sends the text stream of the text to be replied to the client for display.
  • the audio processing server uses ASR technology to stream the text to be replied and output it to the client for display, which improves the timeliness of displaying the text to be replied and enhances the user experience.
  • the audio processing server obtains a second preset amount of text before the text to be replied from the text processing server, performs speech recognition based on the input audio and the second preset amount of text, and converts the input audio into a text stream of the text to be replied.
  • the audio processing server can obtain a second preset amount of text from one or more previous rounds of conversation and combine it with the input audio to generate a text stream of the reply text.
  • the audio processing server can generate the text stream of the reply text, better understanding the semantics and context, and improving the accuracy of the reply text generation.
  • the audio processing server can store the text to be replied and the reply text in each round of interaction, it does not need to be repeatedly obtained from the text processing server. Communication can reduce the latency of acquiring text from historical interactions and improve efficiency, thereby increasing the efficiency of the audio processing server in generating text to be replied.
  • the audio processing server and the text processing server perform data transmission via remote procedure calls.
  • the audio processing server and the text processing server can communicate through RPC (Google Remote Procedure Call) or gRPC to achieve the transmission of the above-mentioned reply text and reply text stream, thereby improving transmission efficiency and the efficiency of subsequent replies to the client.
  • RPC Google Remote Procedure Call
  • gRPC gRPC
  • the audio processor transmits data with the client through the audio service gateway.
  • a long connection is established between the audio service gateway and the client.
  • the client in response to the start of the audio interactive application, the client establishes a long connection with the audio service gateway corresponding to the audio processing server, and the client transmits data with the audio processor through the audio service gateway.
  • a persistent connection is established, such as a WebSocket connection.
  • This persistent connection can be reused, reducing the delay of repeated connection establishment and improving the efficiency of subsequent transmission and audio interaction.
  • network switching or disconnection may occur, which may cause the audio data packets sent by the audio processing server to the client to be lost.
  • the audio played by the client will have problems.
  • Fig. 2 is a structural diagram of another embodiment of the method for processing audio interaction of the present disclosure. As shown in Fig. 2 , the method of this embodiment includes: steps S202 to S206.
  • step S202 in response to the network interruption, the client sends a retransmission request to the audio processing server.
  • the audio server receives the retransmission request sent by the client.
  • the reply audio stream includes multiple audio data packets, each of which includes a sequence number. By setting the sequence number, the order of the audio data packets can be determined. This allows accurate identification of the order of the audio data packets and whether any are missing in the event of video data packet loss or retransmission.
  • the retransmission request includes the sequence number identifier of the last audio data packet received by the client before the network interruption.
  • step S204 the audio processing server determines whether there is an audio data packet to be retransmitted based on the sequence number identifier of the last audio data packet received by the client before the network is disconnected.
  • the data packet to be retransmitted is an audio data packet that has been sent to the client after the last audio data packet received by the client before the network is interrupted.
  • the audio processing server can determine that the client has received the audio data packet based on the retransmission request.
  • the sequence number identifier of the last audio data packet received is used. If the audio data packets after the sequence number identifier have been sent, they may be lost, that is, the audio data packets to be retransmitted.
  • step S206 the audio processing server sends the audio data packet to be retransmitted to the client if there is an audio data packet to be retransmitted.
  • the client receives a retransmitted audio data packet sent by the audio processing server, where the retransmitted data packet is a data packet that has been sent to the client by the audio processing server after the last audio data packet received by the client before the network is interrupted.
  • the text processing server sends the generated text packets of the reply text stream, Text T1, Text T2, etc., to the audio processing server via gRPC.
  • the audio processing server then streams multiple audio packets (stream blocks), S1, S2, etc., to the client via the external network. If a network interruption occurs after S1, the client sends a retransmission request to the audio processing server.
  • the retransmission request may include the sequence number of S1, and the audio processing server determines whether and which audio packets to retransmit based on this sequence number.
  • each audio data packet in the reply audio stream includes the reply text, audio data, and Markdown-filtered reply text corresponding to the audio data packet.
  • Each audio data packet can include a key field, Seq_id, to indicate a sequence number. If a network outage occurs, the client can send a retransmission request to inform the audio processing server that a network error has occurred and that it needs to retransmit all audio data packets after the Seq_id of the last audio data packet it received. The client can identify the Seq_id of each received audio data packet to avoid receiving duplicate audio data packets.
  • the client in the event of a network interruption, the client only needs to notify the audio processing server of the serial number identifier to pull the subsequent audio for playback. If the client receives the reply text stream from the text processing server, breaks it into sentences, and then sends it to the audio processing server to generate a reply audio stream, after a network interruption occurs and waits for the network to recover, the client will continue to send the reply text data packet to the audio processing server. However, the audio processing server will not be able to distinguish whether the received data packet is used to continue synthesizing the audio.
  • the audio processing server and the text processing server communicate through the intranet, the network is stable, and the recovery of network errors is easier to handle, and the cache of reply text and audio is increased, making retransmission and resumption simpler and more efficient.
  • the method of the above embodiment reduces the failure of reply audio playback caused by the interruption of the reply audio stream, thereby improving the user experience.
  • FIG. 4 Other embodiments of the audio interaction processing method of the present disclosure are described below in conjunction with FIG. 4 .
  • the method of this embodiment can be executed by a client.
  • FIG4 is a structural diagram of some embodiments of the audio interaction processing method disclosed in the present invention.
  • the processing method of this embodiment includes steps S402 to S410.
  • the method of this embodiment can be executed by a client.
  • step S402 the input audio is sent to the audio processing server and converted into text to be replied.
  • the text to be replied is sent by the audio processing server to the text processing server to generate a reply text stream.
  • the text to be replied is sent by the audio processing server to the text processing server to generate a reply text stream.
  • step S404 a message identifier corresponding to the reply text stream sent by the text processing server is received.
  • the text processing server may generate a message identifier and send it to the client.
  • the text processing server may generate a message identifier and send it to the client.
  • step S406 a request for a reply audio stream corresponding to the reply text stream is sent to the audio processing server.
  • the request sent by the client includes a message identifier.
  • step S408 a reply audio stream sent by the audio processing server is received.
  • the reply audio stream is generated by the audio processing server after it obtains the reply text stream from the text processing server based on the message identifier.
  • the process of the audio processing server obtaining the reply audio stream from the text processing server can be referred to in the signing implementation and will not be repeated here.
  • step S410 the reply audio stream is played.
  • the client sends the input audio to the audio processing server to be converted into the text to be replied, and then directly interacts with the text processing server.
  • the text to be replied is sent directly to the text processing server to generate a reply text stream.
  • the reply text stream is directly obtained from the text processing server to generate a reply audio stream, and then returned to the client for playback.
  • the audio processing server communicates directly with the text processing server.
  • the transmission efficiency of the text to be replied and the reply text stream is improved, so that the text processing server can obtain the text to be replied in time, and the audio processing server can obtain the reply audio stream in time, thereby improving the generation efficiency of the reply text stream and the reply audio stream.
  • the reply text and reply audio are generated and transmitted in a streaming manner throughout the process, which can provide timely feedback on the input audio, thereby improving the efficiency and timeliness of text and voice replies to the input audio.
  • the audio processing server obtains the reply audio stream from the text processing server in real time and generates a reply audio stream, it can improve the synchronization effect of text and voice replies on the client and enhance the user experience.
  • the client divides the input audio into multiple audio segments; compresses each of the multiple audio segments and stores them in a first queue in order; takes out the first audio segment from the first queue and sends it to the audio processing server for speech recognition; in response to receiving a confirmation message sent by the audio processing server, repeatedly takes out the first audio segment from the first queue and sends it to the audio processing server for speech recognition. Identify until the first queue is empty.
  • the client stores multiple compressed audio clips in the first queue and sends them sequentially to the audio processing server for speech recognition and conversion into text for reply.
  • the audio processing server's reply confirmation ensures that each audio clip has been sent to the audio processing server, reducing the possibility of audio clips being lost due to network interruptions and other situations, and improving the accuracy of voice interaction.
  • the client in response to not receiving a confirmation message from the audio processing server after waiting for a preset period of time, the client resends a most recently sent audio segment.
  • the client does not receive a confirmation message from the audio processing server after waiting for a preset period of time, it may be that the corresponding audio segment is lost. Resending the corresponding audio segment reduces the probability that the audio processing server cannot accurately recognize the input audio and improves the accuracy of subsequent voice interactions.
  • the client sends each audio data packet of the input audio to the audio processing server. If the network is interrupted after the client sends the S1 data packet, and the client waits for a preset time without receiving the confirmation message sent by the audio processing server, it will resend the S1 data packet.
  • the client transcodes the input audio file and divides it into multiple audio segments, which are stored locally in the first queue.
  • the client then streams the audio to the audio processing server for audio recognition.
  • the client Upon receiving confirmation from the audio processing server, the client removes the subsequent audio segments from the queue for subsequent transmission.
  • the client configures the system to determine whether to resume transmission and wait for a preset time period before retrying. The client then retrys the most recently sent audio segment at the head of the queue. This ensures that streaming recognition of the input audio can operate normally and obtain correct recognition results even in the event of network switching or poor network quality.
  • the reply audio stream includes multiple audio data packets, each of the multiple audio data packets includes a serial number identifier of the audio data packet, and the client stores the current audio data packet in the second queue according to the serial number identifier of the current audio data packet received; and plays the audio data packets in the order of the audio data packets in the second queue.
  • Adding sequence numbers to audio data packets allows the client to play each audio data packet in sequence. Even if a network transmission error occurs, the reply audio stream can be accurately played.
  • the client determines whether an audio data packet with the same serial number identifier as the current audio data packet already exists in the second queue based on the serial number identifier of the current audio data packet received; if an audio data packet with the same serial number identifier as the current audio data packet already exists in the second queue, the current audio data packet is discarded; if an audio data packet with the same serial number identifier as the current audio data packet does not exist in the second queue, the current audio data packet is stored in the queue.
  • the client can determine whether there are duplicate audio data packets based on the sequence number of the audio data packet, especially In the case of network interruption, retransmission, and resumption, there may be repeated transmission of audio data packets, which improves the accuracy of reply audio stream transmission and the accuracy of reply audio stream playback.
  • the client receives a text stream of text to be replied sent by the audio processing server; displays the text stream of text to be replied; receives a reply text stream sent by the text processing server; and displays the reply text stream.
  • the reply text stream is generated by the text processing server after receiving the reply text sent by the audio processing server.
  • the client receives the reply text stream sent by the audio processing server, the reply text stream sent by the text processing server, and the reply audio stream sent by the audio processing server for display and playback.
  • the method of the embodiment of the present disclosure can obtain the reply text stream in real time for the input audio, quickly convert it into a reply audio stream, and push it to the client in real time for playback.
  • the reply text to audio playback experience is greatly improved, and the generation of the reply audio stream in combination with the context can improve the accuracy.
  • By retrying and retransmitting the reply audio stream and the input audio stream it can be ensured that even in the event of network quality degradation, network disconnection, jitter, etc. during the ASR recognition process of long texts or users for a long time, ASR and TTS can still be accurately recognized or played normally.
  • FIG. 6A shows the information interaction process in the ASR message link.
  • the audio module (APP Audio) in the audio interaction application of the client establishes a long connection with the audio service gateway (GateWay) when the audio interaction application is turned on.
  • the user sends a voice, starts ASR message transmission, and inputs it into the audio module in the audio interaction application of the client.
  • the user voice is sent to the audio service gateway through the audio module.
  • the audio service gateway belongs to the service side device.
  • the audio service gateway sends the user voice to the audio processing module.
  • the audio processing module and the ASR module can form the audio processing server in the aforementioned embodiment.
  • the ASR module can also be deployed separately, and the audio processing module serves as the audio processing server in the aforementioned embodiment.
  • the audio processing module calls the ASR module through gRPC to realize user voice recognition and obtain the voice recognition text (text to be replied). (3)
  • the audio processing module returns the voice recognition text to the audio service gateway.
  • the audio service gateway returns the voice device text to the audio module in the client.
  • the audio module in the client sends the voice recognition text to the message module (APP message) in the client for display.
  • APP message message
  • the interactive processing module sends a text message to the LLM, requesting the LLM to process the text message.
  • the interactive processing module and the LLM can form the text processing server in the aforementioned embodiment.
  • the LLM can also be deployed separately, and the interactive processing module can serve as the text processing server in the aforementioned embodiment.
  • the interactive processing module sends the full text of the user message (speech recognition text) to the LLM, The LLM returns the polling key to the interaction processing module.
  • the interaction processing module polls the LLM for incremental replies based on the key.
  • the LLM generates replies in a streaming manner, and the interaction processing module only needs to obtain the incremental reply text each time.
  • the interaction processing module sends the reply text to the Frotier (edge) device.
  • the Frotier device sends the reply text to the message module in the client for display.
  • the audio service gateway and Frotier device can serve as the WebSocket access layer to implement transmission between the external network and the internal network.
  • Figure 6B shows the information interaction process in the TTS message link.
  • the audio processing module sends the speech recognition text (text to be replied) to the interaction processing module, and (2) the interaction processing module sends the speech recognition text to the LLM.
  • the interaction processing module sends the Message ID (message identifier) of the reply text stream to the Frotier device.
  • the Frotier device sends the Message ID of the reply text stream to the message module in the client.
  • the message module sends the Message ID of the reply text stream to the audio module in the client to start the TTS session.
  • the audio module sends a request to subscribe to the TTS audio stream (reply audio stream) to the audio service gateway.
  • the audio service gateway sends a request to subscribe to the TTS audio stream (reply audio stream) to the audio processing module.
  • the audio processing module requests the reply text from the interaction processing module.
  • the audio processing module generates a TTS audio stream (reply audio stream) based on the reply text and sends it to the audio service gateway.
  • the audio service gateway sends the TTS audio stream to the audio module in the client for playback.
  • FIG. 6C shows the information interaction flow of the audio interaction processing method.
  • an audio SDK can be added to the client to interact with the audio processing server.
  • the audio SDK can be located in the client or set up independently.
  • the client sends input audio (i.e., audio uplink) to the audio SDK.
  • the audio SDK sends input audio to the audio processing server.
  • the audio processing server requests context content (i.e., the text of historical interactions before this round of interaction) from the text processing server.
  • the text processing server returns the context content to the audio processing server.
  • the audio processing server performs ASR recognition and obtains the ASR recognized text (text to be replied).
  • the audio processing server combines the context content when performing ASR recognition, which is more accurate. (6)
  • the audio processing server sends the ASR recognized text to the audio SDK.
  • the audio SDK sends the ASR recognized text to the client.
  • the client displays the ASR recognized text.
  • the audio processing server sends the ASR-recognized text (the text to be replied) to the text processing server.
  • the text processing server sends the message ID to the client.
  • the text processing server generates the reply text and returns it to the client in a streaming format.
  • the client displays the reply text.
  • the client sends a request for TTS audio data (reply audio stream) to the audio SDK, which carries Message ID.
  • the audio SDK sends a request for TTS audio data to the audio processing server.
  • the audio processing server requests a reply text from the text processing server.
  • the text processing server sends a reply text stream to the audio processing server.
  • the audio processing server generates a TTS audio stream based on the reply text stream.
  • the audio processing server sends the TTS audio stream to the audio SDK.
  • the audio SDK sends the TTS audio stream to the client. (20)
  • the client plays the TTS audio.
  • steps (1) to (8) are the process of generating and displaying the text to be replied
  • steps (9) to (12) are the process of generating and displaying the reply text
  • (13) to (20) are the process of generating and playing the reply audio stream.
  • the steps in the same process are described together. In actual processing, the order of these steps is not limited to the order and numbering shown in the figure.
  • the audio processing server After the audio processing server generates the ASR recognized text, it can be sent to the text processing service segment, that is, (6) and (9) can be synchronized or (9) can be after (6).
  • the present disclosure also provides an audio processing server, which is described below in conjunction with FIG. 7 .
  • Figure 7 is a structural diagram of some embodiments of the audio processing server of the present disclosure.
  • the audio processing server 70 of this embodiment includes: a conversion module 710, a first sending module 720, a receiving module 730, an acquisition module 740, a generation module 750, and a second sending module 760.
  • the conversion module 710 is configured to convert the input audio sent by the client into a text to be replied.
  • the first sending module 720 is configured to send the text to be replied to to the text processing server to generate a reply text stream.
  • the receiving module 730 is configured to receive a request from the client for a reply audio stream corresponding to a reply text stream, wherein the request includes a message identifier corresponding to the reply text stream, and the message identifier is sent from the text processing server to the client.
  • the acquisition module 740 is configured to acquire the reply text stream from the text processing server according to the message identifier.
  • the generating module 750 is configured to generate a reply audio stream according to the reply text stream.
  • the second sending module 760 is configured to send the reply audio stream to the client for playback.
  • the reply audio stream includes multiple audio data packets, each of the multiple audio data packets includes a serial number identifier of the audio data packet
  • the audio processing server 70 also includes a determination module 770
  • the receiving module 730 is also configured to receive a retransmission request sent by the client, wherein the retransmission request includes the serial number identifier of the last audio data packet received by the client before the network interruption
  • the determination module 770 is configured to determine whether there is an audio data packet to be retransmitted based on the serial number identifier of the last audio data packet, wherein the data packet to be retransmitted is an audio data packet sent to the client after the last audio data packet
  • the second sending module 760 is configured to send the audio data packet to be retransmitted to the client if there is an audio data packet to be retransmitted.
  • the reply text stream includes multiple text data packets
  • the acquisition module 740 is configured to Message identifier, periodically pulls the current text data packet generated by the text processing server from the text processing server; or receives the current text data packet generated and pushed by the text processing server, wherein each of the multiple text data packets includes a message identifier.
  • the acquisition module 740 is further configured to acquire a first preset amount of text before the reply text stream from the text processing server, and the generation module 750 is configured to generate a current audio segment corresponding to the current text data packet based on the text segment in the current text data packet, the text segment in one or more text data packets before the current text data packet, and the first preset amount of text, and generate the current audio data packet in the reply audio stream.
  • the generation module 750 is configured to determine whether to perform sentence segmentation after the text segment in the current text data packet based on the text segment in the current text data packet and the text segment in one or more text data packets before the current text data packet; in response to performing sentence segmentation after the text segment in the current text data packet, generate a current audio segment corresponding to the current text data packet based on the text segment in the current text data packet, the text segment in the text data packet before the current text data packet, and a first preset number of texts.
  • the conversion module 710 is configured to perform speech recognition on the input audio and convert it into a text stream of the text to be replied; the second sending module 760 is further configured to send the text stream of the text to be replied to the client for display.
  • the acquisition module 740 is further configured to obtain a second preset amount of text before the text to be replied from the text processing server, and the conversion module 710 is configured to perform speech recognition based on the input audio and the second preset amount of text, and convert the input audio into a text stream of the text to be replied.
  • the audio processing server and the text processing server perform data transmission via remote procedure calls.
  • the audio processor transmits data with the client through the audio service gateway, and in response to the start of the audio interaction application, a long connection is established between the audio service gateway and the client.
  • the present disclosure also provides a client, which is described below in conjunction with FIG8 .
  • FIG8 is a structural diagram of some embodiments of the client of the present disclosure.
  • the client 80 of this embodiment includes: a first sending module 810 , a first receiving module 820 , a second sending module 830 , a second receiving module 840 , and a playing module 850 .
  • the first sending module 810 is configured to send the input audio to the audio processing server to convert it into a text to be replied, wherein the text to be replied is sent to the text processing server to generate a reply text stream.
  • the first receiving module 820 is configured to receive the message tag corresponding to the reply text stream sent by the text processing server. knowledge.
  • the second sending module 830 is configured to send a request for a reply audio stream corresponding to the reply text stream to the audio processing server, wherein the request includes a message identifier.
  • the second receiving module 840 is configured to receive the reply audio stream sent by the audio processing server.
  • the playing module 850 is configured to play the reply audio stream.
  • the first sending module 810 is configured to divide the input audio into multiple audio segments; compress each of the multiple audio segments and store them in sequence in a first queue; take out the first audio segment from the first queue and send it to the audio processing server for speech recognition; in response to receiving a confirmation message sent by the audio processing server, repeatedly take out the first audio segment from the first queue and send it to the audio processing server for speech recognition until the first queue is empty.
  • the first sending module 810 is configured to resend a most recently sent audio segment in response to not receiving a confirmation message sent by the audio processing server after waiting for a preset period of time.
  • the reply audio stream includes multiple audio data packets, each of the multiple audio data packets includes a serial number identifier of the audio data packet, and the playback module 850 is configured to store the current audio data packet into the second queue according to the serial number identifier of the received current audio data packet; and play it in the order of the audio data packets in the second queue.
  • the playback module 850 is configured to determine whether an audio data packet with the same serial number identifier as the current audio data packet already exists in the second queue based on the serial number identifier of the received current audio data packet; if an audio data packet with the same serial number identifier as the current audio data packet already exists in the second queue, the current audio data packet is discarded; if an audio data packet with the same serial number identifier as the current audio data packet does not exist in the second queue, the current audio data packet is stored in the queue.
  • the second sending module 830 is further configured to send a retransmission request to the audio processing server in response to a network interruption, wherein the retransmission request includes a sequence number identifier of the last audio data packet received by the client before the network interruption; the second receiving module 840 is further configured to receive the retransmitted audio data packet sent by the audio processing server, wherein the retransmitted data packet is a data packet that has been sent by the audio processing server to the client after the last audio data packet.
  • the client 80 further includes a third receiving module 860 configured to receive a text stream of text to be replied sent by the audio processing server; and a display module 870 configured to display the text stream of text to be replied.
  • the client 80 further includes a fourth receiving module 880 configured to receive the text processing service
  • the reply text stream is sent by the server, wherein the reply text stream is generated after the text processing server receives the text to be replied sent by the audio processing server; the display module 870 is also configured to display the reply text stream.
  • the client 80 in response to the start of the audio interaction application, the client 80 establishes a long connection with the audio service gateway corresponding to the audio processing server, wherein the client transmits data with the audio processor through the audio service gateway.
  • the above-mentioned units are merely logical modules divided according to the specific functions they implement, and are not intended to limit specific implementation methods. For example, they can be implemented in software, hardware, or a combination of software and hardware. In actual implementation, the above-mentioned units can be implemented as independent physical entities, or can also be implemented by a single entity (for example, a processor (CPU or DSP, etc.), an integrated circuit, etc.). In addition, the above-mentioned units are shown with dotted lines in the drawings to indicate that these units may not actually exist, and the operations/functions they implement can be implemented by the processing circuit itself.
  • the device may also include a memory that can store various information generated by the device and the various units contained in the device during operation, programs and data used for operation, data to be sent by the communication unit, etc.
  • the memory can be volatile memory and/or non-volatile memory.
  • the memory can include but is not limited to random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), and flash memory.
  • RAM random access memory
  • DRAM dynamic random access memory
  • SRAM static random access memory
  • ROM read-only memory
  • flash memory flash memory
  • the device may also include a communication unit that can be used to communicate with other devices.
  • the communication unit can be implemented in an appropriate manner known in the art, for example, including communication components such as an antenna array and/or a radio frequency link, various types of interfaces, communication units, etc. This will not be described in detail here.
  • the device may also include other components not shown, such as a radio frequency link, a baseband processing unit, a network interface, a processor, a controller, etc. This will not be described in detail here.
  • FIG. 9 shows a block diagram of some embodiments of the electronic device of the present disclosure.
  • the electronic device 90 can be various types of devices, for example, including but not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc.
  • the electronic device 90 may include a display panel for displaying data and/or execution results utilized in the scheme of the present disclosure.
  • the display panel can be of various shapes, such as a rectangular panel, an elliptical panel, or a polygonal panel.
  • the display panel can be not only a flat panel, but also a curved panel or even a spherical panel.
  • the electronic device 90 of this embodiment includes: a memory 91 and a Processor 92. It should be noted that the components of the electronic device 90 shown in FIG9 are merely exemplary and non-limiting. The electronic device 90 may also have other components according to actual application requirements. The processor 92 may control the other components in the electronic device 90 to perform desired functions.
  • the memory 91 is configured to store one or more computer-readable instructions.
  • the processor 92 is configured to execute the computer-readable instructions
  • the computer-readable instructions when executed by the processor 92, implement a method according to any of the above-described embodiments. The specific implementation and related explanations of each step of the method can be found in the above-described embodiments, and any repetitive details are omitted here.
  • the processor 92 and the memory 91 may communicate with each other directly or indirectly.
  • the processor 92 and the memory 91 may communicate with each other via a network.
  • the network may include a wireless network, a wired network, and/or any combination of wireless networks and wired networks.
  • the processor 92 and the memory 91 may also communicate with each other via a system bus, which is not limited in this disclosure.
  • the processor 92 can be embodied as various appropriate processors, processing devices, etc., such as a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
  • the central processing unit (CPU) can be an X86 or ARM architecture, etc.
  • the memory 91 can include any combination of various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory.
  • the memory 91 can include, for example, a system memory, which stores, for example, an operating system, an application, a boot loader (Boot Loader), a database, and other programs. Various applications and various data can also be stored in the storage medium.
  • a system memory which stores, for example, an operating system, an application, a boot loader (Boot Loader), a database, and other programs.
  • Various applications and various data can also be stored in the storage medium.
  • the programs constituting the software can be installed from a storage medium or a network to a computer system having a dedicated hardware structure, such as the computer system (or electronic device) 100 shown in Figure 10.
  • the computer system can perform various functions, including functions such as those described above.
  • Figure 10 is a block diagram showing an example structure of a computer system that can be used in embodiments of the present disclosure.
  • a central processing unit (CPU) 1001 performs various processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage portion 1008 to a random access memory (RAM) 1003.
  • ROM read-only memory
  • RAM random access memory
  • the central processing unit is merely exemplary, and it may also be other types of processors, such as the various processors described above.
  • the ROM 1002, RAM 1003, and storage portion 1008 may be various forms of computer-readable storage media, as described below.
  • FIG10 shows the ROM 1002, RAM 1003, and storage device 1008 as separate units, the data required for the CPU 1001 to perform various processes, etc., are also stored as needed. 1008, but one or more of them may be combined or located in the same or different memory or storage modules.
  • the CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004.
  • An input/output interface 1005 is also connected to the bus 1004.
  • an input portion 1006 such as a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.
  • an output portion 1007 including a display, such as a cathode ray tube (CRT), liquid crystal display (LCD), speaker, vibrator, etc.
  • a storage portion 1008 including a hard disk, magnetic tape, etc.
  • a communication portion 1009 including a network interface card, such as a LAN card, modem, etc.
  • the communication portion 1009 allows communication processing to be performed via a network, such as the Internet.
  • FIG10 shows that the various devices or modules in the computer system 100 communicate via the bus 1004, they may also communicate via a network or other means, where the network may include a wireless network, a wired network, and/or any combination of wireless and wired networks.
  • a drive 1010 is also connected to the input/output interface 1005 as needed.
  • a removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 1010 as needed so that a computer program read therefrom is installed in the storage section 1008 as needed.
  • a program constituting the software can be installed from a network such as the Internet or a storage medium such as the removable medium 1011 .
  • an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart.
  • the computer program can be downloaded and installed from the network through the communication device 1009, or installed from the storage device 1008, or installed from the ROM 1002.
  • the computer program is executed by the CPU 1001, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.
  • a computer-readable medium may be a tangible medium that may contain or store a program for use by an instruction execution system, apparatus or device or for use in conjunction with an instruction execution system, apparatus or device.
  • the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two.
  • the computer-readable storage medium may be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination of the above.
  • Computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage , magnetic storage device, or any suitable combination thereof.
  • a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
  • a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof.
  • a computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
  • the program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.
  • the computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
  • the present disclosure further provides a computer program, including: instructions, which, when executed by a processor, cause the processor to perform any of the methods of the above embodiments.
  • the instructions may be embodied as computer program codes.
  • the present disclosure also provides an audio interaction processing system, which is described below in conjunction with FIG11 .
  • Figure 11 is a structural diagram of some embodiments of the audio interaction processing system of the present disclosure. As shown in Figure 11, the audio interaction processing system 11 of this embodiment includes: an audio processing server 70 of any embodiment of the present disclosure and a client 80 of any embodiment of the present disclosure.
  • the system 11 further includes a text processing server 110 configured to generate a reply text stream based on the text to be replied converted from the input audio, and send the reply text stream to the audio processing server 70 and the client 80 .
  • a text processing server 110 configured to generate a reply text stream based on the text to be replied converted from the input audio, and send the reply text stream to the audio processing server 70 and the client 80 .
  • the text processing server 110 is further configured to receive the text to be replied sent by the audio processing server 70 , and send a message identifier corresponding to the reply audio stream to the client 80 .
  • computer program code for performing the operations of the present disclosure may be written in one or more programming languages or a combination thereof, including but not limited to object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages.
  • the program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server.
  • the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN).
  • the computer may, alternatively, be connected to an external computer (eg, via the Internet using an Internet service provider).
  • each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function.
  • the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved.
  • each box in the block diagram and/or flowchart, and the combination of the boxes in the block diagram and/or flowchart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
  • the modules, components, or units described in the embodiments of the present disclosure may be implemented in software or hardware.
  • the names of the modules, components, or units do not necessarily limit the modules, components, or units themselves.
  • exemplary hardware logic components include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.
  • FPGAs field programmable gate arrays
  • ASICs application specific integrated circuits
  • ASSPs application specific standard products
  • SOCs systems on chip
  • CPLDs complex programmable logic devices
  • a method for processing audio interaction is provided, which is executed by an audio processing server, including: converting the input audio sent by the client into text to be replied; sending the text to be replied to the text processing server to generate a reply text stream; receiving a request from the client for a reply audio stream corresponding to the reply text stream, wherein the request includes a message identifier corresponding to the reply text stream, and the message identifier is sent by the text processing server to the client; obtaining the reply text stream from the text processing server according to the message identifier; generating a reply audio stream according to the reply text stream; and sending the reply audio stream to the client for playback.
  • the reply audio stream includes multiple audio data packets, each of the multiple audio data packets includes a sequence number identifier of the audio data packet
  • the processing method further includes: receiving a retransmission request sent by the client, wherein the retransmission request includes the sequence number identifier of the last audio data packet received by the client before the network interruption; determining whether there is an audio data packet to be retransmitted based on the sequence number identifier of the last audio data packet, wherein the data packet to be retransmitted is an audio data packet sent to the client after the last audio data packet; if there is an audio data packet to be retransmitted, sending the audio data packet to be retransmitted to the client.
  • the reply text stream includes multiple text data packets, which are processed from the text according to the message identifier.
  • the server obtains the reply text stream including: periodically pulling the current text data packet generated by the text processing server from the text processing server according to the message identifier; or receiving the current text data packet generated and pushed by the text processing server, wherein each of the multiple text data packets includes a message identifier.
  • the processing method also includes: obtaining a first preset amount of text before the reply text stream from the text processing server, wherein generating a reply audio stream based on the reply text stream includes: generating a current audio segment corresponding to the current text data packet based on a text segment in the current text data packet, a text segment in one or more text data packets before the current text data packet, and the first preset amount of text, and generating a current audio data packet in the reply audio stream.
  • generating a current audio segment corresponding to a current text data packet based on a text segment in a current text data packet, a text segment in a text data packet before the current text data packet, and a first preset number of texts includes: determining whether to perform sentence segmentation after the text segment in the current text data packet based on the text segment in the current text data packet and the text segments in one or more text data packets before the current text data packet; in response to performing sentence segmentation after the text segment in the current text data packet, generating a current audio segment corresponding to the current text data packet based on the text segment in the current text data packet, the text segment in the text data packet before the current text data packet, and the first preset number of texts.
  • converting the input audio sent by the client into the text to be replied includes: performing speech recognition on the input audio and converting it into a text stream of the text to be replied; the processing method also includes: sending the text stream of the text to be replied to the client for display.
  • the processing method also includes: obtaining a second preset amount of text before the text to be replied from the text processing server, wherein performing speech recognition on the input audio and converting it into a text stream of the text to be replied includes: performing speech recognition based on the input audio and the second preset amount of text, and converting the input audio into a text stream of the text to be replied.
  • the audio processing server and the text processing server perform data transmission via remote procedure calls.
  • the audio processor transmits data with the client through the audio service gateway, and in response to the start of the audio interaction application, a long connection is established between the audio service gateway and the client.
  • a method for processing audio interaction is provided, which is executed by a client, including: sending input audio to an audio processing server to convert it into text to be replied, wherein the text to be replied is sent to a text processing server to generate a reply text stream; receiving a message identifier corresponding to the reply text stream sent by the text processing server; sending a request for a reply audio stream corresponding to the reply text stream to the audio processing server, wherein The request includes a message identifier; a reply audio stream sent by the audio processing server is received; and the reply audio stream is played.
  • sending the input audio to the audio processing server includes: dividing the input audio into multiple audio segments; compressing each of the multiple audio segments and storing them in order in a first queue; taking out the first audio segment from the first queue and sending it to the audio processing server for speech recognition; in response to receiving a confirmation message sent by the audio processing server, repeatedly taking out the first audio segment from the first queue and sending it to the audio processing server for speech recognition until the first queue is empty.
  • sending the input audio to the audio processing server further includes: in response to not receiving a confirmation message sent by the audio processing server after waiting for a preset period of time, resending a most recently sent audio segment.
  • the reply audio stream includes multiple audio data packets, each of the multiple audio data packets includes a serial number identifier of the audio data packet, and playing the reply audio stream includes: storing the current audio data packet in a second queue according to the serial number identifier of the received current audio data packet; and playing according to the order of the audio data packets in the second queue.
  • storing the current audio data packet in the second queue according to the serial number identifier of the received current audio data packet includes: determining whether an audio data packet with the same serial number identifier as the current audio data packet already exists in the second queue according to the serial number identifier of the received current audio data packet; discarding the current audio data packet if an audio data packet with the same serial number identifier as the current audio data packet already exists in the second queue; and storing the current audio data packet in the queue if an audio data packet with the same serial number identifier as the current audio data packet does not exist in the second queue.
  • the processing method further includes: in response to a network interruption, sending a retransmission request to the audio processing server, wherein the retransmission request includes the sequence number identifier of the last audio data packet received by the client before the network interruption; receiving the retransmitted audio data packet sent by the audio processing server, wherein the retransmitted data packet is a data packet that has been sent by the audio processing server to the client after the last audio data packet.
  • the processing method also includes: receiving a text stream of text to be replied sent by the audio processing server; displaying the text stream of text to be replied; receiving a reply text stream sent by the text processing server, wherein the reply text stream is generated after the text processing server receives the text to be replied sent by the audio processing server; and displaying the reply text stream.
  • the processing method further includes: in response to the start of the audio interaction application, establishing a long connection with the audio service gateway corresponding to the audio processing server, wherein the client transmits data with the audio processor through the audio service gateway.
  • an audio processing server comprising: a conversion module, Configured to convert the input audio sent by the client into text to be replied; the first sending module is configured to send the text to be replied to the text processing server to generate a reply text stream; the receiving module is configured to receive the client's request for the reply audio stream corresponding to the reply text stream, wherein the request includes a message identifier corresponding to the reply text stream, and the message identifier is sent by the text processing server to the client; the obtaining module is configured to obtain the reply text stream from the text processing server according to the message identifier; the generating module is configured to generate a reply audio stream according to the reply text stream; the second sending module is configured to send the reply audio stream to the client for playback.
  • a conversion module Configured to convert the input audio sent by the client into text to be replied
  • the first sending module is configured to send the text to be replied to the text processing server to generate a reply text stream
  • the receiving module is configured to receive the client's request for the reply audio stream corresponding to the reply text
  • a client including: a first sending module, configured to send input audio to an audio processing server to convert it into text to be replied, wherein the text to be replied is sent to the text processing server to generate a reply text stream; a first receiving module, configured to receive a message identifier corresponding to the reply text stream sent by the text processing server; a second sending module, configured to send a request for a reply audio stream corresponding to the reply text stream to the audio processing server, wherein the request includes the message identifier; a second receiving module, configured to receive the reply audio stream sent by the audio processing server; and a playing module, configured to play the reply audio stream.
  • a system for processing audio interaction comprising: an audio processing server according to any embodiment of the present disclosure and a client according to any embodiment of the present disclosure.
  • the processing system further includes: a text processing server configured to generate a reply text stream based on the text to be replied converted from the input audio, and send the reply text stream to the audio processing server and the client.
  • a text processing server configured to generate a reply text stream based on the text to be replied converted from the input audio, and send the reply text stream to the audio processing server and the client.
  • the text processing server is further configured to receive the text to be replied sent by the audio processing server, and send a message identifier corresponding to the reply audio stream to the client.
  • an electronic device comprising: a processor; and a memory coupled to the processor, for storing instructions, which, when executed by the processor, causes the processor to execute the audio interaction processing method of any embodiment of the present disclosure.
  • a computer-readable storage medium on which a computer program is stored.
  • the program is executed by a processor, the method for processing audio interaction of any embodiment of the present disclosure is performed.
  • a computer program product comprising: instructions, which, when executed by a processor, implement the audio interaction processing method of any embodiment of the present disclosure.
  • a computer program comprising: instructions, which, when executed by a processor, implement the audio interaction processing method of any embodiment of the present disclosure.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computational Linguistics (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • Acoustics & Sound (AREA)
  • Information Transfer Between Computers (AREA)
  • Telephonic Communication Services (AREA)

Abstract

一种音频交互的处理方法、系统、服务端、客户端和电子设备，该方法由音频处理服务端执行，包括：将客户端发送的输入音频，转换为待回复文本（S102）；将待回复文本发送至文本处理服务端，生成回复文本流（S104）；接收客户端对回复文本流对应的回复音频流的请求（S106）；根据消息标识，从文本处理服务端获取回复文本流（S108）；根据回复文本流，生成回复音频流（S110）；将回复音频流发送至客户端，进行播放（S112）。

Description

音频交互的处理方法、系统、服务端、客户端和电子设备 技术领域
本公开涉及计算机技术领域，特别涉及一种音频交互的处理方法、系统、服务端、客户端和电子设备。
背景技术
随着互联网技术的发展，人机语音交互技术在越来越多的场景中应用。例如，智能家居、智能客服、智能助手等等。
在一些人机语音交互的场景中，用户输入语音，不仅会得到机器人回复的语音，还会基于用户输入的语音生成相应的文本，并且机器人回复的文本也会进行相应显示，使得用户可以通过文本和语音可以同步接收回复，更好的理解回复的内容提升用户体验。
发明内容
提供该发明内容部分以便以简要的形式介绍构思，这些构思将在后面的具体实施方式部分被详细描述。该发明内容部分并不旨在标识要求保护的技术方案的关键特征或必要特征，也不旨在用于限制所要求的保护的技术方案的范围。
根据本公开的一些实施例，提供了一种音频交互的处理方法，由音频处理服务端执行，包括：将客户端发送的输入音频，转换为待回复文本；将待回复文本发送至文本处理服务端，生成回复文本流；接收客户端对回复文本流对应的回复音频流的请求，其中，请求包括回复文本流对应的消息标识，消息标识由文本处理服务端发送至客户端；根据消息标识，从文本处理服务端获取回复文本流；根据回复文本流，生成回复音频流；将回复音频流发送至客户端，进行播放。
根据本公开的另一些实施例，提供了一种音频交互的处理方法，由客户端执行，包括：将输入音频发送至音频处理服务端，转换为待回复文本，其中，待回复文本被发送至文本处理服务端生成回复文本流；接收文本处理服务端发送的回复文本流对应的消息标识；向音频处理服务端发送对回复文本流对应的回复音频流的请求，其中，请求包括消息标识；接收音频处理服务端发送的回复音频流；对回复音频流进行播放。
根据本公开的又一些实施例，提供了一种音频处理服务端，包括：转换模块，被 配置为将客户端发送的输入音频，转换为待回复文本；第一发送模块，被配置为将待回复文本发送至文本处理服务端，生成回复文本流；接收模块，被配置为客户端对回复文本流对应的回复音频流的请求，其中，请求包括回复文本流对应的消息标识，消息标识由文本处理服务端发送至客户端；获取模块，被配置为根据消息标识，从文本处理服务端获取回复文本流；生成模块，被配置为根据回复文本流，生成回复音频流；第二发送模块，被配置为将回复音频流发送至客户端，进行播放。
根据本公开的再一些实施例，提供一种客户端，包括：第一发送模块，被配置为将输入音频发送至音频处理服务端，转换为待回复文本，其中，待回复文本被发送至文本处理服务端生成回复文本流；第一接收模块，被配置为接收文本处理服务端发送的回复文本流对应的消息标识；第二发送模块，被配置为向音频处理服务端发送对回复文本流对应的回复音频流的请求，其中，请求包括消息标识；第二接收模块，被配置为接收音频处理服务端发送的回复音频流；播放模块，被配置为对回复音频流进行播放。
根据本公开的又一些实施例，提供一种音频交互的处理系统，包括：本公开中的任一实施例的音频处理服务端以及本公开中的任一实施例的客户端。
根据本公开的再一些实施例，提供一种电子设备，包括：处理器；以及耦接至处理器的存储器，用于存储指令，指令被处理器执行时，使处理器执行本公开中的任一实施例的音频交互的处理方法。
根据本公开的又一些实施例，提供一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时执行本公开中的任一实施例的音频交互的处理方法。
根据本公开的再一些实施例，提供一种计算机程序产品，包括：指令，指令被处理器执行时实现本公开中的任一实施例的音频交互的处理方法。
根据本公开的又一些实施例，提供一种计算机程序，包括：指令，指令被处理器执行时实现本公开中的任一实施例的音频交互的处理方法。
通过以下参照附图对本公开的示例性实施例的详细描述，本公开的其它特征、方面及其优点将会变得清楚。
附图说明
下面参照附图说明本公开的优选实施例。此处所说明的附图用来提供对本公开的进一步理解，各附图连同下面的具体描述一起包含在本说明书中并形成说明书的一部 分，用于解释本公开。应当理解的是，下面描述中的附图仅仅涉及本公开的一些实施例，而非对本公开构成限制。在附图中：
图1示出本公开一些实施例的音频交互的处理方法的流程示意图；
图2示出本公开另一些实施例的音频交互的处理方法的流程示意图；
图3示出本公开一些实施例的音频处理服务端进行音频数据包重传的示意图；
图4示出本公开又一些实施例的音频交互的处理方法的流程示意图；
图5示出本公开一些实施例的客户端进行音频数据包重传的示意图；
图6A示出本公开一些实施例的ASR消息链路中的信息交互流程的示意图；
图6B示出本公开一些实施例的TTS消息链路中的信息交互流程的示意图；
图6C示出本公开一些实施例的音频交互的处理方法的信息交互的流程示意图；
图7示出本公开一些实施例的音频处理服务端的结构示意图；
图8示出本公开一些实施例的客户端的结构示意图；
图9示出本公开一些实施例的电子设备的结构示意图；
图10示出本公开一些实施例的计算机系统的结构示意图；
图11示出本公开一些实施例的音频交互的处理系统的结构示意图。
应当明白，为了便于描述，附图中所示出的各个部分的尺寸并不一定是按照实际的比例关系绘制的。在各附图中使用了相同或相似的附图标记来表示相同或者相似的部件。因此，一旦某一项在一个附图中被定义，则在随后的附图中可能不再对其进行进一步讨论。
具体实施方式
下面将结合本公开实施例中的附图，对本公开实施例中的技术方案进行清楚、完整地描述，但是显然，所描述的实施例仅仅是本公开一部分实施例，而不是全部的实施例。以下对实施例的描述实际上也仅仅是说明性的，决不作为对本公开及其应用或使用的任何限制。应当理解的是，本公开可以通过各种形式来实现，而且不应该被解释为限于这里阐述的实施例。
应当理解，本公开的方法实施方式中记载的各个步骤可以按照不同的顺序执行，和/或并行执行。此外，方法实施方式可以包括附加的步骤和/或省略执行示出的步骤。本公开的范围在此方面不受限制。除非另外具体说明，否则在这些实施例中阐述的部件和步骤的相对布置、数字表达式和数值应被解释为仅仅是示例性的，不限制本公开 的范围。
本公开中使用的术语“包括”及其变型意指至少包括后面的元件/特征、但不排除其他元件/特征的开放性术语，即“包括但不限于”。此外，本公开使用的术语“包含”及其变型意指至少包含后面的元件/特征、但不排除其他元件/特征的开放性术语，即“包含但不限于”。因此，包括与包含是同义的。术语“基于”意指“至少部分地基于”。
整个说明书中所称“一个实施例”、“一些实施例”或“实施例”意味着与实施例结合描述的特定的特征、结构或特性被包括在本发明的至少一个实施例中。例如，术语“一个实施例”表示“至少一个实施例”；术语“另一实施例”表示“至少一个另外的实施例”；术语“一些实施例”表示“至少一些实施例”。而且，短语“在一个实施例中”、“在一些实施例中”或“在实施例中”在整个说明书中各个地方的出现不一定全都指的是同一个实施例，但是也可以指同一个实施例。
需要注意，本公开中提及的“第一”、“第二”等概念仅用于对不同的装置、模块或单元进行区分，并非用于限定这些装置、模块或单元所执行的功能的顺序或者相互依存关系。除非另有指定，否则“第一”、“第二”等概念并非意图暗示如此描述的对象必须按时间上、空间上、排名上的给定顺序或任何其他方式的给定顺序。
需要注意，本公开中提及的“一个”、“多个”的修饰是示意性而非限制性的，本领域技术人员应当理解，除非在上下文另有明确指出，否则应该理解为“一个或多个”。
本公开实施方式中的多个装置之间所交互的消息或者信息的名称仅用于说明性的目的，而并不是用于对这些消息或信息的范围进行限制。
下面结合附图对本公开的实施例进行详细说明，但是本公开并不限于这些具体的实施例。下面这些具体实施例可以相互结合，对于相同或者相似的概念或过程可能在某些实施例不再赘述。此外，在一个或多个实施例中，特定的特征、结构或特性可以由本领域的普通技术人员从本公开将清楚的任何合适的方式组合。
用户与机器人交互的过程中，可以基于文本、语音或者二者结合进行交互。在一些场景中，用户输入语音(音频)可以实时转换为文本进行显示，机器人可以同时回复文本和语音，提升用户的交互体验。
机器人要实现文本和语音的同时回复，需要配置能够处理文本的文本处理服务端和能够处理音频的音频处理服务端。对于用户机器人文本和语音回复的及时性以及同 步程度，直接影响着用户的体验。
本公开提出一种音频交互的处理方法，提高针对输入音频进行文本和语音回复的效率和及时性。
下面结合图1～6C描述本公开音频交互的处理方法的一些实施例。
图1为本公开的音频交互的处理方法的一些实施例的结构图。如图1所示，该实施例的方法包括：步骤S102～S112。该实施例的方法可以由音频处理服务端执行。音频处理服务端可以由硬件或软件实现，例如为可以实现音频处理的服务器或应用程序等。
在步骤S102中，将客户端发送的输入音频，转换为待回复文本。
客户端可以由硬件或软件实现，例如为可以实现终端或终端中设置的应用程序等。在人机交互场景中，用户可以通过客户端输入音频，客户端可以将音频文件转码后划分为多段音频，进而将多段音频压缩后使用队列进行本地存储，然后采用流式传输的方式发送到音频处理服务端进行音频识别，转换为待回复文本。音频处理服务端可以依托于机器学习模型实现，例如，机器学习模型可以实现语音和文本的双向转换。例如，音频处理服务端为具备ASR(Automatic Speech Recognition，自动语音识别)和TTS(Text To Speech，从文本到语音)功能的服务端。
在步骤S104中，将待回复文本发送至文本处理服务端，生成回复文本流。
音频处理服务端将待回复文本直接发送至文本处理服务端，相对于将待回复文本发送至客户端后，由客户端发送至文本处理服务端的方法，提高了文本处理服务端获取待回复文本的效率，进而提高了整体处理效率。
文本处理服务端负责根据待回复文本生成回复文本流。文本处理服务端可以依托于机器学习模型实现，例如基于大语言模型(Large Language Model，简称：LLM)或者基础模型(Foundation Model)实现。例如，文本处理服务端为文本处理服务端可以流式输出实时生成的回复文本片段，即生成回复文本流，可以提高回复文本的生成的及时性和效率，不需要等到生成回复文本的全部内容后再进行反馈，提升用户体验。
在步骤S106中，接收客户端对回复文本流对应的回复音频流的请求。
例如，该请求包括回复文本流对应的消息标识，消息标识由文本处理服务端发送至客户端。文本处理服务端接收到待回复文本后，可以生成对应的消息标识发送至客户端。消息标识用于标识针对待回复文本生成的回复文本流。客户端接收到该消息标 识后生成回复音频流的请求，用于向音频处理服务端请求回复音频流。
在步骤S108中，根据消息标识，从文本处理服务端获取回复文本流。
音频处理服务端可以根据消息标识从文本处理服务端直接获取回复文本流，相对于文本服务端将回复文本流发送至客户端，再由客户端将回复文本流发送至音频处理服务端，提高了音频处理服务端获取回复文本流的效率，进而提高了音频处理服务端后续生成回复音频的效率。
音频处理服务端从文本处理服务端获取的是回复文本流，即文本处理服务端采用流式传输将生成的回复文本片段实时传输到音频处理服务端，而不是生成整个回复文本后再发送至音频处理服务端，提高了传输效率，以及音频处理服务端后续的处理效率。
在步骤S110中，根据回复文本流，生成回复音频流。
音频处理服务端将接收到的回复文本流中各个回复文本片段，实时转换为音频，并流式输出回复音频流。
在步骤S112中，将回复音频流发送至客户端，进行播放。
音频处理服务端将实时生成的回复音频片段流式输出，形成回复音频流传输到客户端实时播放，不需要生成全部回复音频后再返回至客户端进行播放。
上述实施例的方法，提供了音频交互的整体实现方案。音频处理服务端接收客户端输入的音频后转换为待回复文本，进而与文本处理服务端直接进行交互，一方面，将待回复文本直接发送至文本处理服务端以生成回复文本流，另一方面，直接从文本处理服务端获取回复文本流，生成回复音频流，返回客户端播放。音频处理服务端与文本处理服务端直接通信，相对于通过客户端转发待回复文本和回复文本流的方式，提高了待回复文本和回复文本流的传输效率，使得文本处理服务端能够及时获取待回复文本，音频处理服务端能够及时获取回复音频流，从而提高了回复文本流和回复音频流的生成效率。并且整个过程中回复文本和回复音频都采用流式生成和流式传输的方式，能够及时针对输入的音频进行反馈，提高针对输入音频进行文本和语音回复的效率和及时性。
此外，由于音频处理服务端从文本处理服务端实时获取回复音频流，并生成回复音频流，可以提高在客户端进行文本和语音回复的同步效果，提升用户体验。
下面结合一些具体实施例进一步描述本公开的音频交互的处理方法。
在一些实施例中，回复文本流包括多个文本数据包，音频处理服务端根据消息标 识，周期性从文本处理服务端拉取文本处理服务端生成的当前文本数据包，或者接收文本处理服务端推送的生成的当前文本数据包，其中，多个文本数据包中每个文本数据包包括消息标识。
文本处理服务端流式生成文本数据包，采用流式传输形成回复文本流。例如，文本处理服务端生成的整个回复文本为“你好，很高兴和你聊天”，采用流式生成文本数据包，可以针对这句话生成多个数据包，例如，“你好”、“很高兴”、“和你”、“聊天”分别作为不同数据包的内容。流式生成和传输的方式，使得回复的及时性更好，效率更高，用户的体验也更好。每个文本数据包包括消息标识(message_id)，使得音频处理服务端可以准确识别属于针对输入的音频进行回复的各个文本数据包。音频处理服务端可以采用消息标识进行轮询，获取各个文本数据包，也可以由文本处理服务端在生成每个文本数据包后推送至音频处理服务端。
由音频处理服务端拉取回复音频流，可以降低对消息链路的侵入，提升整体链路的稳定性，音频处理服务端进行ASR和TTS两个链路相对隔离，如果通过文本处理服务端推送回复文本流，需要配合音频处理服务端提供主动推送能力，对现有的链路改动成本较高。
在一些实施例中，音频处理服务端从文本处理服务端获取回复文本流之前的第一预设数量的文本，根据当前文本数据包中的文本片段、当前文本数据包之前的一个或多个文本数据包中的文本片段、以及第一预设数量的文本，生成当前文本数据包对应的当前音频片段，并生成回复音频流中的当前音频数据包。
每轮交互过程包括客户端输入音频，转换为待回复文本，生成回复文本流，生成回复音频流。针对每轮交互过程，音频处理服务端可以获取之前一轮或多轮对话的第一预设数量的文本，结合当前文本数据包中的文本片段，当前文本数据包之前的文本片段，生成当前文本数据包对应的当前音频片段。音频处理服务端可以结合多轮对话的上下文生成当前音频片段，能够更好的理解语义和语境，提高了音频生成的准确性，提高了朗读的准确性。
由于回复文本一般在文本处理服务端存储，因此可以从文本处理服务端获取历史一轮或多轮交互的文本。如果音频处理服务端可以存储历史每轮交互过程中的待回复文本和回复文本，则可以每次获取回复文本流后进行存储即可，不需要反复从文本处理服务端获取。通过打通文本处理服务端和音频处理服务端之间的通信，可以降低回复文本流和历史交互的文本的获取时延，提高效率，从而提高音频处理服务端生成回 复音频流的效率。
在一些实施例中，音频处理服务端根据当前文本数据包中的文本片段、当前文本数据包之前的一个或多个文本数据包中的文本片段，确定是否在当前文本数据包中的文本片段后进行断句；响应于在当前文本数据包中的文本片段后进行断句，根据当前文本数据包中的文本片段、当前文本数据包之前的文本数据包中的文本片段、以及第一预设数量的文本，生成当前文本数据包对应的当前音频片段。
音频处理服务端根据当前文本数据包中的文本片段、当前文本数据包之前的一个或多个文本数据包中的文本片段，进行断句判断，如果需要断句，则生成当前音频片段，如果不需要断句则继续接收文本数据包后，在需要断句时再生成音频片段。音频处理服务端结合上下文进行断句判断，提高了生成的回复音频的流畅度和朗读的流畅度。此外，相对于文本处理服务端将回复文本流发送至客户端，由客户端进行断句再发送至音频处理服务端生成音频的方式，提高了生成音频的效率。
文本处理服务端流式生成回复文本流，发送至客户端进行显示，音频处理服务端流式生成回复音频流发送至客户端进行播放，此外，在客户端输入音频后，音频处理服务端也可以流式生成会生成待回复文本，发送至客户端进行显示。在交互过程中，对于需要发送至客户端进行显示或播放的内容，都采用流式生成和传输的方式，降低了在客户端进行显示和播放的时延，提高了客户端显示和播放的及时性和效率。
在一些实施例中，音频处理服务端对输入音频进行语音识别，转换为待回复文本的文本流；将待回复文本的文本流发送至客户端进行显示。
音频处理服务端采用ASR技术，流式生成待回复文本的文本流，流式输出至客户端进行显示，提高了显示待回复文本的及时性，提升用户体验。
在一些实施例中，音频处理服务端从文本处理服务端获取待回复文本之前的第二预设数量的文本，根据输入音频和第二预设数量的文本，进行语音识别，将输入音频转换为待回复文本的文本流。
针对每轮交互过程，音频处理服务端可以获取之前一轮或多轮对话的第二预设数量的文本，结合输入音频，生成待回复文本的文本流。音频处理服务端可以结合多轮对话的上下文生成待回复文本的文本流，能够更好的理解语义和语境，提高了待回复文本生成的准确性。
如果音频处理服务端可以存储历史每轮交互过程中的待回复文本和回复文本，不需要反复从文本处理服务端获取。通过打通文本处理服务端和音频处理服务端之间的 通信，可以降低历史交互的文本的获取时延，提高效率，从而提高音频处理服务端生成待回复文本的效率。
在一些实施例中，音频处理服务端与文本处理服务端通过远程过程调用方式进行数据传输。
例如，音频处理服务端和文本处理服务端之间可以通过RPC(google Remote Procedure Call，远程过程调用)或gRPC通信，实现上述待回复文本和回复文本流的传输，提高传输效率，以及后续对客户端回复的效率。
在一些实施例中，音频处理器通过音频服务网关与客户端进行数据传输，响应于音频交互应用的启动，音频服务网关与客户端之间建立长连接。相应的，客户端响应于音频交互应用的启动，与音频处理服务端对应的音频服务网关建立长连接，客户端通过音频服务网关与音频处理器进行数据传输。
音频交互应用启动即建立长连接，该长连接例如为Websocket连接。该长连接可以复用，减少连接反复建立的时延，提高后续传输和音频交互的效率。
在客户端与音频服务端或文本服务端进行通信的过程中，会出现网络切换或断网的情况，可能会导致音频处理服务端发送至客户端的音频数据包丢失，当网络恢复后，导致客户端播放的音频出现问题。
下面结合图2描述本公开音频交互的处理方法的另一些实施例。
图2为本公开的音频交互的处理方法的另一些实施例的结构图。如图2所示，该实施例的方法包括：步骤S202～S206。
在步骤S202中，客户端响应于网络中断，向音频处理服务端发送重传请求，相应的，音频服务端接收客户端发送的重传请求。
在一些实施例中，回复音频流包括多个音频数据包，多个音频数据包中每个音频数据包包括该音频数据包的序号标识。通过设置序号标识，可以确定各个音频数据包之间的顺序，从而在出现影片数据包丢失、重传等情况，可以确定准确的识别各个音频数据包的顺序和是否出现缺失。
例如，重传请求包括客户端在网络中断前收到的最后一个音频数据包的序号标识。
在步骤S204中，音频处理服务端根据客户端在网络中断前收到的最后一个音频数据包的序号标识，确定是否存在待重传音频数据包。
例如，待重传数据包为在客户端在网络中断前收到的最后一个音频数据包之后向客户端已发送的音频数据包。音频处理服务端根据重传请求可以确定客户端已经接收 到的最后一个音频数据包的序号标识，那么在该序号标识之后的音频数据包如果已经发送，则可能丢失，即为待重传音频数据包。
在步骤S206中，音频处理服务端在存在待重传音频数据包的情况下，向客户端发送待重传音频数据包。
客户端接收音频处理服务端发送的重传音频数据包，重传数据包为音频处理服务端在客户端在网络中断前收到的最后一个音频数据包之后向客户端已发送过的数据包。
如图3所示，文本处理服务端将生成的回复文本流的各个文本数据包，Text T1、Text T2……通过gRPC发送至音频处理服务端。音频处理服务端流式生成多个音频数据包(Stream block，流式数据块)，S1、S2……通过外网传输至客户端。如果在S1后发生网络中断，客户端则向音频处理服务端发送重传请求。重传请求中可以包括S1的序号标识，音频处理服务端则根据该序号标识确定是否需要重传，以及重传哪些音频数据包。
例如，回复音频流中每个音频数据包包括该音频数据包对应的回复文本、音频数据以及Markdown过滤后回复文本。每个音频数据包中可以包括一个关键字段Seq_id表示序号标识。如果发生断网等情况，客户端可以发送重传请求告诉音频处理服务端发生网络错误，需要重传其接收到的最后一个音频数据包的Seq_id之后的所有音频数据包。客户端可以对接收到的各个音频数据包的Seq_id进行识别，避免接收重复的音频数据包。
上述实施例的方法，在发生网络中断的情况下，客户端只需要通知音频处理服务端序号标识，即可拉取后续的音频进行播放。如果由客户端接收文本处理服务端的回复文本流后进行断句再发送至音频处理服务端生成回复音频流，发生网络中断后等待网络恢复，客户端会将回复文本数据包继续发送至音频处理服务端，然而音频处理服务端会无法区分接收到的数据包是否用于继续合成音频。音频处理服务端和文本处理服务端之间通过内网通信，网络稳定且网络错误的恢复更容易处理，并增加对回复文本和音频的缓存，使得重传、续传更简单，效率更高。上述实施例的方法，减少回复音频流中断导致的回复音频播放失败，提升用户体验。
下面结合图4描述本公开的音频交互的处理方法的另一些实施例，该实施例的方法可以由客户端执行。
图4为本公开的音频交互的处理方法的一些实施例的结构图。如图4所示，该实 施例的处理方法包括：步骤S402～S410。该实施例的方法可以由客户端执行。
在步骤S402中，将输入音频发送至音频处理服务端，转换为待回复文本。
待回复文本被音频处理服务端发送至文本处理服务端生成回复文本流，具体可以参考前述实施例，在此不再赘述。
在步骤S404中，接收文本处理服务端发送的回复文本流对应的消息标识。
文本处理服务端接收到待回复文本后，可以生成消息标识并发送至客户端，具体可以参考前述实施例，在此不再赘述。
在步骤S406中，向音频处理服务端发送对回复文本流对应的回复音频流的请求。
客户端发送的请求包括消息标识。
在步骤S408中，接收音频处理服务端发送的回复音频流。
回复音频流是音频处理服务端根据消息标识从文本处理服务端获取回复文本流后生成的。音频处理服务端从文本处理服务端获取回复音频流的过程，可以参考签署实施了，在此不再赘述。
在步骤S410中，对回复音频流进行播放。
上述实施例的方法，客户端将输入音频发送至音频处理服务端转换为待回复文本，进而与文本处理服务端直接进行交互，一方面，将待回复文本直接发送至文本处理服务端以生成回复文本流，另一方面，直接从文本处理服务端获取回复文本流，生成回复音频流，返回客户端播放。音频处理服务端与文本处理服务端直接通信，相对于通过客户端转发待回复文本和回复文本流的方式，提高了待回复文本和回复文本流的传输效率，使得文本处理服务端能够及时获取待回复文本，音频处理服务端能够及时获取回复音频流，从而提高了回复文本流和回复音频流的生成效率。并且整个过程中回复文本和回复音频都采用流式生成和流式传输的方式，能够及时针对输入的音频进行反馈，提高针对输入音频进行文本和语音回复的效率和及时性。
此外，由于音频处理服务端从文本处理服务端实时获取回复音频流，并生成回复音频流，可以提高在客户端进行文本和语音回复的同步效果，提升用户体验。
下面结合一些实施例进一步描述本公开的音频交互的处理方法。
在一些实施例中，客户端将输入音频划分为多个音频片段；将多个音频片段中每个音频片段进行压缩后按顺序存入第一队列；从第一队列中取出第一个音频片段，发送至音频处理服务端进行语音识别；响应于接收到音频处理服务端发送的确认收到的信息，重复执行从第一队列中取出第一个音频片段，发送至音频处理服务端进行语音 识别，直至第一队列为空。
客户端将压缩的多个音频片段存入第一队列，可以按照顺序发送至音频处理服务端进行语音识别转换为待回复文本。通过音频处理服务端的回复确认可以确定每一个音频片段都发送至音频处理服务端，减少由于网络中断等情况导致的音频片段的丢失而无法发现的情况，提高语音交互的准确性。
在一些实施例中，客户端响应于等待预设时长未接收到音频处理服务端发送的确认收到的信息，重新发送最近发送的一个音频片段。
客户端等待预设时长未接收到音频处理服务端发送的确认收到的信息，可能是对应的音频片段丢失，重新发送对应的音频片段，降低音频处理服务端无法准确识别输入音频的概率，提高后续语音交互的准确性。
如图5所示，客户端向音频处理服务端发送输入音频的各个音频数据包，如果客户端发送S1数据包之后，网络发生中断，客户端等待预设时长无法接收到音频处理服务端发送的确认收到的信息，则重新发送S1数据包。
客户端将输入音频文件转码后划分为多个音频片段在本地存入第一队列，通过流式传输到音频处理服务端进行音频识别，当接收到音频处理服务端的确认收到的信息后将之后的音频片段出队，并进行后续传输。当检测发送网络中断等异常的情况那么通过配置判断是否需要进行续传以及每次等待预设时长进行重试，将最近发送的队头的音频片段进行重试即可，这样就能够保证在网络切换或网络质量较差的情况下输入音频的流式识别能够正常运行，得到正确的识别结果。
在一些实施例中，回复音频流包括多个音频数据包，多个音频数据包中每个音频数据包包括该音频数据包的序号标识，客户端根据接收的当前音频数据包的序号标识，将当前音频数据包存入第二队列；按照第二队列中各个音频数据包的顺序进行播放。
在音频数据包中添加序号标识，可以使客户端按照顺序播放各个音频数据包，即使出现网络传输错误，也可以实现回复音频流的准确播放。
在一些实施例中，客户端根据接收的当前音频数据包的序号标识，确定第二队列中是否已存在与当前音频数据包的序号标识相同的音频数据包；在第二队列中已存在与当前音频数据包的序号标识相同的音频数据包，将当前音频数据包丢弃；在第二队列中不存在与当前音频数据包的序号标识相同的音频数据包，将当前音频数据包存入队列。
客户端根据音频数据包的序号标识可以确定是否存在重复的音频数据包，尤其针 对网络中断重传、续传等情况下，可能存在音频数据包的重复传输，提高了回复音频流传输的准确性，提高回复音频流播放的准确性。
在一些实施例中，客户端接收音频处理服务端发送的待回复文本的文本流；将待回复文本的文本流进行显示；接收文本处理服务端发送的回复文本流；将回复文本流进行显示。
回复文本流是文本处理服务端接收到音频处理服务端发送的待回复文本后生成的。客户端会接收音频处理服务端发送的待回复文本的文本流、文本处理服务端发送的回复文本流、音频处理服务端发送的回复音频流，进行显示和播放。
本公开实施例的方法针对输入音频可以实时获取回复文本流，快速转换为回复音频流，并且实时推送至客户端，进行播放。在流式输出的实时性更高的场景中极大提升了回复文本转音频播放体验，并且结合上下文实现回复音频流的生成可以提高准确性。通过对回复音频流、输入音频流进行重试重传，能够确保在长文本或用户长时间ASR识别过程中即便出现网络质量下降、断网、抖动等情况依旧能够准确进行ASR、TTS的正常识别或播播放。
下面结合一些应用例描述本公开的音频交互的处理方法。
图6A为ASR消息链路中的信息交互流程。如图6A所示，(0)客户端(client)的音频交互应用中的音频模块(APP Audio)在音频交互应用开启时，与音频服务网关(GateWay)建立长连接。用户发送语音，开始ASR消息传输，输入到客户端的音频交互应用中的音频模块，(1)通过音频模块将用户语音(输入音频)发送至音频服务网关。音频服务网关属于服务侧设备。(2)音频服务网关将用户语音发送至音频处理模块。音频处理模块与ASR模块可以形成前述实施例中的音频处理服务端。ASR模块也可以单独部署，音频处理模块单独作为前述实施例中的音频处理服务端。音频处理模块通过gRPC调用ASR模块实现用户语音识别，得到语音识别文本(待回复文本)。(3)音频处理模块向音频服务网关返回语音识别文本。(4)音频服务网关将语音设备文本返回至客户端中的音频模块，(5)客户端中的音频模块将语音识别文本发送至客户端中的消息模块(APP message)进行显示。(6)音频处理模块得到语音识别文本后，会向交互处理模块发送语音识别文本。(7)交互处理模块向LLM发送文本消息，请求LLM进行文本消息处理。交互处理模块与LLM可以形成前述实施例中的文本处理服务端。LLM也可以单独部署，交互处理模块单独作为前述实施例中的文本处理服务端。(8)交互处理模块将用户消息全文(语音识别文本)发送至LLM， LLM向交互处理模块返回轮询用的key，(9)交互处理模块根据key轮询LLM的增量回复。LLM流式生成回复，交互处理模块只需要每次获取增量回复文本即可。(10)交互处理模块将回复文本发送至Frotier(边界)设备，(11)Frotier设备将回复文本发送至客户端中的消息模块进行显示。
音频服务网关和Frotier设备可以作为WebSocket接入层实现外网和内网之间的传输。
图6B为TTS消息链路中的信息交互流程。如图6B所示，(1)音频处理模块将语音识别文本(待回复文本)发送至交互处理模块，(2)交互处理模块将语音识别文本发送至LLM。(3)交互处理模块向Frotier设备发送回复文本流的Message id(消息标识)。(4)Frotier设备将回复文本流的Message id发送至客户端中的消息模块。(5)消息模块将回复文本流的Message id发送至客户端中音频模块开始TTS会话。(6)音频模块向音频服务网关发送订阅TTS音频流(回复音频流)的请求。(7)音频服务网关向音频处理模块发送订阅TTS音频流(回复音频流)的请求。(8)音频处理模块向交互处理模块请求回复文本。(9)音频处理模块根据回复文本生成TTS音频流(回复音频流)发送至音频服务网关。(10)音频服务网关将TTS音频流发送至客户端中的音频模块进行播放。
如图6C为音频交互的处理方法的信息交互流程。如图6C所示，为实现本公开的方案，可以在客户端中新增音频SDK用于与音频处理服务端进行交互。音频SDK可以位于客户端内，也可以独立设置。(1)客户端向音频SDK发送输入音频(即Audio上行)。(2)音频SDK向音频处理服务端发送输入音频。(3)音频处理服务端向文本处理服务端请求上下文内容(即本轮交互之前的历史交互的文本)。(4)文本处理服务端向音频处理服务端返回上下文内容。(5)音频处理服务端进行ASR识别，得到ASR识别文本(待回复文本)。音频处理服务端进行ASR识别时结合上下文内容，更加准确。(6)音频处理服务端将ASR识别文本发送至音频SDK。(7)音频SDK将ASR识别文本发送至客户端。(8)客户端将ASR识别文本进行显示。
(9)音频处理服务端向文本处理服务端发送ASR识别文本(待回复文本)。(10)文本处理服务端向客户端发送消息标识(Message id)。(11)文本处理服务端生成回复文本后以流式传输方式向客户端返回回复文本流。(12)客户端将回复文本进行显示。
(13)客户端向音频SDK发送TTS音频数据(回复音频流)的请求，其中携带 Message id。(14)音频SDK向音频处理服务端发送TTS音频数据的请求。(15)音频处理服务端向文本处理服务端请求回复文本。(16)文本处理服务端向音频处理服务端发送回复文本流。(17)音频处理服务端根据回复文本流生成TTS音频流。(18)音频处理服务端向音频SDK发送TTS音频流。(19)音频SDK向客户端发送TTS音频流。(20)客户端播放TTS音频。
上述步骤(1)～(8)为生成待回复文本并显示的过程，(9)～(12)为生成回复文本并显示的过程。(13)～(20)为生成回复音频流并播放的过程，为了便于理解将同一过程中的步骤放在一起进行描述，实际处理过程中，这些步骤的顺序并不限于图中所示顺序和编号。例如，音频处理服务端生成ASR识别文本后就可以发送至文本处理服务段，即(6)和(9)可以是同步或者(9)在(6)之后。
本公开还提供一种音频处理服务端，下面结合图7进行描述。
图7为本公开的音频处理服务端的一些实施例的结构图。如图7所示，该实施例的音频处理服务端70包括：转换模块710，第一发送模块720，接收模块730，获取模块740，生成模块750，第二发送模块760。
转换模块710被配置为将客户端发送的输入音频，转换为待回复文本。
第一发送模块720被配置为将待回复文本发送至文本处理服务端，生成回复文本流。
接收模块730被配置为客户端对回复文本流对应的回复音频流的请求，其中，请求包括回复文本流对应的消息标识，消息标识由文本处理服务端发送至客户端。
获取模块740被配置为根据消息标识，从文本处理服务端获取回复文本流。
生成模块750被配置为根据回复文本流，生成回复音频流。
第二发送模块760被配置为将回复音频流发送至客户端，进行播放。
在一些实施例中，回复音频流包括多个音频数据包，多个音频数据包中每个音频数据包包括该音频数据包的序号标识，音频处理服务端70还包括确定模块770，接收模块730还被配置为接收客户端发送的重传请求，其中，重传请求包括客户端在网络中断前收到的最后一个音频数据包的序号标识；确定模块770被配置为根据最后一个音频数据包的序号标识，确定是否存在待重传音频数据包，其中，待重传数据包为在最后一个音频数据包之后向客户端已发送的音频数据包；第二发送模块760被配置为在存在待重传音频数据包的情况下，向客户端发送待重传音频数据包。
在一些实施例中，回复文本流包括多个文本数据包，获取模块740被配置为根据 消息标识，周期性从文本处理服务端拉取文本处理服务端生成的当前文本数据包；或者接收文本处理服务端推送的生成的当前文本数据包，其中，多个文本数据包中每个文本数据包包括消息标识。
在一些实施例中，获取模块740还被配置为从文本处理服务端获取回复文本流之前的第一预设数量的文本，生成模块750被配置为根据当前文本数据包中的文本片段、当前文本数据包之前的一个或多个文本数据包中的文本片段、以及第一预设数量的文本，生成当前文本数据包对应的当前音频片段，并生成回复音频流中的当前音频数据包。
在一些实施例中，生成模块750被配置为根据当前文本数据包中的文本片段、当前文本数据包之前的一个或多个文本数据包中的文本片段，确定是否在当前文本数据包中的文本片段后进行断句；响应于在当前文本数据包中的文本片段后进行断句，根据当前文本数据包中的文本片段、当前文本数据包之前的文本数据包中的文本片段、以及第一预设数量的文本，生成当前文本数据包对应的当前音频片段。
在一些实施例中，转换模块710被配置为对输入音频进行语音识别，转换为待回复文本的文本流；第二发送模块760还被配置为将待回复文本的文本流发送至客户端进行显示。
在一些实施例中，获取模块740还被配置为从文本处理服务端获取待回复文本之前的第二预设数量的文本，转换模块710被配置为根据输入音频和第二预设数量的文本，进行语音识别，将输入音频转换为待回复文本的文本流。
在一些实施例中，音频处理服务端与文本处理服务端通过远程过程调用方式进行数据传输。
在一些实施例中，音频处理器通过音频服务网关与客户端进行数据传输，响应于音频交互应用的启动，音频服务网关与客户端之间建立长连接。
本公开还提供一种客户端，下面结合图8进行描述。
图8为本公开的客户端的一些实施例的结构图。如图8所示，该实施例的客户端80包括：第一发送模块810，第一接收模块820，第二发送模块830，第二接收模块840，播放模块850。
第一发送模块810被配置为将输入音频发送至音频处理服务端，转换为待回复文本，其中，待回复文本被发送至文本处理服务端生成回复文本流。
第一接收模块820被配置为接收文本处理服务端发送的回复文本流对应的消息标 识。
第二发送模块830被配置为向音频处理服务端发送对回复文本流对应的回复音频流的请求，其中，请求包括消息标识。
第二接收模块840被配置为接收音频处理服务端发送的回复音频流。
播放模块850被配置为对回复音频流进行播放。
在一些实施例中，第一发送模块810被配置为将输入音频划分为多个音频片段；将多个音频片段中每个音频片段进行压缩后按顺序存入第一队列；从第一队列中取出第一个音频片段，发送至音频处理服务端进行语音识别；响应于接收到音频处理服务端发送的确认收到的信息，重复执行从第一队列中取出第一个音频片段，发送至音频处理服务端进行语音识别，直至第一队列为空。
在一些实施例中，第一发送模块810被配置为响应于等待预设时长未接收到音频处理服务端发送的确认收到的信息，重新发送最近发送的一个音频片段。
在一些实施例中，回复音频流包括多个音频数据包，多个音频数据包中每个音频数据包包括该音频数据包的序号标识，播放模块850被配置为根据接收的当前音频数据包的序号标识，将当前音频数据包存入第二队列；按照第二队列中各个音频数据包的顺序进行播放。
在一些实施例中，播放模块850被配置为根据接收的当前音频数据包的序号标识，确定第二队列中是否已存在与当前音频数据包的序号标识相同的音频数据包；在第二队列中已存在与当前音频数据包的序号标识相同的音频数据包，将当前音频数据包丢弃；在第二队列中不存在与当前音频数据包的序号标识相同的音频数据包，将当前音频数据包存入队列。
在一些实施例中，第二发送模块830还被配置为响应于网络中断，向音频处理服务端发送重传请求，其中，重传请求包括客户端在网络中断前收到的最后一个音频数据包的序号标识；第二接收模块840还被配置为接收音频处理服务端发送的重传音频数据包，其中，重传数据包为音频处理服务端在最后一个音频数据包之后向客户端已发送过的数据包。
在一些实施例中，客户端80还包括第三接收模块860，被配置为接收音频处理服务端发送的待回复文本的文本流；显示模块870，被配置为将待回复文本的文本流进行显示。
在一些实施例中，客户端80还包括第四接收模块880，被配置为接收文本处理服 务端发送的回复文本流，其中，回复文本流是文本处理服务端接收到音频处理服务端发送的待回复文本后生成的；显示模块870还被配置为将回复文本流进行显示。
在一些实施例中，响应于音频交互应用的启动，客户端80与音频处理服务端对应的音频服务网关建立长连接，其中，客户端通过音频服务网关与音频处理器进行数据传输。
应注意，上述各个单元(模块)仅是根据其所实现的具体功能划分的逻辑模块，而不是用于限制具体的实现方式，例如可以以软件、硬件或者软硬件结合的方式来实现。在实际实现时，上述各个单元可被实现为独立的物理实体，或者也可由单个实体(例如，处理器(CPU或DSP等)、集成电路等)来实现。此外，上述各个单元在附图中用虚线示出指示这些单元可以并不实际存在，而它们所实现的操作/功能可由处理电路本身来实现。
此外，尽管未示出，该设备也可以包括存储器，其可以存储由设备、设备所包含的各个单元在操作中产生的各种信息、用于操作的程序和数据、将由通信单元发送的数据等。存储器可以是易失性存储器和/或非易失性存储器。例如，存储器可以包括但不限于随机存储存储器(RAM)、动态随机存储存储器(DRAM)、静态随机存取存储器(SRAM)、只读存储器(ROM)、闪存存储器。当然，存储器可也位于该设备之外。可选地，尽管未示出，但是该设备也可以包括通信单元，其可用于与其它装置进行通信。在一个示例中，通信单元可以被按照本领域已知的适当方式来实现，例如包括天线阵列和/或射频链路等通信部件，各种类型的接口、通信单元等等。这里将不再详细描述。此外，设备还可以包括未示出的其它部件，诸如射频链路、基带处理单元、网络接口、处理器、控制器等。这里将不再详细描述。
本公开的一些实施例还提供一种电子设备。图9示出本公开的电子设备的一些实施例的框图。例如，在一些实施例中，电子设备90可以为各种类型的设备，例如可以包括但不限于诸如移动电话、笔记本电脑、数字广播接收器、PDA(个人数字助理)、PAD(平板电脑)、PMP(便携式多媒体播放器)、车载终端(例如车载导航终端)等等的移动终端以及诸如数字TV、台式计算机等等的固定终端。例如，电子设备90可以包括显示面板，以用于显示根据本公开的方案中所利用的数据和/或执行结果。例如，显示面板可以为各种形状，例如矩形面板、椭圆形面板或多边形面板等。另外，显示面板不仅可以为平面面板，也可以为曲面面板，甚至球面面板。
如图9所示，该实施例的电子设备90包括：存储器91以及耦接至该存储器91的 处理器92。应当注意，图9所示的电子设备90的组件只是示例性的，而非限制性的，根据实际应用需要，该电子设备90还可以具有其他组件。处理器92可以控制电子设备90中的其它组件以执行期望的功能。
在一些实施例中，存储器91用于存储一个或多个计算机可读指令。处理器92用于运行计算机可读指令时，计算机可读指令被处理器92运行时实现根据上述任一实施例的方法。关于该方法的各个步骤的具体实现以及相关解释内容可以参见上述的实施例，重复之处在此不作赘述。
例如，处理器92和存储器91之间可以直接或间接地互相通信。例如，处理器92和存储器91可以通过网络进行通信。网络可以包括无线网络、有线网络、和/或无线网络和有线网络的任意组合。处理器92和存储器91之间也可以通过系统总线实现相互通信，本公开对此不作限制。
例如，处理器92可以体现为各种适当的处理器、处理装置等，诸如中央处理器(CPU)、图形处理器(Graphics Processing Unit，GPU)、网络处理器(NP)等；还可以是数字信号处理器(DSP)、专用集成电路(ASIC)、现场可编程门阵列(FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。中央处理元(CPU)可以为X86或ARM架构等。例如，存储器91可以包括各种形式的计算机可读存储介质的任意组合，例如易失性存储器和/或非易失性存储器。存储器91例如可以包括系统存储器，系统存储器例如存储有操作系统、应用程序、引导装载程序(Boot Loader)、数据库以及其他程序等。在存储介质中还可以存储各种应用程序和各种数据等。
另外，根据本公开的一些实施例，根据本公开的各种操作/处理在通过软件和/或固件实现的情况下，可从存储介质或网络向具有专用硬件结构的计算机系统，例如图10所示的计算机系统(或电子设备)100安装构成该软件的程序，该计算机系统在安装有各种程序时，能够执行各种功能，包括诸如前文的功能等等。图10是示出根据本公开的实施例的中可采用的计算机系统的示例结构的框图。
在图10中，中央处理单元(CPU)1001根据只读存储器(ROM)1002中存储的程序或从存储部分1008加载到随机存取存储器(RAM)1003的程序执行各种处理。在RAM 1003中，也根据需要存储当CPU 1001执行各种处理等时所需的数据。中央处理单元仅仅是示例性的，其也可以是其它类型的处理器，诸如前文的各种处理器。ROM 1002、RAM 1003和存储部分1008可以是各种形式的计算机可读存储介质，如下文。需要注意的是，虽然图10中分别示出了ROM 1002、RAM 1003和存储装置 1008，但是它们中的一个或多个可以合并或者位于相同或不同的存储器或存储模块中。
CPU 1001、ROM 1002和RAM 1003经由总线1004彼此连接。输入/输出接口1005也连接到总线1004。
下述部件连接到输入/输出接口1005：输入部分1006，诸如触摸屏、触摸板、键盘、鼠标、图像传感器、麦克风、加速度计、陀螺仪等；输出部分1007，包括显示器，比如阴极射线管(CRT)、液晶显示器(LCD)，扬声器，振动器等；存储部分1008，包括硬盘，磁带等；和通信部分1009，包括网络接口卡比如LAN卡、调制解调器等。通信部分1009允许经由网络比如因特网执行通信处理。容易理解的是，虽然图10中示出计算机系统100中的各个装置或模块是通过总线1004来通信的，但它们也可以通过网络或其它方式进行通信，其中，网络可以包括无线网络、有线网络、和/或无线网络和有线网络的任意组合。
根据需要，驱动器1010也连接到输入/输出接口1005。可拆卸介质1011比如磁盘、光盘、磁光盘、半导体存储器等等根据需要被安装在驱动器1010上，使得从中读出的计算机程序根据需要被安装到存储部分1008中。
在通过软件实现上述系列处理的情况下，可以从网络比如因特网或存储介质比如可拆卸介质1011安装构成软件的程序。
根据本公开的实施例，上文参考流程图描述的过程可以被实现为计算机软件程序。例如，本公开的实施例包括一种计算机程序产品，其包括承载在计算机可读介质上的计算机程序，该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中，该计算机程序可以通过通信装置1009从网络上被下载和安装，或者从存储装置1008被安装，或者从ROM 1002被安装。在该计算机程序被CPU 1001执行时，执行本公开实施例的方法中限定的上述功能。
需要说明的是，在本公开的上下文中，计算机可读介质可以是有形的介质，其可以包含或存储以供指令执行系统、装置或设备使用或与指令执行系统、装置或设备结合地使用的程序。计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是，但不限于：电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器 件、磁存储器件、或者上述的任意合适的组合。在本公开中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。而在本公开中，计算机可读信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读信号介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：电线、光缆、RF(射频)等等，或者上述的任意合适的组合。
上述计算机可读介质可以是上述电子设备中所包含的；也可以是单独存在，而未装配入该电子设备中。
在一些实施例中，本公开还提供了一种计算机程序，包括：指令，指令当由处理器执行时使处理器执行上述任一个实施例的方法。例如，指令可以体现为计算机程序代码。
本公开还提供一种音频交互的处理系统，下面结合图11进行描述。
图11为本公开的音频交互的处理系统的一些实施例的结构图。如图11所示，该实施例的音频交互的处理系统11包括：本公开任意实施例的音频处理服务端70以及本公开任意实施例的客户端80。
在一些实施例中，系统11还包括文本处理服务端110，被配置为基于所述输入音频转换的待回复文本生成回复文本流，将所述回复文本流发送至所述音频处理服务端70和所述客户端80。
在一些实施例中，文本处理服务端110还被配置为接收所述音频处理服务端70发送的待回复文本，向所述客户端80发送所述回复音频流对应的消息标识。
在本公开的实施例中，可以以一种或多种程序设计语言或其组合来编写用于执行本公开的操作的计算机程序代码，上述程序设计语言包括但不限于面向对象的程序设计语言，诸如Java、Smalltalk、C++，还包括常规的过程式程序设计语言，诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络(，包括局域网(LAN)或广域网(WAN))连接到用户 计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。
附图中的流程图和框图，图示了按照本公开各种实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，该模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
描述于本公开实施例中所涉及到的模块、部件或单元可以通过软件的方式实现，也可以通过硬件的方式来实现。其中，模块、部件或单元的名称在某种情况下并不构成对该模块、部件或单元本身的限定。
本文中以上描述的功能可以至少部分地由一个或多个硬件逻辑部件来执行。例如，非限制性地，可以使用的示例性的硬件逻辑部件包括：现场可编程门阵列(FPGA)、专用集成电路(ASIC)、专用标准产品(ASSP)、片上系统(SOC)、复杂可编程逻辑设备(CPLD)等等。
根据本公开的一些实施例，提供的一种音频交互的处理方法，由音频处理服务端执行，包括：将客户端发送的输入音频，转换为待回复文本；将待回复文本发送至文本处理服务端，生成回复文本流；接收客户端对回复文本流对应的回复音频流的请求，其中，请求包括回复文本流对应的消息标识，消息标识由文本处理服务端发送至客户端；根据消息标识，从文本处理服务端获取回复文本流；根据回复文本流，生成回复音频流；将回复音频流发送至客户端，进行播放。
在一些实施例中，回复音频流包括多个音频数据包，多个音频数据包中每个音频数据包包括该音频数据包的序号标识，该处理方法还包括：接收客户端发送的重传请求，其中，重传请求包括客户端在网络中断前收到的最后一个音频数据包的序号标识；根据最后一个音频数据包的序号标识，确定是否存在待重传音频数据包，其中，待重传数据包为在最后一个音频数据包之后向客户端已发送的音频数据包；在存在待重传音频数据包的情况下，向客户端发送待重传音频数据包。
在一些实施例中，回复文本流包括多个文本数据包，根据消息标识，从文本处理 服务端获取回复文本流包括：根据消息标识，周期性从文本处理服务端拉取文本处理服务端生成的当前文本数据包；或者接收文本处理服务端推送的生成的当前文本数据包，其中，多个文本数据包中每个文本数据包包括消息标识。
在一些实施例中，该处理方法还包括：从文本处理服务端获取回复文本流之前的第一预设数量的文本，其中，根据回复文本流，生成回复音频流包括：根据当前文本数据包中的文本片段、当前文本数据包之前的一个或多个文本数据包中的文本片段、以及第一预设数量的文本，生成当前文本数据包对应的当前音频片段，并生成回复音频流中的当前音频数据包。
在一些实施例中，根据当前文本数据包中的文本片段、当前文本数据包之前的文本数据包中的文本片段、以及第一预设数量的文本，生成当前文本数据包对应的当前音频片段包括：根据当前文本数据包中的文本片段、当前文本数据包之前的一个或多个文本数据包中的文本片段，确定是否在当前文本数据包中的文本片段后进行断句；响应于在当前文本数据包中的文本片段后进行断句，根据当前文本数据包中的文本片段、当前文本数据包之前的文本数据包中的文本片段、以及第一预设数量的文本，生成当前文本数据包对应的当前音频片段。
在一些实施例中，将客户端发送的输入音频，转换为待回复文本包括：对输入音频进行语音识别，转换为待回复文本的文本流；该处理方法还包括：将待回复文本的文本流发送至客户端进行显示。
在一些实施例中，该处理方法还包括：从文本处理服务端获取待回复文本之前的第二预设数量的文本，其中，对输入音频进行语音识别，转换为待回复文本的文本流包括：根据输入音频和第二预设数量的文本，进行语音识别，将输入音频转换为待回复文本的文本流。
在一些实施例中，音频处理服务端与文本处理服务端通过远程过程调用方式进行数据传输。
在一些实施例中，音频处理器通过音频服务网关与客户端进行数据传输，响应于音频交互应用的启动，音频服务网关与客户端之间建立长连接。
根据本公开的另一些实施例，提供的一种音频交互的处理方法，由客户端执行，包括：将输入音频发送至音频处理服务端，转换为待回复文本，其中，待回复文本被发送至文本处理服务端生成回复文本流；接收文本处理服务端发送的回复文本流对应的消息标识；向音频处理服务端发送对回复文本流对应的回复音频流的请求，其中， 请求包括消息标识；接收音频处理服务端发送的回复音频流；对回复音频流进行播放。
在一些实施例中，将输入音频发送至音频处理服务端包括：将输入音频划分为多个音频片段；将多个音频片段中每个音频片段进行压缩后按顺序存入第一队列；从第一队列中取出第一个音频片段，发送至音频处理服务端进行语音识别；响应于接收到音频处理服务端发送的确认收到的信息，重复执行从第一队列中取出第一个音频片段，发送至音频处理服务端进行语音识别，直至第一队列为空。
在一些实施例中，将输入音频发送至音频处理服务端还包括：响应于等待预设时长未接收到音频处理服务端发送的确认收到的信息，重新发送最近发送的一个音频片段。
在一些实施例中，回复音频流包括多个音频数据包，多个音频数据包中每个音频数据包包括该音频数据包的序号标识，对回复音频流进行播放包括：根据接收的当前音频数据包的序号标识，将当前音频数据包存入第二队列；按照第二队列中各个音频数据包的顺序进行播放。
在一些实施例中，根据接收的当前音频数据包的序号标识，将当前音频数据包存入第二队列包括：根据接收的当前音频数据包的序号标识，确定第二队列中是否已存在与当前音频数据包的序号标识相同的音频数据包；在第二队列中已存在与当前音频数据包的序号标识相同的音频数据包，将当前音频数据包丢弃；在第二队列中不存在与当前音频数据包的序号标识相同的音频数据包，将当前音频数据包存入队列。
在一些实施例中，该处理方法还包括：响应于网络中断，向音频处理服务端发送重传请求，其中，重传请求包括客户端在网络中断前收到的最后一个音频数据包的序号标识；接收音频处理服务端发送的重传音频数据包，其中，重传数据包为音频处理服务端在最后一个音频数据包之后向客户端已发送过的数据包。
在一些实施例中，该处理方法还包括：接收音频处理服务端发送的待回复文本的文本流；将待回复文本的文本流进行显示；接收文本处理服务端发送的回复文本流，其中，回复文本流是文本处理服务端接收到音频处理服务端发送的待回复文本后生成的；将回复文本流进行显示。
在一些实施例中，该处理方法还包括：响应于音频交互应用的启动，与音频处理服务端对应的音频服务网关建立长连接，其中，客户端通过音频服务网关与音频处理器进行数据传输。
根据本公开的又一些实施例，提供了一种音频处理服务端，包括：转换模块，被 配置为将客户端发送的输入音频，转换为待回复文本；第一发送模块，被配置为将待回复文本发送至文本处理服务端，生成回复文本流；接收模块，被配置为客户端对回复文本流对应的回复音频流的请求，其中，请求包括回复文本流对应的消息标识，消息标识由文本处理服务端发送至客户端；获取模块，被配置为根据消息标识，从文本处理服务端获取回复文本流；生成模块，被配置为根据回复文本流，生成回复音频流；第二发送模块，被配置为将回复音频流发送至客户端，进行播放。
根据本公开的再一些实施例，提供一种客户端，包括：第一发送模块，被配置为将输入音频发送至音频处理服务端，转换为待回复文本，其中，待回复文本被发送至文本处理服务端生成回复文本流；第一接收模块，被配置为接收文本处理服务端发送的回复文本流对应的消息标识；第二发送模块，被配置为向音频处理服务端发送对回复文本流对应的回复音频流的请求，其中，请求包括消息标识；第二接收模块，被配置为接收音频处理服务端发送的回复音频流；播放模块，被配置为对回复音频流进行播放。
根据本公开的又一些实施例，提供一种音频交互的处理系统，包括：本公开中的任一实施例的音频处理服务端以及本公开中的任一实施例的客户端。
在一些实施例中，处理系统还包括：文本处理服务端，被配置为基于输入音频转换的待回复文本生成回复文本流，将回复文本流发送至音频处理服务端和客户端。
在一些实施例中，文本处理服务端还被配置为接收音频处理服务端发送的待回复文本，向客户端发送回复音频流对应的消息标识。
根据本公开的再一些实施例，提供一种电子设备，包括：处理器；以及耦接至处理器的存储器，用于存储指令，指令被处理器执行时，使处理器执行本公开中的任一实施例的音频交互的处理方法。
根据本公开的又一些实施例，提供一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时执行本公开中的任一实施例的音频交互的处理方法。
根据本公开的再一些实施例，提供一种计算机程序产品，包括：指令，指令被处理器执行时实现本公开中的任一实施例的音频交互的处理方法。
根据本公开的又一些实施例，提供一种计算机程序，包括：指令，指令被处理器执行时实现本公开中的任一实施例的音频交互的处理方法。
以上描述仅为本公开的一些实施例以及对所运用技术原理的说明。本领域技术人员应当理解，本公开中所涉及的公开范围，并不限于上述技术特征的特定组合而成的 技术方案，同时也应涵盖在不脱离上述公开构思的情况下，由上述技术特征或其等同特征进行任意组合而形成的其它技术方案。例如上述特征与本公开中公开的(但不限于)具有类似功能的技术特征进行互相替换而形成的技术方案。
在本文提供的描述中，阐述了许多特定细节。然而，理解的是，可以在没有这些特定细节的情况下实施本发明的实施例。在其他情况下，为了不模糊该描述的理解，没有对众所周知的方法、结构和技术进行详细展示。
此外，虽然采用特定次序描绘了各操作，但是这不应当理解为要求这些操作以所示出的特定次序或以顺序次序执行来执行。在一定环境下，多任务和并行处理可能是有利的。同样地，虽然在上面论述中包含了若干具体实现细节，但是这些不应当被解释为对本公开的范围的限制。在单独的实施例的上下文中描述的某些特征还可以组合地实现在单个实施例中。相反地，在单个实施例的上下文中描述的各种特征也可以单独地或以任何合适的子组合的方式实现在多个实施例中。
虽然已经通过示例对本公开的一些特定实施例进行了详细说明，但是本领域的技术人员应该理解，以上示例仅是为了进行说明，而不是为了限制本公开的范围。本领域的技术人员应该理解，可在不脱离本公开的范围和精神的情况下，对以上实施例进行修改。本公开的范围由所附权利要求来限定。

Claims (26)

  1. 一种音频交互的处理方法，由音频处理服务端执行，包括：
    将客户端发送的输入音频，转换为待回复文本；
    将所述待回复文本发送至文本处理服务端，生成回复文本流；
    接收所述客户端对所述回复文本流对应的回复音频流的请求，其中，所述请求包括所述回复文本流对应的消息标识，所述消息标识由所述文本处理服务端发送至所述客户端；
    根据所述消息标识，从所述文本处理服务端获取所述回复文本流；
    根据所述回复文本流，生成所述回复音频流；
    将所述回复音频流发送至所述客户端，进行播放。
  2. 根据权利要求1所述的处理方法，其中，所述回复音频流包括多个音频数据包，所述多个音频数据包中每个音频数据包包括该音频数据包的序号标识，所述处理方法还包括：
    接收所述客户端发送的重传请求，其中，所述重传请求包括所述客户端在网络中断前收到的最后一个音频数据包的序号标识；
    根据所述最后一个音频数据包的序号标识，确定是否存在待重传音频数据包，其中，所述待重传数据包为在所述最后一个音频数据包之后向所述客户端已发送的音频数据包；
    在存在所述待重传音频数据包的情况下，向所述客户端发送所述待重传音频数据包。
  3. 根据权利要求1或2所述的处理方法，其中，所述回复文本流包括多个文本数据包，所述根据所述消息标识，从所述文本处理服务端获取所述回复文本流包括：
    根据所述消息标识，周期性从所述文本处理服务端拉取所述文本处理服务端生成的当前文本数据包；或者
    接收所述文本处理服务端推送的生成的当前文本数据包，其中，所述多个文本数据包中每个文本数据包包括所述消息标识。
  4. 根据权利要求3所述的处理方法，还包括：
    从所述文本处理服务端获取所述回复文本流之前的第一预设数量的文本，
    其中，所述根据所述回复文本流，生成所述回复音频流包括：
    根据所述当前文本数据包中的文本片段、所述当前文本数据包之前的一个或多个文本数据包中的文本片段、以及所述第一预设数量的文本，生成所述当前文本数据包对应的当前音频片段，并生成所述回复音频流中的当前音频数据包。
  5. 根据权利要求4所述的处理方法，其中，所述根据所述当前文本数据包中的文本片段、所述当前文本数据包之前的文本数据包中的文本片段、以及所述第一预设数量的文本，生成所述当前文本数据包对应的当前音频片段包括：
    根据所述当前文本数据包中的文本片段、所述当前文本数据包之前的一个或多个文本数据包中的文本片段，确定是否在所述当前文本数据包中的文本片段后进行断句；
    响应于在所述当前文本数据包中的文本片段后进行断句，根据所述当前文本数据包中的文本片段、所述当前文本数据包之前的文本数据包中的文本片段、以及所述第一预设数量的文本，生成所述当前文本数据包对应的当前音频片段。
  6. 根据权利要求1-5任一项所述的处理方法，其中：
    所述将客户端发送的输入音频，转换为待回复文本包括：
    对所述输入音频进行语音识别，转换为所述待回复文本的文本流；
    所述处理方法还包括：
    将所述待回复文本的文本流发送至所述客户端进行显示。
  7. 根据权利要求6所述的处理方法，还包括：
    从所述文本处理服务端获取所述待回复文本之前的第二预设数量的文本，
    其中，所述对所述输入音频进行语音识别，转换为所述待回复文本的文本流包括：
    根据所述输入音频和所述第二预设数量的文本，进行语音识别，将所述输入音频转换为所述待回复文本的文本流。
  8. 根据权利要求1-7任一项所述的处理方法，其中，所述音频处理服务端与所述文本处理服务端通过远程过程调用方式进行数据传输。
  9. 根据权利要求1-8任一项所述的处理方法，其中，所述音频处理器通过音频服务网关与所述客户端进行数据传输，响应于音频交互应用的启动，所述音频服务网关与所述客户端之间建立长连接。
  10. 一种音频交互的处理方法，由客户端执行，包括：
    将输入音频发送至音频处理服务端，转换为待回复文本，其中，所述待回复文本被所述音频处理服务端发送至文本处理服务端生成回复文本流；
    接收所述文本处理服务端发送的所述回复文本流对应的消息标识；
    向所述音频处理服务端发送对所述回复文本流对应的回复音频流的请求，其中，所述请求包括所述消息标识；
    接收所述音频处理服务端发送的所述回复音频流，其中，所述回复音频流是所述音频处理服务端根据所述消息标识从所述文本处理服务端获取所述回复文本流后生成的；
    对所述回复音频流进行播放。
  11. 根据权利要求10所述的处理方法，其中，所述将输入音频发送至音频处理服务端包括：
    将所述输入音频划分为多个音频片段；
    将所述多个音频片段中每个音频片段进行压缩后按顺序存入第一队列；
    从所述第一队列中取出第一个音频片段，发送至所述音频处理服务端进行语音识别；
    响应于接收到所述音频处理服务端发送的确认收到的信息，重复执行所述从所述第一队列中取出第一个音频片段，发送至所述音频处理服务端进行语音识别，直至所述第一队列为空。
  12. 根据权利要求11所述的处理方法，其中，所述将输入音频发送至音频处理服务端还包括：
    响应于等待预设时长未接收到所述音频处理服务端发送的确认收到的信息，重新发送最近发送的一个音频片段。
  13. 根据权利要求11或12所述的处理方法，其中，所述回复音频流包括多个音频数据包，所述多个音频数据包中每个音频数据包包括该音频数据包的序号标识，所述对所述回复音频流进行播放包括：
    根据接收的当前音频数据包的序号标识，将所述当前音频数据包存入第二队列；
    按照所述第二队列中各个音频数据包的顺序进行播放。
  14. 根据权利要求13所述的处理方法，其中，所述根据接收的当前音频数据包的序号标识，将所述当前音频数据包存入第二队列包括：
    根据接收的当前音频数据包的序号标识，确定所述第二队列中是否已存在与所述当前音频数据包的序号标识相同的音频数据包；
    在所述第二队列中已存在与所述当前音频数据包的序号标识相同的音频数据包，将所述当前音频数据包丢弃；
    在所述第二队列中不存在与所述当前音频数据包的序号标识相同的音频数据包，将所述当前音频数据包存入队列。
  15. 根据权利要求10-14任一项所述的处理方法，还包括：
    响应于网络中断，向所述音频处理服务端发送重传请求，其中，所述重传请求包括所述客户端在网络中断前收到的最后一个音频数据包的序号标识；
    接收所述音频处理服务端发送的重传音频数据包，其中，所述重传数据包为所述音频处理服务端在所述最后一个音频数据包之后向所述客户端已发送过的数据包。
  16. 根据权利要求10-15任一项所述的处理方法，还包括：
    接收所述音频处理服务端发送的所述待回复文本的文本流；
    将所述待回复文本的文本流进行显示；
    接收所述文本处理服务端发送的所述回复文本流，其中，所述回复文本流是所述文本处理服务端接收到所述音频处理服务端发送的所述待回复文本后生成的；
    将所述回复文本流进行显示。
  17. 根据权利要求10-16任一项所述的处理方法，还包括：
    响应于音频交互应用的启动，与所述音频处理服务端对应的音频服务网关建立长连接，其中，所述客户端通过音频服务网关与所述音频处理器进行数据传输。
  18. 一种音频处理服务端，包括：
    转换模块，被配置为将客户端发送的输入音频，转换为待回复文本；
    第一发送模块，被配置为将所述待回复文本发送至文本处理服务端，生成回复文本流；
    接收模块，被配置为所述客户端对所述回复文本流对应的回复音频流的请求，其中，所述请求包括所述回复文本流对应的消息标识，所述消息标识由所述文本处理服务端发送至所述客户端；
    获取模块，被配置为根据所述消息标识，从所述文本处理服务端获取所述回复文本流；
    生成模块，被配置为根据所述回复文本流，生成所述回复音频流；
    第二发送模块，被配置为将所述回复音频流发送至所述客户端，进行播放。
  19. 一种客户端，包括：
    第一发送模块，被配置为将输入音频发送至音频处理服务端，转换为待回复文本，其中，所述待回复文本被发送至文本处理服务端生成回复文本流；
    第一接收模块，被配置为接收所述文本处理服务端发送的所述回复文本流对应的消息标识；
    第二发送模块，被配置为向所述音频处理服务端发送对所述回复文本流对应的回复音频流的请求，其中，所述请求包括所述消息标识；
    第二接收模块，被配置为接收所述音频处理服务端发送的所述回复音频流；
    播放模块，被配置为对所述回复音频流进行播放。
  20. 一种音频交互的处理系统，包括：权利要求18所述的音频处理服务端以及权利要求19所述的客户端。
  21. 根据权利要求20所述的处理系统，还包括：
    文本处理服务端，被配置为基于所述输入音频转换的待回复文本生成回复文本流，将所述回复文本流发送至所述音频处理服务端和所述客户端。
  22. 根据权利要求21所述的处理系统，其中，
    所述文本处理服务端还被配置为接收所述音频处理服务端发送的待回复文本，向所述客户端发送所述回复音频流对应的消息标识。
  23. 一种电子设备，包括：
    处理器；以及
    耦接至所述处理器的存储器，用于存储指令，所述指令被所述处理器执行时，使所述处理器执行如权利要求1-9任一项所述的音频交互的处理方法，或者权利要求10-17任一项所述的音频交互的处理方法。
  24. 一种计算机可读存储介质，其上存储有计算机程序，其中，该程序被处理器执行时实现权利要求1-9任一项所述的音频交互的处理方法，或者权利要求10-17任一项所述的音频交互的处理方法。
  25. 一种计算机程序产品，包括：指令，该指令被处理器执行时实现权利要求1-9任一项所述的音频交互的处理方法，或者权利要求10-17任一项所述的音频交互的处理方法。
  26. 一种计算机程序，包括：指令，该指令被处理器执行时实现权利要求1-9任一项所述的音频交互的处理方法，或者权利要求10-17任一项所述的音频交互的处理方法。
PCT/CN2024/086741 2024-04-09 2024-04-09 音频交互的处理方法、系统、服务端、客户端和电子设备 Pending WO2025213356A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/CN2024/086741 WO2025213356A1 (zh) 2024-04-09 2024-04-09 音频交互的处理方法、系统、服务端、客户端和电子设备
CN202480002169.2A CN119096295A (zh) 2024-04-09 2024-04-09 音频交互的处理方法、系统、服务端、客户端和电子设备

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2024/086741 WO2025213356A1 (zh) 2024-04-09 2024-04-09 音频交互的处理方法、系统、服务端、客户端和电子设备

Publications (1)

Publication Number Publication Date
WO2025213356A1 true WO2025213356A1 (zh) 2025-10-16

Family

ID=93663345

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2024/086741 Pending WO2025213356A1 (zh) 2024-04-09 2024-04-09 音频交互的处理方法、系统、服务端、客户端和电子设备

Country Status (2)

Country Link
CN (1) CN119096295A (zh)
WO (1) WO2025213356A1 (zh)

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103167431A (zh) * 2011-12-19 2013-06-19 北京新媒传信科技有限公司 一种增强语音短消息实时性的方法和系统
JP2015164269A (ja) * 2014-02-28 2015-09-10 株式会社東芝 自動音声応答装置および自動音声応答方法
US20200251111A1 (en) * 2019-02-06 2020-08-06 Microstrategy Incorporated Interactive interface for analytics
CN113392201A (zh) * 2021-06-18 2021-09-14 中国工商银行股份有限公司 信息交互方法、装置、电子设备、介质和程序产品
CN113905137A (zh) * 2021-11-11 2022-01-07 北京沃东天骏信息技术有限公司 一种通话方法及装置、存储介质
WO2022233065A1 (zh) * 2021-05-07 2022-11-10 华为技术有限公司 反馈方法及电子设备
CN117056486A (zh) * 2023-08-16 2023-11-14 上海墨百意信息科技有限公司 智能应答方法及装置、电子设备、存储介质
CN117253478A (zh) * 2023-08-16 2023-12-19 腾讯科技（深圳）有限公司 一种语音交互方法和相关装置

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103167431A (zh) * 2011-12-19 2013-06-19 北京新媒传信科技有限公司 一种增强语音短消息实时性的方法和系统
JP2015164269A (ja) * 2014-02-28 2015-09-10 株式会社東芝 自動音声応答装置および自動音声応答方法
US20200251111A1 (en) * 2019-02-06 2020-08-06 Microstrategy Incorporated Interactive interface for analytics
WO2022233065A1 (zh) * 2021-05-07 2022-11-10 华为技术有限公司 反馈方法及电子设备
CN113392201A (zh) * 2021-06-18 2021-09-14 中国工商银行股份有限公司 信息交互方法、装置、电子设备、介质和程序产品
CN113905137A (zh) * 2021-11-11 2022-01-07 北京沃东天骏信息技术有限公司 一种通话方法及装置、存储介质
CN117056486A (zh) * 2023-08-16 2023-11-14 上海墨百意信息科技有限公司 智能应答方法及装置、电子设备、存储介质
CN117253478A (zh) * 2023-08-16 2023-12-19 腾讯科技（深圳）有限公司 一种语音交互方法和相关装置

Also Published As

Publication number Publication date
CN119096295A (zh) 2024-12-06

Similar Documents

Publication Publication Date Title
US11792130B2 (en) Audio/video communication method, terminal, server, computer device, and storage medium
US12051420B2 (en) Information processing method, system, apparatus, electronic device and storage medium
US12106008B2 (en) Screen sharing display method and apparatus, device, and storage medium
US12445224B2 (en) Data transmission method and apparatus, device, and medium
CN112770312B (zh) 数据传输方法、装置、计算机可读介质及电子设备
CN103780972A (zh) 服务器、客户端装置及其控制方法
CN111163330A (zh) 直播视频的渲染方法、装置、系统、设备及存储介质
CN111752579A (zh) 蓝牙终端升级方法、装置、存储介质与电子设备
CN114356335A (zh) 数据处理方法、装置、设备及介质
WO2021103741A1 (zh) 内容处理方法、装置、计算机设备及存储介质
WO2025179742A1 (zh) 投屏方法、装置及系统
US9912617B2 (en) Method and apparatus for voice communication based on voice activity detection
WO2025213356A1 (zh) 音频交互的处理方法、系统、服务端、客户端和电子设备
WO2025213848A1 (zh) 音频处理方法、模型训练方法、装置、存储介质及电子设备
CN117857517A (zh) 数据处理方法、协议转换方法、装置、设备和存储介质
CN118540550A (zh) 一种基于浏览器的多路视频播放方法
CN112866622B (zh) 信息处理方法、装置、服务器、存储介质和系统
CN110955398B (zh) 一种二维码显示方法、装置、计算机设备及存储介质
CN114595080A (zh) 数据处理方法、装置、电子设备及计算机可读存储介质
US20250260846A1 (en) Method and apparatus for processing live audio and video stream, and device and medium
US20250184545A1 (en) Interactive live streaming method, apparatus, electronic device, and storage medium
CN115334331B (zh) 一种教学直播的通讯方法、设备及存储介质
US20250330506A1 (en) Live interaction establishment method and apparatus, and device, storage medium and program product
CN106851134A (zh) 传输图像数据的方法、装置及系统
CN117279041A (zh) 数据发送、数据接收方法、设备、介质及程序产品

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24934442

Country of ref document: EP

Kind code of ref document: A1