WO2021135713A1 - 一种文本转语音的处理方法、终端及服务器 - Google Patents
一种文本转语音的处理方法、终端及服务器 Download PDFInfo
- Publication number
- WO2021135713A1 WO2021135713A1 PCT/CN2020/130466 CN2020130466W WO2021135713A1 WO 2021135713 A1 WO2021135713 A1 WO 2021135713A1 CN 2020130466 W CN2020130466 W CN 2020130466W WO 2021135713 A1 WO2021135713 A1 WO 2021135713A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- audio data
- terminal
- server
- audio
- text
- Prior art date
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 12
- 238000000034 method Methods 0.000 claims abstract description 65
- 230000015654 memory Effects 0.000 claims description 49
- 238000004891 communication Methods 0.000 claims description 39
- 230000006835 compression Effects 0.000 claims description 34
- 238000007906 compression Methods 0.000 claims description 34
- 238000004590 computer program Methods 0.000 claims description 27
- 239000000872 buffer Substances 0.000 claims description 10
- 230000005236 sound signal Effects 0.000 claims description 6
- 238000004458 analytical method Methods 0.000 claims description 4
- 238000013473 artificial intelligence Methods 0.000 abstract 1
- 230000003993 interaction Effects 0.000 abstract 1
- 230000015572 biosynthetic process Effects 0.000 description 28
- 238000003786 synthesis reaction Methods 0.000 description 28
- 230000008569 process Effects 0.000 description 21
- 230000007246 mechanism Effects 0.000 description 20
- 238000012545 processing Methods 0.000 description 18
- 238000010586 diagram Methods 0.000 description 16
- 230000003068 static effect Effects 0.000 description 13
- 230000006870 function Effects 0.000 description 12
- 238000001308 synthesis method Methods 0.000 description 12
- 238000010295 mobile communication Methods 0.000 description 10
- 238000005516 engineering process Methods 0.000 description 9
- 238000005070 sampling Methods 0.000 description 8
- 238000013461 design Methods 0.000 description 7
- 238000007726 management method Methods 0.000 description 7
- 230000004044 response Effects 0.000 description 6
- 210000000988 bone and bone Anatomy 0.000 description 5
- 230000000694 effects Effects 0.000 description 5
- 229920001621 AMOLED Polymers 0.000 description 4
- 238000006243 chemical reaction Methods 0.000 description 4
- 230000001133 acceleration Effects 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 3
- 238000013500 data storage Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000012546 transfer Methods 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 230000003190 augmentative effect Effects 0.000 description 2
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 230000000977 initiatory effect Effects 0.000 description 2
- 239000010985 leather Substances 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 208000003028 Stuttering Diseases 0.000 description 1
- 230000003416 augmentation Effects 0.000 description 1
- 238000013529 biological neural network Methods 0.000 description 1
- 230000036772 blood pressure Effects 0.000 description 1
- 210000004556 brain Anatomy 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000019771 cognition Effects 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 210000002569 neuron Anatomy 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 239000002096 quantum dot Substances 0.000 description 1
- 230000005855 radiation Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000006641 stabilisation Effects 0.000 description 1
- 238000011105 stabilization Methods 0.000 description 1
- 238000001356 surgical procedure Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
- G10L13/02—Methods for producing synthetic speech; Speech synthesisers
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
- G10L13/02—Methods for producing synthetic speech; Speech synthesisers
- G10L13/04—Details of speech synthesis systems, e.g. synthesiser structure or memory management
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
- G10L13/02—Methods for producing synthetic speech; Speech synthesisers
- G10L13/04—Details of speech synthesis systems, e.g. synthesiser structure or memory management
- G10L13/047—Architecture of speech synthesisers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/16—Sound input; Sound output
- G06F3/165—Management of the audio stream, e.g. setting of volume, audio stream path
Definitions
- This application relates to the field of audio technology, and in particular to a text-to-speech processing method, terminal and server.
- TTS text-to-speech
- the TTS technology mainly converts each text into a corresponding voice, and then splices (or is called synthesis) the voice into an audio stream for playback.
- speech synthesis mainly includes offline synthesis and online synthesis.
- Offline synthesis can be implemented by the terminal itself, but the terminal needs to store voices corresponding to massive texts.
- Online synthesis can be server synthesis. Specifically, the terminal is connected to the server, and the text is sent to the server. After the server synthesizes the audio corresponding to the text into the audio stream, it sends the audio stream to the terminal for playback.
- the time when the server returns the audio stream cannot be controlled. For example, after the terminal is connected to the server, it is slow for the server to synthesize the audio stream, and it takes a long time to return the audio stream, which results in a longer delay for the terminal to play audio and affects user experience.
- the purpose of this application is to provide a text-to-speech processing method, terminal, and server to reduce the delay of the terminal playing voice.
- a text-to-speech processing method may be executed by a terminal or a chip in the terminal.
- the method includes: detecting an instruction for instructing text-to-speech; sending the text to a server; downloading the audio data converted from the text from the server; judging whether the first frame is downloaded within a preset time and can be played If the audio data that can be played in the first frame is downloaded within the preset time, continue to download the remaining audio data.
- the network quality of the terminal or server changes dynamically.
- the terminal download speed is faster, and when the network quality is poor, the download speed may be slower.
- the terminal sends the text to the server to request the server to implement text-to-speech, and then the terminal downloads voice data from the server. If the audio data that can be played in the first frame is downloaded within the preset time period, the remaining audio data will continue to be downloaded. Otherwise, the audio signal corresponding to the text is determined in the offline database, and the audio signal is synthesized into audio data. Therefore, the time delay of the terminal voice playback is shortened as much as possible to avoid the situation that the user cannot play the voice after the user enters the text-to-speech instruction.
- the audio data that can be played in the first frame may be audio data with a preset length that is downloaded for the first time.
- the audio data that can be played in the first frame may be data that includes at least one frame of complete audio downloaded for the first time.
- the terminal and the server agree on a compression strategy.
- the audio data compressed by the server each time has the same data length and a preset length; it can also be agreed that the audio data compressed by the server each time includes at least one complete frame.
- Audio the terminal downloads audio data of a preset length each time, and the audio data downloaded each time includes at least one frame of complete audio.
- the terminal downloads audio data of a preset length for the first time within a preset time period, or when it downloads data including at least one complete frame of audio for the first time, it can continue to download the remaining audio data, otherwise the offline synthesis method is used. Therefore, the time delay of the terminal voice playback is shortened as much as possible to avoid the situation that the user cannot play the voice after the user enters the text-to-speech instruction.
- the terminal may also: store the downloaded audio data in the buffer area; when it is determined that the memory occupation of the audio data in the buffer area reaches the threshold, play the audio data that can be played in the first frame .
- a certain amount of audio data stored in the buffer area can ensure that after the audio data that can be played in the first frame is played, the next frame of audio data is continued to be played, so as to ensure that the first frame is played without jamming.
- the terminal playing the audio data that can be played in the first frame includes: passing the audio data that can be played in the first frame to a parser for analysis to obtain decompressed audio data; The audio data is passed to the decoder for decoding to obtain audio data in a preset audio format; the audio data decoded by the decoder is passed to the player for playback.
- the above is the process of smoothly playing the audio data after the terminal downloads the first frame of audio data that can be played from the server. In practical applications, it may include more or less of the above processes. Not limited.
- the audio data downloaded from the server may be, but not limited to, in Opus format.
- the Opus format is a low-latency, high-fidelity speech encoding format, which has better fidelity than other encoding formats.
- a text-to-speech processing method is also provided.
- the method can be executed by a server or a processing chip in the server.
- the method includes: receiving a text-to-speech request sent by a terminal, the request carrying text; converting the text into audio data; compressing the audio data, and the audio data obtained by each compression has the same data length; Send the compressed voice data to the terminal respectively.
- the terminal and the server agree on a compression strategy.
- the server is scheduled to use a static compression mechanism, for example, the audio data compressed each time have the same data length.
- the data length of the audio data is a preset length; it can also be agreed that the audio data compressed by the server each time includes at least one frame of complete audio.
- the terminal downloads audio data of a preset length each time, and the audio data downloaded each time includes at least one frame of complete audio.
- the terminal downloads audio data of a preset length for the first time within a preset time period, or when it downloads data including at least one complete frame of audio for the first time, it can continue to download the remaining audio data, otherwise the offline synthesis method is used. Therefore, the time delay of the terminal voice playback is shortened as much as possible to avoid the situation that the user cannot play the voice after the user enters the text-to-speech instruction.
- the audio format of the audio data obtained by each compression may be but not limited to the Opus format.
- the Opus format is a low-latency, high-fidelity speech encoding format, which has better fidelity than other encoding formats.
- the terminal and the server agree on a compression strategy that can be implemented in multiple ways, including but not limited to the way 1 to way 3 as described below.
- the server can notify the terminal after determining the use of the static compression mechanism and the data length of the audio data compressed each time.
- the server sends first indication information to the terminal.
- the first indication information is used to indicate the static compression mechanism used by the server and the data length of the audio data compressed each time is a preset length, for example, 108 bytes.
- the terminal may require the server to use a static compression mechanism, and require the data length of the audio data compressed each time to be a preset length.
- the terminal sends second instruction information to the server, where the second instruction information is used to instruct the server to use a static compression mechanism and instruct the server to compress the audio data each time the data length is a preset length, for example, 108 bytes.
- Manner 3 When the server manufacturer generates the server, the server is set to use a static compression mechanism by default, and the data length of the audio data compressed each time is a preset length; for example, 108 bytes. When the terminal downloads an audio stream, the default download data length is an audio stream with a preset length.
- the above method can make the compression strategy agreed between the terminal and the server more flexible, and there are more options.
- the embodiments of the present application also provide a terminal, the terminal includes modules/units that execute the first aspect or any one of the possible design methods of the first aspect; these modules/units can be implemented by hardware, It can also be implemented by hardware executing corresponding software.
- the embodiments of the present application also provide a server.
- the server includes modules/units that execute the second aspect or any one of the possible design methods of the second aspect; these modules/units can be implemented by hardware, It can also be implemented by hardware executing corresponding software.
- a terminal including: a communication module for communicating with other devices; one or more processors; one or more memories; one or more computers are stored in the one or more memories
- a program, the one or more computer programs include instructions, and when the instructions are executed by the one or more processors, the terminal causes the terminal to execute the method steps provided in the above-mentioned first aspect.
- a server including: a communication module for communicating with other devices; one or more processors; one or more memories; one or more computers are stored in the one or more memories
- a program, the one or more computer programs include instructions, and when the instructions are executed by the one or more processors, the server executes the method steps provided in the above-mentioned second aspect.
- an embodiment of the present application provides a chip, which is coupled with a memory in a terminal, and is used to implement the first aspect of the embodiment of the present application and any possible design technical solution of the first aspect; an embodiment of the present application "Coupled" means that two components are directly or indirectly combined with each other.
- an embodiment of the present application provides a chip, which is coupled with a memory in a server, and is used to execute the second aspect of the embodiment of the present application and any possible design technical solution of the second aspect; an embodiment of the present application "Coupled" means that two components are directly or indirectly combined with each other.
- a communication system including the terminal provided in the third aspect or the fifth aspect and the server provided in the fourth or sixth aspect.
- a computer-readable storage medium stores a computer program.
- the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes The method provided in the first aspect above.
- a computer-readable storage medium stores a computer program.
- the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes As the method provided in the second aspect above.
- a program product stores a computer program, the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the operations provided in the above-mentioned first aspect.
- a program product stores a computer program, the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the program provided in the second aspect.
- Fig. 1 is a schematic diagram of a text-to-speech process in the prior art
- FIG. 2 is a schematic diagram of a system architecture provided by an embodiment of the application.
- FIG. 3 is a schematic structural diagram of a terminal provided by an embodiment of this application.
- FIG. 4 is a schematic diagram of a text-to-speech process provided by an embodiment of this application.
- FIG. 5 is a schematic diagram of a dynamic compression mechanism provided by an embodiment of this application.
- FIG. 6 is a schematic diagram of another process of text-to-speech provided by an embodiment of this application.
- FIG. 7 is a schematic diagram of a cache queue provided by an embodiment of the application.
- FIG. 8 is a schematic structural diagram of a device provided by an embodiment of this application.
- FIG. 9 is a functional block diagram of another device provided by an embodiment of this application.
- FIG. 10 is a schematic structural diagram of another device provided by an embodiment of this application.
- Terminals such as mobile phones, tablet computers, notebook computers, handheld computers, mobile internet devices (MID), wearable devices (for example, watches, bracelets, smart helmets, etc.), virtual reality (virtual reality) , VR) equipment, augmented reality (augmented reality, AR) equipment, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (personal digital assistant, PDA), industrial control (industrial control) Wireless terminals, wireless terminals in self-driving (self-driving), wireless terminals in remote medical surgery, wireless terminals in smart grid (smart grid), wireless terminals in transportation safety, and smart Wireless terminals in the city (smart city), wireless terminals in the smart home (smart home), etc.
- the audio processing method provided in the embodiments of the present application may be applicable to any of the above-mentioned terminals.
- the device used to implement the terminal-side functions of the embodiments of the present application may be a terminal, or a device capable of supporting the realization of the function, such as a chip system, and the device may be installed in the terminal.
- the server is used to realize the conversion from text to speech and synthesize the speech into an audio stream. It can be referred to as "TTS server”.
- the device used to implement the server-side function of the embodiment of the present application may be a server, or a device capable of supporting the realization of the function, such as a chip system, and the device may be installed in the server.
- the current online audio synthesis can be performed by the server.
- the terminal is connected to the server, and the text is sent to the server.
- the server synthesizes the audio corresponding to the text into the audio stream, it sends the audio stream to the terminal for playback.
- Fig. 1 shows a schematic diagram of a process in which a terminal implements text-to-speech in the prior art. After the terminal is connected to the server (for example, the TTS server), audio playback is realized through at least the following five steps.
- the server for example, the TTS server
- the terminal downloads the audio stream from the server. 2. Cache the downloaded audio stream to the buffer area.
- the threshold of the buffer area is 10M, that is, when the audio stream buffered in the buffer area is greater than 10M, the audio stream enters the next step.
- the parser parses the audio stream, such as decompressing the audio stream and determining the number of frames included in the audio stream. 4.
- the decoder decodes the parsed audio stream to obtain an audio stream in pulse code modulation (PCM) format. 5.
- PCM pulse code modulation
- the time when the server returns the audio stream cannot be controlled. For example, after the terminal successfully connects to the server, it takes a long time for the server to synthesize the audio stream, and it takes a long time to return the audio stream, which results in a longer overall audio playback delay and affects user experience.
- an embodiment of the present application provides a text-to-speech processing method.
- the terminal requests the server to synthesize the voice online, and if the audio data that can be played in the first frame is returned within the preset time period, the online synthesis method can continue to be used. If the preset duration does not return the audio data that can be played in the first frame, use offline synthesis to reduce the audio playback delay.
- Fig. 2 shows a functional block diagram of an example of a system architecture provided by an embodiment of the present application.
- the terminal includes an application side and a TTS service system.
- the application side includes multiple applications (application, app for short), for example, voice assistant, WeChat, mail, and so on.
- the TTS service system service is used to implement text-to-speech.
- the TTS service system can choose offline synthesis mode or online synthesis mode. For example, the TTS service system preferentially selects the online synthesis mode, that is, the TTS service system sends a connection request to the server side for requesting to establish a connection with the server side. If you can successfully connect to the server within the preset time, use the online synthesis method; if you cannot connect to the server within the preset time, use the offline synthesis method.
- the TTS service system can determine the size of the text content. If the text content is small, the offline synthesis method can be used; if the text content is large, the online synthesis method can be used.
- the TTS service system preferentially uses the offline synthesis method, and if the voice corresponding to a certain text does not exist in the offline database in the terminal, the online synthesis method is used.
- the TTS service system After the TTS service system is successfully connected to the server, it can send text to the server. After the server determines the speech synthesis audio stream corresponding to the text, it sends the audio stream to the TTS service system. The TTS service system sends the audio stream to a player (such as a speaker) for playback. Or, after the TTS service system determines all or part of the text corresponding to the voice in the offline database, it sends the determined voice to the server, and the server performs voice synthesis.
- a player such as a speaker
- the terminal may switch to the offline synthesis mode. For example, in the process of online synthesis of the server, when the network is poor and the connection is interrupted, the terminal can send an error code indicating that the network is interrupted. After receiving the error code, the terminal can switch to offline synthesis.
- one is the primary server and the other is the standby server.
- the terminal successfully connects to the main server within the first preset time period, online synthesis is performed through the main server. If the main server is not successfully connected within the first preset time period, then the backup server is connected, and if the backup server is successfully connected within the second preset time period, online synthesis is performed through the backup server. If the connection to the backup server fails within the second preset time period, the offline synthesis method is used again.
- an application on the application side requests TTS service system text-to-speech can include multiple situations.
- the voice assistant receives a voice signal input by the user, and the voice signal is "open WeChat.”
- the TTS service system determines the response signal corresponding to the sound signal. For example, the response signal is "WeChat has been opened for you", and the TTS service system synthesizes the text of "WeChat has been opened for you” into speech.
- the Toutiao application detects the instruction for text-to-speech, it requests the TTS service system for text-to-speech from the TTS service system to realize the voice broadcast news.
- the application on the application side may log in to the TTS service system before requesting the TTS service system to perform text-to-speech.
- the application sends login information to the TTS service system, and the login information may include information such as the name of the application, the account currently logged in by the application, and the password.
- the TTS service system authenticates the application based on the login information, for example, determines whether the login information is consistent with the pre-stored login information. If they are consistent, agree to log in to the TTS service system and perform text-to-speech; if they are inconsistent, refuse to log in to TTS service system. If the authentication fails, the application can switch to offline synthesis to ensure the smooth execution of speech synthesis.
- FIG. 3 shows a schematic structural diagram of an example of a terminal provided in an embodiment of the present application.
- the terminal 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, and a battery 142, Antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone interface 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, A display screen 194, a subscriber identification module (SIM) card interface 195, and so on.
- SIM subscriber identification module
- the processor 110 may include one or more processing units.
- the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), and an image signal processor. (image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural-network processing unit (NPU) Wait.
- the different processing units may be independent devices or integrated in one or more processors.
- the controller may be the nerve center and command center of the terminal 100. The controller can generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching instructions and executing instructions.
- a memory may also be provided in the processor 110 to store instructions and data. In some embodiments, the memory in the processor 110 is a cache memory.
- the memory can store instructions or data that have just been used or recycled by the processor 110. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. Repeated accesses are avoided, the waiting time of the processor 110 is reduced, and the efficiency of the system is improved.
- the USB interface 130 is an interface that complies with the USB standard specification, and specifically may be a Mini USB interface, a Micro USB interface, a USB Type C interface, and so on.
- the USB interface 130 can be used to connect a charger to charge the terminal 100, and can also be used to transfer data between the terminal 100 and peripheral devices.
- the charging management module 140 is used to receive charging input from the charger.
- the power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110.
- the power management module 141 receives input from the battery 142 and/or the charge management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160.
- the wireless communication function of the terminal 100 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor.
- the antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals.
- Each antenna in the terminal 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization.
- antenna 1 can be multiplexed as a diversity antenna of a wireless local area network.
- the antenna can be used in combination with a tuning switch.
- the mobile communication module 150 may provide a wireless communication solution including 2G/3G/4G/5G and the like applied to the terminal 100.
- the mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), and the like.
- the mobile communication module 150 can receive electromagnetic waves by the antenna 1, and perform processing such as filtering, amplifying and transmitting the received electromagnetic waves to the modem processor for demodulation.
- the mobile communication module 150 can also amplify the signal modulated by the modem processor, and convert it into electromagnetic waves for radiation via the antenna 1.
- at least part of the functional modules of the mobile communication module 150 may be provided in the processor 110.
- at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 may be provided in the same device.
- the wireless communication module 160 can provide applications on the terminal 100 including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), and global navigation satellite systems. (global navigation satellite system, GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (infrared, IR) and other wireless communication solutions.
- WLAN wireless local area networks
- BT Bluetooth
- GNSS global navigation satellite system
- FM frequency modulation
- NFC near field communication
- IR infrared technology
- the wireless communication module 160 may be one or more devices integrating at least one communication processing module.
- the wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110.
- the wireless communication module 160 may also receive a signal to be sent from the processor 110, perform frequency modulation, amplify, and convert it into electromagnetic waves to radiate through the antenna 2.
- the antenna 1 of the terminal 100 is coupled with the mobile communication module 150, and the antenna 2 is coupled with the wireless communication module 160, so that the terminal 100 can communicate with the network and other devices through wireless communication technology.
- the wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), broadband Code division multiple access (wideband code division multiple access, WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC , FM, and/or IR technology, etc.
- the GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), and quasi-zenith satellite system (quasi). -zenith satellite system, QZSS) and/or satellite-based augmentation systems (SBAS).
- GPS global positioning system
- GLONASS global navigation satellite system
- BDS Beidou navigation satellite system
- QZSS quasi-zenith satellite system
- SBAS satellite-based augmentation systems
- the display screen 194 is used to display the display interface of the application and the like.
- the display screen 194 includes a display panel.
- the display panel can use liquid crystal display (LCD), organic light-emitting diode (OLED), active matrix organic light-emitting diode or active-matrix organic light-emitting diode (active-matrix organic light-emitting diode).
- LCD liquid crystal display
- OLED organic light-emitting diode
- active-matrix organic light-emitting diode active-matrix organic light-emitting diode
- AMOLED flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diode (QLED), etc.
- the terminal 100 may include one or N display screens 194, and N is a positive integer greater than one.
- the terminal 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.
- the ISP is used to process the data fed back from the camera 193. For example, when taking a picture, the shutter is opened, the light is transmitted to the photosensitive element of the camera through the lens, the light signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP for processing and is converted into an image visible to the naked eye.
- ISP can also optimize the image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene.
- the ISP may be provided in the camera 193.
- the camera 193 is used to capture still images or videos.
- the object generates an optical image through the lens and is projected to the photosensitive element.
- the photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor.
- CMOS complementary metal-oxide-semiconductor
- the photosensitive element converts the optical signal into an electrical signal, and then transfers the electrical signal to the ISP to convert it into a digital image signal.
- ISP outputs digital image signals to DSP for processing.
- DSP converts digital image signals into standard RGB, YUV and other formats of image signals.
- the terminal 100 may include one or N cameras 193, and N is a positive integer greater than one.
- the one or N cameras 193 may include at least one wide-angle camera, and the wide-angle camera is used to collect images with a large field of view.
- Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the terminal 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.
- Video codecs are used to compress or decompress digital video.
- the terminal 100 may support one or more video codecs.
- the terminal 100 can play or record videos in multiple encoding formats, for example: moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, and so on.
- MPEG moving picture experts group
- NPU is a neural-network (NN) computing processor.
- NN neural-network
- applications such as intelligent cognition of the terminal 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, and so on.
- the internal memory 121 may be used to store computer executable program code, where the executable program code includes instructions.
- the processor 110 executes various functional applications and data processing of the terminal 100 by running instructions stored in the internal memory 121.
- the internal memory 121 may include a storage program area and a storage data area.
- the storage program area can store an operating system, and software codes of at least one application program (for example, an iQiyi application, a WeChat application, etc.).
- the data storage area can store data (such as images, videos, etc.) generated during the use of the terminal 100.
- the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), and the like.
- UFS universal flash storage
- the external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal.
- the external memory card communicates with the processor 110 through the external memory interface 120 to realize the data storage function. For example, save pictures, videos and other files in an external memory card.
- the terminal 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. For example, music playback, recording, etc.
- the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, and the environment Light sensor 180L, bone conduction sensor 180M, etc.
- the pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal.
- the pressure sensor 180A may be provided on the display screen 194.
- the gyro sensor 180B may be used to determine the movement posture of the terminal 100.
- the angular velocity of the terminal 100 around three axes ie, x, y, and z axes
- the gyro sensor 180B can be used for image stabilization.
- the air pressure sensor 180C is used to measure air pressure.
- the terminal 100 calculates the altitude based on the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.
- the magnetic sensor 180D includes a Hall sensor.
- the terminal 100 may use the magnetic sensor 180D to detect the opening and closing of the flip holster.
- the terminal 100 when the terminal 100 is a flip phone, the terminal 100 can detect the opening and closing of the flip according to the magnetic sensor 180D. Furthermore, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, features such as automatic unlocking of the flip cover are set.
- the acceleration sensor 180E can detect the magnitude of the acceleration of the terminal 100 in various directions (generally three axes). When the terminal 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to recognize the posture of the terminal 100, applied to applications such as horizontal and vertical screen switching, and pedometer.
- the terminal 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the terminal 100 may use the distance sensor 180F to measure the distance to achieve fast focusing.
- the proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector such as a photodiode.
- the light emitting diode may be an infrared light emitting diode.
- the terminal 100 emits infrared light to the outside through the light emitting diode.
- the terminal 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the terminal 100.
- the terminal 100 may determine that there is no object near the terminal 100.
- the terminal 100 can use the proximity light sensor 180G to detect that the user holds the terminal 100 close to the ear to talk, so as to automatically turn off the screen to save power.
- the proximity light sensor 180G can also be used in leather case mode, and the pocket mode will automatically unlock and lock the screen.
- the ambient light sensor 180L is used to sense the brightness of the ambient light.
- the terminal 100 can adaptively adjust the brightness of the display screen 194 according to the perceived brightness of the ambient light.
- the ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures.
- the ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the terminal 100 is in a pocket to prevent accidental touch.
- the fingerprint sensor 180H is used to collect fingerprints.
- the terminal 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photographs, fingerprint answering calls, and so on.
- the temperature sensor 180J is used to detect temperature.
- the terminal 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold value, the terminal 100 executes to reduce the performance of the processor located near the temperature sensor 180J, so as to reduce power consumption and implement thermal protection.
- the terminal 100 when the temperature is lower than another threshold, the terminal 100 heats the battery 142 to avoid abnormal shutdown of the terminal 100 due to low temperature.
- the terminal 100 boosts the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.
- Touch sensor 180K also called “touch panel”.
- the touch sensor 180K may be disposed on the display screen 194, and the touch screen is composed of the touch sensor 180K and the display screen 194, which is also called a “touch screen”.
- the touch sensor 180K is used to detect touch operations acting on or near it.
- the touch sensor can pass the detected touch operation to the application processor to determine the type of touch event.
- the visual output related to the touch operation can be provided through the display screen 194.
- the touch sensor 180K may also be disposed on the surface of the terminal 100, which is different from the position of the display screen 194.
- the bone conduction sensor 180M can acquire vibration signals.
- the bone conduction sensor 180M can obtain the vibration signal of the vibrating bone mass of the human voice.
- the bone conduction sensor 180M can also contact the human pulse and receive the blood pressure pulse signal.
- the button 190 includes a power-on button, a volume button, and so on.
- the button 190 may be a mechanical button. It can also be a touch button.
- the terminal 100 may receive key input, and generate key signal input related to user settings and function control of the terminal 100.
- the motor 191 can generate vibration prompts.
- the motor 191 can be used for incoming call vibration notification, and can also be used for touch vibration feedback. For example, touch operations applied to different applications (such as photographing, audio playback, etc.) can correspond to different vibration feedback effects.
- the touch vibration feedback effect can also support customization.
- the indicator 192 may be an indicator light, which may be used to indicate the charging status, power change, or to indicate messages, missed calls, notifications, and so on.
- the SIM card interface 195 is used to connect to the SIM card. The SIM card can be inserted into the SIM card interface 195 or pulled out from the SIM card interface 195 to achieve contact and separation with the terminal 100.
- the mobile phone may also include more or less components than those shown in the figure, or combine certain components, or split certain components, or Different component arrangements.
- the combination/connection relationship between the components in FIG. 2 can also be adjusted and modified.
- one or more refers to one, two, or more than two; "and/or” describes the association relationship of associated objects, indicating that there may be three relationships; for example, A and/or B can mean the situation where A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural.
- the character “Yes” generally indicates that the associated objects before and after are an "or" relationship.
- references described in this specification to "one embodiment” or “some embodiments”, etc. mean that one or more embodiments of the present application include a specific feature, structure, or characteristic described in combination with the embodiment. Therefore, the sentences “in one embodiment”, “in some embodiments”, “in some other embodiments”, “in some other embodiments”, etc. appearing in different places in this specification are not necessarily All refer to the same embodiment, but mean “one or more but not all embodiments” unless it is specifically emphasized otherwise.
- the terms “including”, “including”, “having” and their variations all mean “including but not limited to”, unless otherwise specifically emphasized.
- the term “when” or “after” can be interpreted as meaning “if" or “after” or “in response to determining" or “in response to detecting ".
- the phrase “when determining" or “if detected (statement or event)” can be interpreted as meaning “if determined" or “in response to determining" or “when detected (Condition or event stated)” or “in response to detection of (condition or event stated)”.
- relationship terms such as first and second are used to distinguish one entity from another entity, and any actual relationship and order between these entities are not limited.
- FIG. 4 shows a schematic flowchart of an example of a text-to-speech processing method provided by an embodiment of the present application. As shown in Figure 4, the process of the method includes:
- the terminal and the server agree on a compression strategy.
- the compression strategy includes a static compression mechanism (constents bitrate, CBR).
- CBR includes a static compression mechanism (constents bitrate, CBR).
- CBR includes a static compression mechanism
- the so-called static compression mechanism means that the data length of each compressed audio data is the same; the data length can be understood as the number of bytes and bits occupied by the data, such as The data length is 108 bytes.
- the compression strategy may also include that at least one frame of complete audio is included in each compressed audio data.
- a complete frame of audio means that each compressed audio data includes a frame header and frame tail corresponding to a complete frame of audio.
- the compression strategy may further include that the data length of the audio data compressed each time is a preset length.
- the preset length may be determined in advance, for example, determined by the following formula:
- Data length ((number of samples per frame/8)*bit rate)/sampling rate
- the number of samples per frame is the data sampled from one frame of complete audio; the sampling rate, that is, the sampling frequency, can be understood as the data sampled per second. Generally, the higher the sampling frequency, the more realistic the restoration of the sound.
- the bit rate can be understood as the number of bits transmitted per second, also known as the data transmission rate.
- the bit rate/sampling rate may be a preset value. Or, it is related to the sound quality, for example, the sound quality is better, the bit rate/sampling rate is higher, the sound quality is poor, and the bit rate/sampling rate is lower. Therefore, the terminal may determine the bit rate/sampling rate based on the sound quality of the player, and then determine the preset length through the above formula, that is, the size of the data length of each compressed audio data.
- the server can notify the terminal after determining the use of the static compression mechanism and the data length of the audio data compressed each time.
- the server sends first indication information to the terminal.
- the first indication information is used to indicate the static compression mechanism used by the server and the data length of the audio data compressed each time is a preset length, for example, 108 bytes.
- the terminal may require the server to use a static compression mechanism, and require the data length of the audio data compressed each time to be a preset length.
- the terminal sends second instruction information to the server, where the second instruction information is used to instruct the server to use a static compression mechanism and instruct the server to compress the audio data each time the data length is a preset length, for example, 108 bytes.
- Manner 3 When the server manufacturer generates the server, the server is set to use a static compression mechanism by default, and the data length of the audio data compressed each time is a preset length; for example, 108 bytes. When the terminal downloads an audio stream, the default download data length is an audio stream with a preset length.
- the terminal plays the audio stream, it is played in units of frames.
- the frame header or called the dock
- the frame tail or called the code tail
- the terminal parses the frame header and frame end of each frame in the audio stream, and then plays it frame by frame.
- the server and the terminal agree that the data length of the audio data compressed each time is the same and is a preset length, and the data length includes at least one frame of complete audio, that is, includes the frame header and frame tail corresponding to at least one frame of audio. Therefore, when the terminal downloads the audio stream, it can be downloaded with the agreed data length.
- the agreed data length is 108 bytes
- the data length of the audio data compressed by the server each time is 108 bytes
- the terminal downloads 108 bytes of audio data each time.
- the data includes at least one complete frame of audio, that is, includes the frame header and frame tail corresponding to at least one frame of audio. Therefore, the 108-byte audio data downloaded by the terminal each time is an audio stream that can be played, because the 108-byte audio data includes at least one frame of complete audio.
- the server uses a dynamic compression mechanism (variable bitrate, VBR).
- VBR variable bitrate
- the so-called dynamic compression mechanism means that the data length of the audio data compressed each time is different.
- the audio data compressed by the server includes 2 compressed packets, the length of the audio data of one compressed packet is 100K, and the length of the audio data of the other compressed packet is 60K. Since the length of dynamic compressed audio data is not fixed, the audio data contained in a compressed packet may be less than one frame, for example, it does not include a complete frame of audio, for example, it only includes the frame header of one frame of audio but does not include the frame tail.
- Figure 5 shows a schematic diagram of the dynamic compression mechanism.
- the audio data compressed by the server includes two compressed packages
- the first compressed package includes 100K
- the second compressed package includes 60K.
- the first 40K in the first compressed package and the second compressed package belong to 1 frame of complete audio. Then, when the terminal only receives the first compressed package, it cannot be played.
- each time the terminal downloads the compressed package it needs to analyze whether the compressed package includes at least one frame of complete audio through a parser, and if so, the audio data is played. Before the parser parses the compressed package, it cannot be determined whether the compressed package can be played.
- the terminal and the server agree that the data length of the audio data compressed each time (that is, the compression ratio obtained each time) is the same, and is a preset length (for example, 108 bytes), and the data length includes At least one frame of complete audio, that is, includes the frame header and frame tail corresponding to at least one frame of audio. Therefore, the terminal downloads audio data of a preset length (for example, 108 bytes) each time, so the audio data downloaded each time can be played, because the audio data downloaded each time includes at least one frame of complete audio. In other words, each time the terminal downloads the compressed audio data, it does not need to wait for the parser to analyze it, and it can also determine that the audio data can be played.
- a preset length for example, 108 bytes
- the terminal detects a text-to-speech request.
- the terminal may include a TTS service system.
- Different applications in the terminal can call the TTS service system to request text-to-speech conversion.
- different applications can include voice assistants, WeChat, and so on. Therefore, the aforementioned request for initiating text-to-speech can be a request for an application to call the TTS service system.
- the voice assistant as an example, the user sends out a voice signal of "Hello, Huawei, help me open Weibo". After detecting the sound signal, the terminal calls the TTS service system to realize the conversion of text to speech through the TTS service system, and then outputs the voice message of "opened Weibo".
- the terminal starts a timer and starts timing.
- the terminal can start a timer.
- the terminal sends the text to be converted to the server.
- 403 and 402 can be executed at the same time or at different time, and the execution order of the two is not limited.
- connection between the terminal and the server for example, an IP connection, a TCP connection, etc., is not limited in the embodiment of the present application.
- the server performs voice conversion based on the received text, and compresses the converted audio data based on an agreed compression strategy.
- the format of the audio data may be Opus format, for example.
- the terminal and the server agree that the data length of the audio data compressed each time is the same, and the data length includes at least one frame of complete audio, that is, includes the frame header and frame tail corresponding to at least one frame of audio. For example, if the agreed data length is 108 bytes, the audio data compressed by the server each time includes 108 bytes, and the 108 bytes include at least one frame of complete audio.
- the terminal downloads audio data.
- the terminal downloads 108 bytes of audio data each time, and the 108 bytes include at least one frame of complete audio.
- the terminal judges whether the audio data that can be played in the first frame is downloaded before the timer reaches the threshold; if not, execute 407, and if yes, execute 408.
- the threshold may be set in advance, for example, the terminal is set by default before leaving the factory, or it may be user-defined, for example, it may be 1.5s, which is not limited in the embodiment of the present application.
- the terminal and the server agree that the data length of the audio data compressed each time is the same and is a preset length, and the data length includes at least one frame of complete audio, that is, includes the frame header and frame corresponding to at least one frame of audio tail. Therefore, the terminal downloads audio data of a preset length that can be played each time.
- the audio data that can be played in the first frame includes audio data of a preset length (for example, 108 bytes) downloaded by the terminal for the first time.
- the audio data that can be played in the first frame is the data that is downloaded for the first time and includes at least one frame of complete audio
- the audio data that can be played in the first frame is the data that is downloaded for the first time and includes the frame header corresponding to at least one frame of complete audio. Data at the end of the frame.
- the terminal determining whether to download the audio data that can be played in the first frame may include: determining whether 108 bytes are downloaded. Suppose that before the timer threshold (for example, 1.5s) is reached, 108 bytes are successfully downloaded, that is, the first frame of audio data that can be played is downloaded. Therefore, the terminal can continue to download the remaining audio data. Assuming that before 1.5s, 108 bytes were not successfully downloaded, the online synthesis method was abandoned and the offline synthesis method was used.
- the timer threshold for example, 1.5s
- the terminal adopts an offline synthesis mechanism.
- the offline synthesis mechanism can be used.
- the offline synthesis mechanism may be that the terminal determines the speech corresponding to the text from the offline database, and then synthesizes the speech into audio data.
- the offline database can be stored in the terminal in advance.
- the terminal continues to download audio data.
- the terminal downloads audio data that can be played in the first frame before the timer threshold is reached, it can continue to download audio data. At this time, the terminal can choose to turn off the timer.
- the terminal after the terminal sends the text to the server, if the first frame of audio data that can be played is not downloaded within the preset time period, it is converted to offline synthesis to reduce the audio playback delay of the terminal.
- step 408 409-410 as shown in FIG. 6 may be included.
- the terminal buffers the audio data.
- the terminal When determining that the buffered audio data reaches the upper limit, the terminal transmits the downloaded audio data that can be played in the first frame to the parser for analysis.
- the upper limit is, for example, 1M, 5M, etc., which is not limited in the embodiment of the present application.
- Fig. 7 shows a schematic diagram of a cache queue provided by an embodiment of the present application.
- the upper limit (or called the threshold) of the buffer queue is 1M.
- the audio data for example, the 108-byte audio data downloaded for the first time
- the terminal continues to download the audio data and store it in the buffer queue, which helps the audio playback without stuttering.
- the upper limit of the cache in the embodiment of the present application is set relatively low (less than 10M in the prior art), which can ensure that the audio data is transmitted to the parser for analysis as soon as possible.
- the process of parsing the audio data by the parser for example, decompressing the audio data (the audio data downloaded from the server is the compressed audio data), parsing the frame header, frame end, etc. in the audio data for playback
- the device recognizes the frame head and frame end.
- the decoder decodes the parsed audio data to obtain audio data in a preset audio format. It can be understood that the decoding process of the audio data by the decoder can be understood as a process of formatting the audio data, for example, converting the format of the audio data into a playable audio format, for example, the PCM format.
- the terminal transmits the decoded audio data to the player for playback.
- a device 800 which may be the terminal described above, and the device 800 may have an audio output interface 805, such as a speaker.
- the electronic device may further include a communication interface 801: one or more processors 802, one or more memories 803; one or more computer programs 804; each of the above-mentioned devices may be connected through one or more communication buses 806.
- the electronic device may also include other devices such as a display screen.
- one or more computer programs 804 are stored in the aforementioned memory 803 and are configured to be executed by the one or more processors 802.
- the one or more computer programs 804 include instructions, and the aforementioned instructions can be used to execute such as Figure 4 corresponds to each step of the terminal in the embodiment.
- the electronic device shown in FIG. 8 may be the mobile phone shown in FIG. 4.
- the one or more processors 802 may be the processor 110
- the audio output interface 805 may be a speaker
- the one or more memories 803 may be the internal memory 121.
- the device 900 may be the server described above, or a device in the server, such as a chip system.
- the chip system can be composed of chips, or it can include chips and other discrete devices.
- the device 900 includes a processing unit 901 and a communication unit 902.
- the communication unit 902 is configured to receive a text-to-speech request sent by the terminal, and the request carries text;
- the processing unit 901 is configured to convert the text into audio data; compress the audio data, and obtain the result after each compression
- the data length of the audio data is the same;
- the communication unit 902 is also used to send compressed voice data to the terminal respectively.
- the data length of the audio data compressed by the processing unit 901 each time may be a preset length.
- the audio data compressed by the processing unit 901 each time may include at least one frame of complete audio.
- the communication unit 902 may also notify the terminal of the data length of the audio data compressed by the server each time.
- the audio format of the audio data compressed by the processing unit 901 each time may be, but is not limited to, the Opus format.
- the division of modules in the embodiments of this application is illustrative, and it is only a logical function division. In actual implementation, there may be other division methods.
- the functional modules in the various embodiments of this application can be integrated into one process. In the device, it can also exist alone physically, or two or more modules can be integrated into one module.
- the above-mentioned integrated modules can be implemented in the form of hardware or software functional modules.
- the device 1000 may be the server mentioned above, or a device in the server, such as a chip system.
- the chip system can be composed of chips, or it can include chips and other discrete devices.
- the apparatus 1000 may include a communication interface 1001: one or more processors 1002, one or more memories 1003; one or more computer programs 1004; each of the foregoing devices may be connected through one or more communication buses 1006.
- the communication interface 1001 is used to communicate with other devices through a transmission medium.
- One or more computer programs 1004 are stored in the aforementioned memory 1003 and are configured to be executed by the one or more processors 1002.
- the one or more computer programs 1004 include instructions, and the aforementioned instructions can be used for execution as shown in FIG. 4 Corresponds to the steps of the server in the embodiment.
- the communication interface 1001 may be a transceiver, a circuit, a bus, a module, or other types of communication interfaces.
- the processor 1002 may be a general-purpose processor, a digital signal processor, an application specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the The disclosed methods, steps and logic block diagrams.
- the general-purpose processor may be a microprocessor or any conventional processor or the like.
- the steps of the method disclosed in combination with the embodiments of the present application may be directly embodied as execution and completion by a hardware processor, or execution and completion by a combination of hardware and software modules in the processor.
- the memory 1003 may be a non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SSD), etc., and may also be a volatile memory (volatile memory), such as random access memory (random access memory). -access memory, RAM).
- the memory is any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and that can be accessed by a computer, but is not limited to this.
- the memory in the embodiment of the present application may also be a circuit or any other device capable of realizing a storage function.
- the memory 1003 is coupled with the processor 1002.
- the coupling in the embodiments of the present application is an interval coupling or a communication connection between devices, units or modules, which can be electrical, mechanical or other forms, and is used for information exchange between devices, units or modules.
- the memory 1003 may also be located outside the apparatus 1000.
- the processor 1002 can operate in cooperation with the memory 1003.
- the processor 1002 can execute program instructions stored in the memory 1003.
- At least one of the at least one memory 1003 may also be included in the processor 1002.
- the embodiment of the present application does not limit the connection medium between the communication interface 1001, the processor 1002, and the memory 1003.
- the memory 1003, the processor 1002, and the communication interface 1001 may be connected by a bus, and the bus may be divided into an address bus, a data bus, and a control bus.
- the apparatus in the embodiment shown in FIG. 9 may be implemented by the apparatus 1000 shown in FIG. 10.
- the processing unit 901 may be implemented by the processor 1002
- the communication unit 902 may be implemented by the communication interface 1001.
- the computer program product includes one or more computer instructions.
- the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.
- the computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.
- the computer instructions may be transmitted from a website, computer, server, or data center.
- the computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or a data center integrated with one or more available media.
- the usable medium may be a magnetic medium (for example, a floppy disk, a hard disk, and a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)).
Landscapes
- Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- Acoustics & Sound (AREA)
- Multimedia (AREA)
- Telephone Function (AREA)
- Telephonic Communication Services (AREA)
Abstract
一种文本转语音的处理方法、终端及服务器,用以降低终端播放语音的时延。方法及装置可应用于人工智能、人机交互等领域。其中,终端检测到用于指示文本转语音的指令;将文本发送服务器;从服务器下载由文本转换而成的音频数据;判断预设时长内是否下载到首帧可以播放的音频数据;若预设时长内下载到首帧可以播放的音频数据,继续下载剩余的音频数据。
Description
相关申请的交叉引用
本申请要求在2019年12月30日提交中国专利局、申请号为201911404905.4、申请名称为“一种文本转语音的处理方法、终端及服务器”的中国专利申请的优先权,其全部内容通过引用结合在本申请中。
本申请涉及音频技术领域,尤其涉及一种文本转语音的处理方法、终端及服务器。
目前,文本到语音(text to speech,TTS)技术成为人机对话的重要技术,实现了“让机器说话”的目的。TTS技术的用途很广,包括电子邮件的阅读、语音助手、语音提示等等。
TTS技术主要是将每个文字转换成对应的语音,然后将语音拼接(或称为合成)为音频流来播放。目前,语音合成主要包括离线合成和在线合成。离线合成可以由终端自行实现,但是需要终端内存储海量文字对应的语音。在线合成可以是服务器合成。具体来说,终端连接到服务器,将文本发送服务器,服务器将文本对应的音频合成音频流之后,将音频流发送给终端播放。
但是,在线合成过程中,服务器返回音频流的时间无法控制。例如,终端连接服务器之后,服务器合成音频流较慢,花费较长时间才返回音频流,导致终端播放音频的时延较长,影响用户体验。
发明内容
本申请的目的在于提供了一种文本转语音的处理方法、终端及服务器,用以降低终端播放语音的时延。
第一方面,提供一种文本转语音的处理方法,该方法可以由终端执行或者由终端中的芯片执行。该方法包括:检测到用于指示文本转语音的指令;将所述文本发送服务器;从所述服务器下载由所述文本转换而成的音频数据;判断预设时长内是否下载到首帧可以播放的音频数据;若预设时长内下载到首帧可以播放的音频数据,继续下载剩余的音频数据。
应理解,终端或服务器的网络质量动态变化,网络质量较好时,终端下载速度较快,网络质量较差时,可能下载速度较慢。在本申请实施例中,终端将文本发送服务器,以请求服务器实现文本转语音之后,终端从服务器下载语音数据。如果预设时长内下载了首帧可以播放的音频数据,则继续下载剩余的音频数据。否则在离线数据库中确定所述文本对应的音频信号,并将所述音频信号合成音频数据。因此,尽可能的缩短终端语音播放的时延,以避免用户输入文本转语音的指示之后,迟迟无法播放出语音的情况。
其中,所述首帧可以播放的音频数据可以为首次下载的包括预设长度的音频数据。或者,所述首帧可以播放的音频数据可以为首次下载的包括至少一帧完整音频的数据。
在本申请实施例中,终端与服务器约定压缩策略,例如预定服务器每次压缩的音频数据的数据长度相同,且为预设长度;还可以约定服务器每次压缩的音频数据中包括至少一帧完整的音频。这样的话,终端每次下载预设长度的音频数据,而且每次下载的音频数据都包括至少一帧完整的音频。也就是说,终端在预设时长内首次下载到预设长度的音频数据,或者,首次下载到包括至少一帧完整音频的数据时,可以继续下载剩余音频数据,否则使用离线合成方式。因此,尽可能的缩短终端语音播放的时延,以避免用户输入文本转语音的指示之后,迟迟无法播放出语音的情况。
在一种可能的设计中,终端还可以:将下载的音频数据存储到缓存区域中;确定所述缓存区域内的音频数据的内存占用量达到门限时,播放所述首帧可以播放的音频数据。
应理解,缓存区域中存储的一定数量的音频数据,可以保证播放首帧可以播放的音频数据之后,继续播放下一帧音频数据,以确保首帧播放完毕,不卡顿。
在一些实施例中,终端播放所述首帧可以播放的音频数据,包括:将所述首帧可以播放的音频数据传递给解析器进行解析,得到解压后的音频数据;将经过解析器解析后的音频数据传递给解码器进行解码,得到预设音频格式的音频数据;将经过解码器解码后的音频数据传递给播放器播放。
需要说明的是,上述是终端从服务器下载了首帧可以播放的音频数据之后,将音频数据顺利播放的流程,在实际应用中,可以包括可以上述更多或很少的流程,本申请实施例不作限定。
在一些实施例中,从服务器下载的音频数据可以但不限于为Opus格式。Opus格式是低延时、高保真的语音编码格式,相对于其他的编码格式保真性较好。
第二方面,还提供一种文本转语音的处理方法,该方法可以由服务器执行或者由服务器中的处理芯片执行。该方法包括:接收终端发送的本文转语音的请求,所述请求中携带文本;将所述文本转换为音频数据;对所述音频数据进行压缩,每次压缩得到的音频数据的数据长度相同;向所述终端分别发送压缩后的语音数据。
在本申请实施例中,终端与服务器约定压缩策略,例如预定服务器使用静态压缩机制,例如每次压缩的音频数据的数据长度相同。还可以约定音频数据的数据长度为预设长度;还可以约定服务器每次压缩的音频数据中包括至少一帧完整的音频。这样的话,终端每次下载预设长度的音频数据,而且每次下载的音频数据都包括至少一帧完整的音频。也就是说,终端在预设时长内首次下载到预设长度的音频数据,或者,首次下载到包括至少一帧完整音频的数据时,可以继续下载剩余音频数据,否则使用离线合成方式。因此,尽可能的缩短终端语音播放的时延,以避免用户输入文本转语音的指示之后,迟迟无法播放出语音的情况。
在一种可能的设计中,所述每次压缩得到的音频数据的音频格式为可以但不限于Opus格式。Opus格式是低延时、高保真的语音编码格式,相对于其他的编码格式保真性较好。
其中,终端与服务器约定压缩策略可以有多种实现方式,包括但不限定于如下所述的方式1-方式3。
方式1,服务器确定使用静态压缩机制以及每次压缩的音频数据的数据长度之后,可以通知终端。例如,服务器向终端发送第一指示信息,例如,第一指示信息用于指示服务器使用的静态压缩机制且每次压缩的音频数据的数据长度为预设长度,例如为108字节。
方式2,终端可以要求服务器使用静态压缩机制,并要求每次压缩的音频数据的数据 长度为预设长度。例如,终端向服务器发送第二指示信息,该第二指示信息用于指示服务器使用静态压缩机制,并指示服务器每次压缩的音频数据的数据长度为预设长度,例如为108字节。
方式3,服务器生产商生成服务器时,默认设置服务器使用静态压缩机制,且每次压缩的音频数据的数据长度为预设长度;例如为108字节。终端下载音频流时,默认下载数据长度为预设长度的音频流。
上述方式可以使得终端与服务器约定的压缩策略更加灵活,可选择方式更多。
第三方面,本申请实施例还提供了一种终端,所述终端包括执行第一方面或者第一方面的任意一种可能的设计的方法的模块/单元;这些模块/单元可以通过硬件实现,也可以通过硬件执行相应的软件实现。
第四方面,本申请实施例还提供了一种服务器,所述服务器包括执行第二方面或者第二方面的任意一种可能的设计的方法的模块/单元;这些模块/单元可以通过硬件实现,也可以通过硬件执行相应的软件实现。
第五方面,还提供一种终端,包括:通信模块,用于与其他设备通信;一个或多个处理器;一个或多个存储器;所述一个或多个存储器中存储有一个或多个计算机程序,所述一个或多个计算机程序包括指令,当所述指令被所述一个或多个处理器执行时,使得所述终端执行如上述第一方面提供的方法步骤。
第六方面,还提供一种服务器,包括:通信模块,用于与其他设备通信;一个或多个处理器;一个或多个存储器;所述一个或多个存储器中存储有一个或多个计算机程序,所述一个或多个计算机程序包括指令,当所述指令被所述一个或多个处理器执行时,使得所述服务器执行如上述第二方面提供的方法步骤。
第七方面,本申请实施例提供一种芯片,所述芯片与终端中的存储器耦合,用于执行本申请实施例第一方面及其第一方面任一可能设计的技术方案;本申请实施例中“耦合”是指两个部件彼此直接或间接地结合。
第八方面,本申请实施例提供一种芯片,所述芯片与服务器中的存储器耦合,用于执行本申请实施例第二方面及其第二方面任一可能设计的技术方案;本申请实施例中“耦合”是指两个部件彼此直接或间接地结合。
第九方面,还提供一种通信系统,包括如上述第三方面或第五方面提供的终端以及如上述第四方面或第六方面提供的服务器。
第十方面,还提供一种计算机可读存储介质,所述计算机可读存储介质存储有计算机程序,所述计算机程序包括程序指令,当所述程序指令被计算机执行时,使所述计算机执行如上述第一方面提供的方法。
第十一方面,还提供一种计算机可读存储介质,所述计算机可读存储介质存储有计算机程序,所述计算机程序包括程序指令,当所述程序指令被计算机执行时,使所述计算机执行如上述第二方面提供的方法。
第十二方面,还提供一种程序产品,所述程序产品存储有计算机程序,所述计算机程序包括程序指令,当所述程序指令被计算机执行时,使所述计算机执行如上述第一方面提供的方法。
第十三方面,还提供一种程序产品,所述程序产品存储有计算机程序,所述计算机程序包括程序指令,当所述程序指令被计算机执行时,使所述计算机执行如上述第二方面提 供的方法。
上述第三方面至第十三方面中每一方面可以达到的技术效果,具体请参照上述第一方面和第二方面中的相应设计的效果描述,这里不再重复赘述。
图1为现有技术中文本转语音的流程示意图;
图2为本申请一实施例提供的系统架构的示意图;
图3为本申请一实施例提供的终端的结构示意图;
图4为本申请一实施例提供的文本转语音的流程示意图;
图5为本申请一实施例提供的动态压缩机制的示意图;
图6为本申请一实施例提供的文本转语音的另一流程示意图;
图7为本申请一实施例提供的缓存队列的示意图;
图8为本申请一实施例提供的一种装置的结构示意图;
图9为本申请一实施例提供的另一种装置的功能框图;
图10为本申请一实施例提供的又一种装置的结构示意图。
下面将结合本申请以下实施例中的附图,对本申请实施例中的技术方案进行详尽描述。
以下,先对本申请实施例中涉及的部分用语进行解释说明,以便于本领域技术人员容易理解。
终端,诸如手机(mobile phone)、平板电脑、笔记本电脑、掌上电脑、移动互联网设备(mobile internet device,MID)、可穿戴设备(例如,手表、手环、智能头盔等),虚拟现实(virtual reality,VR)设备、增强现实(augmented reality,AR)设备、超级移动个人计算机(ultra-mobile personal computer,UMPC)、上网本、个人数字助理(personal digital assistant,PDA),工业控制(industrial control)中的无线终端、无人驾驶(self driving)中的无线终端、远程手术(remote medical surgery)中的无线终端、智能电网(smart grid)中的无线终端、运输安全(transportation safety)中的无线终端、智慧城市(smart city)中的无线终端、智慧家庭(smart home)中的无线终端等。本申请实施例提供的音频处理方法可以适用于上述任一种终端。
本申请实施例中,用于实现本申请实施例终端侧功能的装置可以是终端,也可以是能够支持实现该功能的装置,例如芯片系统,该装置可以被安装在终端中。
服务器,用于实现从文本到语音的转换,并将语音合成音频流,可以简称“TTS服务器”。
本申请实施例中,用于实现本申请实施例服务器侧功能的装置可以是服务器,也可以是能够支持实现该功能的装置,例如芯片系统,该装置可以被安装在服务器中。
目前的音频在线合成可以由服务器执行。具体来说,终端连接到服务器,将文本发送服务器,服务器将文本对应的音频合成音频流之后,将音频流发送给终端播放。
图1示出了现有技术中终端实现文本转语音的过程的示意图。终端与服务器(例如,TTS服务器)连接之后,至少经过如下5个环节实现音频播放。
1、终端从服务器下载音频流。2、将下载的音频流缓存到缓存区。缓存区的阈值为10M,即当缓存区缓存的音频流大于10M时,音频流进入下一个环节。3,解析器对音频流进行解析,例如对音频流解压、确定音频流包括的帧数等。4、解码器对经过解析后的音频流进行解码,得到脉冲编码调制(pulse code modulation,PCM)格式的音频流。5,PCM格式的音频流传递到播放器进行播放。
上述过程中,服务器返回音频流的时间无法控制。例如,终端成功连接服务器之后,服务器合成音频流较慢,花费较长时间才返回音频流,导致整体音频播放时延较长,影响用户体验。
鉴于此,本申请实施例提供一种文本转语音的处理方法。终端请求服务器在线合成语音,如果预设时长内返回了首帧可以播放的音频数据,可以继续使用在线合成方式。如果预设时长未返回首帧可以播放的音频数据,使用离线合成方式,以降低音频播放时延。
图2示出了本申请实施例提供的系统架构的一种示例的功能框图。如图2所示,终端包括应用侧和TTS服务系统。其中,应用侧包括多个应用(application,简称app),例如,语音助手、微信、邮件等。TTS服务系统服务用于实现文本转语音。
应用侧中的一个应用请求TTS服务系统文本转语音。TTS服务系统可以选择离线合成方式或在线合成方式。例如,TTS服务系统优先选择在线合成方式,即TTS服务系统向服务器侧发送连接请求,用于请求与服务器侧建立连接。若预设时长内能够与服务器成功连接,则使用在线合成方式,若预设时长内无法与服务器连接,则使用离线合成方式。
再例如,TTS服务系统可以判断文本内容的大小,若文本内容较少,可以使用离线合成方式;若文本内容较大,可以使用在线合成方式。
再例如,TTS服务系统优先使用离线合成方式,若终端内的离线数据库中不存在某个文本对应的语音时,再使用在线合成方式。
TTS服务系统与服务器连接成功之后,可以向服务器发送文本。服务器确定文本对应的语音合成音频流之后,向TTS服务系统发送音频流。TTS服务系统将音频流发送给播放器(例如扬声器)播放。或者,TTS服务系统在离线数据库中确定出了全部或部分文本对应的语音后,将确定出的语音发送给服务器,由服务器进行语音合成。
在一些实施例中,服务器在线合成的过程中,若服务器网络较差,终端可以转离线合成方式。例如,服务器在线合成的过程中,网络差而导致连接中断时,可以向终端发送用于指示网络中断的错误码。终端接收到该错误码后,可以转离线合成。
在另一些实施例中,用于语音合成的服务器可以不止一个。以两个为例,一个是主服务器,另一个是备用服务器。终端在第一预设时长内成功连接到主服务器时,通过主服务器进行在线合成。若第一预设时长内未成功连接到主服务器,则连接备用服务器,若第二预设时长内成功连接到备用服务器,通过该备用服务器进行在线合成。若第二预设时长内连接备用服务器失败,则再使用离线合成方式。
其中,应用侧中的一个应用请求TTS服务系统文本转语音可以包括多种情况。例如,以语音助手为例,语音助手收到用户输入的声音信号,该声音信号为“打开微信”。TTS服务系统确定该声音信号对应的响应信号,例如响应信号为“已为您打开微信”,TTS服务系统将“已为您打开微信”的文本合成为语音。再例如,以今日头条为例。今日头条应用检测到用于指示文本转语音的指令后,向TTS服务系统请求TTS服务系统文本转语音,实现语音播放新闻。
在另一些实施例中,应用侧中的应用请求TTS服务系统进行文本转语音之前,可以登录到TTS服务系统。例如,应用向TTS服务系统发送登录信息,所述登录信息可以包括应用的名称、应用当前登录的账号、密码等信息。TTS服务系统基于所述登录信息对应用进行鉴权,例如判断登录信息与预存的登录信息是否一致,若一致,则同意登录到TTS服务系统,并进行文本转语音;若不一致,则拒绝登录到TTS服务系统。若鉴权失败,应用可以转离线合成,以确保顺利执行语音合成。
图3示出了本申请实施例提供的终端的一种示例的结构示意图。如图3所示,终端100可以包括处理器110,外部存储器接口120,内部存储器121,通用串行总线(universal serial bus,USB)接口130,充电管理模块140,电源管理模块141,电池142,天线1,天线2,移动通信模块150,无线通信模块160,音频模块170,扬声器170A,受话器170B,麦克风170C,耳机接口170D,传感器模块180,按键190,马达191,指示器192,摄像头193,显示屏194,以及用户标识模块(subscriber identification module,SIM)卡接口195等。
处理器110可以包括一个或多个处理单元,例如:处理器110可以包括应用处理器(application processor,AP),调制解调处理器,图形处理器(graphics processing unit,GPU),图像信号处理器(image signal processor,ISP),控制器,存储器,视频编解码器,数字信号处理器(digital signal processor,DSP),基带处理器,和/或神经网络处理器(neural-network processing unit,NPU)等。其中,不同的处理单元可以是独立的器件,也可以集成在一个或多个处理器中。其中,控制器可以是终端100的神经中枢和指挥中心。控制器可以根据指令操作码和时序信号,产生操作控制信号,完成取指令和执行指令的控制。处理器110中还可以设置存储器,用于存储指令和数据。在一些实施例中,处理器110中的存储器为高速缓冲存储器。该存储器可以保存处理器110刚用过或循环使用的指令或数据。如果处理器110需要再次使用该指令或数据,可从所述存储器中直接调用。避免了重复存取,减少了处理器110的等待时间,因而提高了系统的效率。
USB接口130是符合USB标准规范的接口,具体可以是Mini USB接口,Micro USB接口,USB Type C接口等。USB接口130可以用于连接充电器为终端100充电,也可以用于终端100与外围设备之间传输数据。充电管理模块140用于从充电器接收充电输入。电源管理模块141用于连接电池142,充电管理模块140与处理器110。电源管理模块141接收电池142和/或充电管理模块140的输入,为处理器110,内部存储器121,外部存储器,显示屏194,摄像头193,和无线通信模块160等供电。
终端100的无线通信功能可以通过天线1,天线2,移动通信模块150,无线通信模块160,调制解调处理器以及基带处理器等实现。天线1和天线2用于发射和接收电磁波信号。终端100中的每个天线可用于覆盖单个或多个通信频带。不同的天线还可以复用,以提高天线的利用率。例如:可以将天线1复用为无线局域网的分集天线。在另外一些实施例中,天线可以和调谐开关结合使用。
移动通信模块150可以提供应用在终端100上的包括2G/3G/4G/5G等无线通信的解决方案。移动通信模块150可以包括至少一个滤波器,开关,功率放大器,低噪声放大器(low noise amplifier,LNA)等。移动通信模块150可以由天线1接收电磁波,并对接收的电磁波进行滤波,放大等处理,传送至调制解调处理器进行解调。移动通信模块150还可以对经调制解调处理器调制后的信号放大,经天线1转为电磁波辐射出去。在一些实施例中,移动通信模块150的至少部分功能模块可以被设置于处理器110中。在一些实施例中,移动 通信模块150的至少部分功能模块可以与处理器110的至少部分模块被设置在同一个器件中。
无线通信模块160可以提供应用在终端100上的包括无线局域网(wireless local area networks,WLAN)(如无线保真(wireless fidelity,Wi-Fi)网络),蓝牙(bluetooth,BT),全球导航卫星系统(global navigation satellite system,GNSS),调频(frequency modulation,FM),近距离无线通信技术(near field communication,NFC),红外技术(infrared,IR)等无线通信的解决方案。无线通信模块160可以是集成至少一个通信处理模块的一个或多个器件。无线通信模块160经由天线2接收电磁波,将电磁波信号调频以及滤波处理,将处理后的信号发送到处理器110。无线通信模块160还可以从处理器110接收待发送的信号,对其进行调频,放大,经天线2转为电磁波辐射出去。
在一些实施例中,终端100的天线1和移动通信模块150耦合,天线2和无线通信模块160耦合,使得终端100可以通过无线通信技术与网络以及其他设备通信。所述无线通信技术可以包括全球移动通讯系统(global system for mobile communications,GSM),通用分组无线服务(general packet radio service,GPRS),码分多址接入(code division multiple access,CDMA),宽带码分多址(wideband code division multiple access,WCDMA),时分码分多址(time-division code division multiple access,TD-SCDMA),长期演进(long term evolution,LTE),BT,GNSS,WLAN,NFC,FM,和/或IR技术等。所述GNSS可以包括全球卫星定位系统(global positioning systemo,GPS),全球导航卫星系统(global navigation satellite system,GLONASS),北斗卫星导航系统(beidou navigation satellite system,BDS),准天顶卫星系统(quasi-zenith satellite system,QZSS)和/或星基增强系统(satellite based augmentation systems,SBAS)。
显示屏194用于显示应用的显示界面等。显示屏194包括显示面板。显示面板可以采用液晶显示屏(liquid crystal display,LCD),有机发光二极管(organic light-emitting diode,OLED),有源矩阵有机发光二极体或主动矩阵有机发光二极体(active-matrix organic light emitting diode的,AMOLED),柔性发光二极管(flex light-emitting diode,FLED),Miniled,MicroLed,Micro-oLed,量子点发光二极管(quantum dot light emitting diodes,QLED)等。在一些实施例中,终端100可以包括1个或N个显示屏194,N为大于1的正整数。
终端100可以通过ISP,摄像头193,视频编解码器,GPU,显示屏194以及应用处理器等实现拍摄功能。
ISP用于处理摄像头193反馈的数据。例如,拍照时,打开快门,光线通过镜头被传递到摄像头感光元件上,光信号转换为电信号,摄像头感光元件将所述电信号传递给ISP处理,转化为肉眼可见的图像。ISP还可以对图像的噪点,亮度,肤色进行算法优化。ISP还可以对拍摄场景的曝光,色温等参数优化。在一些实施例中,ISP可以设置在摄像头193中。
摄像头193用于捕获静态图像或视频。物体通过镜头生成光学图像投射到感光元件。感光元件可以是电荷耦合器件(charge coupled device,CCD)或互补金属氧化物半导体(complementary metal-oxide-semiconductor,CMOS)光电晶体管。感光元件把光信号转换成电信号,之后将电信号传递给ISP转换成数字图像信号。ISP将数字图像信号输出到DSP加工处理。DSP将数字图像信号转换成标准的RGB,YUV等格式的图像信号。在一些实施例中,终端100可以包括1个或N个摄像头193,N为大于1的正整数。在一些示例中, 所述1个或N个摄像头193中可以包括至少一个广角摄像头,广角摄像头用于采集具有大视场角的图像。
数字信号处理器用于处理数字信号,除了可以处理数字图像信号,还可以处理其他数字信号。例如,当终端100在频点选择时,数字信号处理器用于对频点能量进行傅里叶变换等。
视频编解码器用于对数字视频压缩或解压缩。终端100可以支持一种或多种视频编解码器。这样,终端100可以播放或录制多种编码格式的视频,例如:动态图像专家组(moving picture experts group,MPEG)1,MPEG2,MPEG3,MPEG4等。
NPU为神经网络(neural-network,NN)计算处理器,通过借鉴生物神经网络结构,例如借鉴人脑神经元之间传递模式,对输入信息快速处理,还可以不断的自学习。通过NPU可以实现终端100的智能认知等应用,例如:图像识别,人脸识别,语音识别,文本理解等。
内部存储器121可以用于存储计算机可执行程序代码,所述可执行程序代码包括指令。处理器110通过运行存储在内部存储器121的指令,从而执行终端100的各种功能应用以及数据处理。内部存储器121可以包括存储程序区和存储数据区。其中,存储程序区可存储操作系统,以及至少一个应用程序(例如爱奇艺应用,微信应用等)的软件代码等。存储数据区可存储终端100使用过程中所产生的数据(例如图像、视频等)等。此外,内部存储器121可以包括高速随机存取存储器,还可以包括非易失性存储器,例如至少一个磁盘存储器件,闪存器件,通用闪存存储器(universal flash storage,UFS)等。
外部存储器接口120可以用于连接外部存储卡,例如Micro SD卡,实现扩展终端的存储能力。外部存储卡通过外部存储器接口120与处理器110通信,实现数据存储功能。例如将图片,视频等文件保存在外部存储卡中。
终端100可以通过音频模块170,扬声器170A,受话器170B,麦克风170C,耳机接口170D,以及应用处理器等实现音频功能。例如音乐播放,录音等。
其中,传感器模块180可以包括压力传感器180A,陀螺仪传感器180B,气压传感器180C,磁传感器180D,加速度传感器180E,距离传感器180F,接近光传感器180G,指纹传感器180H,温度传感器180J,触摸传感器180K,环境光传感器180L,骨传导传感器180M等。
压力传感器180A用于感受压力信号,可以将压力信号转换成电信号。在一些实施例中,压力传感器180A可以设置于显示屏194。陀螺仪传感器180B可以用于确定终端100的运动姿态。在一些实施例中,可以通过陀螺仪传感器180B确定终端100围绕三个轴(即,x,y和z轴)的角速度。
陀螺仪传感器180B可以用于拍摄防抖。气压传感器180C用于测量气压。在一些实施例中,终端100通过气压传感器180C测得的气压值计算海拔高度,辅助定位和导航。磁传感器180D包括霍尔传感器。终端100可以利用磁传感器180D检测翻盖皮套的开合。在一些实施例中,当终端100是翻盖机时,终端100可以根据磁传感器180D检测翻盖的开合。进而根据检测到的皮套的开合状态或翻盖的开合状态,设置翻盖自动解锁等特性。加速度传感器180E可检测终端100在各个方向上(一般为三轴)加速度的大小。当终端100静止时可检测出重力的大小及方向。还可以用于识别终端100姿态,应用于横竖屏切换,计步器等应用。
距离传感器180F,用于测量距离。终端100可以通过红外或激光测量距离。在一些实施例中,拍摄场景,终端100可以利用距离传感器180F测距以实现快速对焦。接近光传感器180G可以包括例如发光二极管(LED)和光检测器,例如光电二极管。发光二极管可以是红外发光二极管。终端100通过发光二极管向外发射红外光。终端100使用光电二极管检测来自附近物体的红外反射光。当检测到充分的反射光时,可以确定终端100附近有物体。当检测到不充分的反射光时,终端100可以确定终端100附近没有物体。终端100可以利用接近光传感器180G检测用户手持终端100贴近耳朵通话,以便自动熄灭屏幕达到省电的目的。接近光传感器180G也可用于皮套模式,口袋模式自动解锁与锁屏。
环境光传感器180L用于感知环境光亮度。终端100可以根据感知的环境光亮度自适应调节显示屏194亮度。环境光传感器180L也可用于拍照时自动调节白平衡。环境光传感器180L还可以与接近光传感器180G配合,检测终端100是否在口袋里,以防误触。指纹传感器180H用于采集指纹。终端100可以利用采集的指纹特性实现指纹解锁,访问应用锁,指纹拍照,指纹接听来电等。
温度传感器180J用于检测温度。在一些实施例中,终端100利用温度传感器180J检测的温度,执行温度处理策略。例如,当温度传感器180J上报的温度超过阈值,终端100执行降低位于温度传感器180J附近的处理器的性能,以便降低功耗实施热保护。在另一些实施例中,当温度低于另一阈值时,终端100对电池142加热,以避免低温导致终端100异常关机。在其他一些实施例中,当温度低于又一阈值时,终端100对电池142的输出电压执行升压,以避免低温导致的异常关机。
触摸传感器180K,也称“触控面板”。触摸传感器180K可以设置于显示屏194,由触摸传感器180K与显示屏194组成触摸屏,也称“触控屏”。触摸传感器180K用于检测作用于其上或附近的触摸操作。触摸传感器可以将检测到的触摸操作传递给应用处理器,以确定触摸事件类型。可以通过显示屏194提供与触摸操作相关的视觉输出。在另一些实施例中,触摸传感器180K也可以设置于终端100的表面,与显示屏194所处的位置不同。
骨传导传感器180M可以获取振动信号。在一些实施例中,骨传导传感器180M可以获取人体声部振动骨块的振动信号。骨传导传感器180M也可以接触人体脉搏,接收血压跳动信号。
按键190包括开机键,音量键等。按键190可以是机械按键。也可以是触摸式按键。终端100可以接收按键输入,产生与终端100的用户设置以及功能控制有关的键信号输入。马达191可以产生振动提示。马达191可以用于来电振动提示,也可以用于触摸振动反馈。例如,作用于不同应用(例如拍照,音频播放等)的触摸操作,可以对应不同的振动反馈效果。触摸振动反馈效果还可以支持自定义。指示器192可以是指示灯,可以用于指示充电状态,电量变化,也可以用于指示消息,未接来电,通知等。SIM卡接口195用于连接SIM卡。SIM卡可以通过插入SIM卡接口195,或从SIM卡接口195拔出,实现与终端100的接触和分离。
可以理解的是,图2所示的部件并不构成对终端100的具体限定,手机还可以包括比图示更多或更少的部件,或者组合某些部件,或者拆分某些部件,或者不同的部件布置。此外,图2中的部件之间的组合/连接关系也是可以调整修改的。
下面结合图2所示的终端100的结构,对本发明实施例进行详细阐述。以下实施例中所使用的术语只是为了描述特定实施例的目的,而并非旨在作为对本申请的限制。如在本 申请的说明书和所附权利要求书中所使用的那样,单数表达形式“一个”、“一种”、“所述”、“上述”、“该”和“这一”旨在也包括例如“一个或多个”这种表达形式,除非其上下文中明确地有相反指示。还应当理解,在本申请实施例中,“一个或多个”是指一个、两个或两个以上;“和/或”,描述关联对象的关联关系,表示可以存在三种关系;例如,A和/或B,可以表示:单独存在A,同时存在A和B,单独存在B的情况,其中A、B可以是单数或者复数。字符“以是一般表示前后关联对象是一种“或”的关系。
在本说明书中描述的参考“一个实施例”或“一些实施例”等意味着在本申请的一个或多个实施例中包括结合该实施例描述的特定特征、结构或特点。由此,在本说明书中的不同之处出现的语句“在一个实施例中”、“在一些实施例中”、“在其他一些实施例中”、“在另外一些实施例中”等不是必然都参考相同的实施例,而是意味着“一个或多个但不是所有的实施例”,除非是以其他方式另外特别强调。术语“包括”、“包含”、“具有”及它们的变形都意味着“包括但不限于”,除非是以其他方式另外特别强调。
以上实施例中所用,根据上下文,术语“当…时”或“当…后”可以被解释为意思是“如果…”或“在…后”或“响应于确定…”或“响应于检测到…”。类似地,根据上下文,短语“在确定…时”或“如果检测到(所陈述的条件或事件)”可以被解释为意思是“如果确定…”或“响应于确定…”或“在检测到(所陈述的条件或事件)时”或“响应于检测到(所陈述的条件或事件)”。另外,在上述实施例中,使用诸如第一、第二之类的关系术语来区份一个实体和另一个实体,而并不限制这些实体之间的任何实际的关系和顺序。
图4示出了本申请实施例提供的文本转语音的处理方法的一种示例的流程示意图。如图4所示,该方法的流程包括:
400,终端与服务器约定压缩策略。
其中,压缩策略包括静态压缩机制(constents bitrate,CBR),所谓静态压缩机制即每次压缩的音频数据的数据长度相同;所述数据长度可以理解为数据所占的字节、比特数等,例如数据长度为108字节。
所述压缩策略还可以包括每次压缩的音频数据中包括至少一帧完整的音频。这里一帧完整的音频是指每次压缩的音频数据中包括一帧完整的音频对应的帧头和帧尾。
所述压缩策略还可以包括每次压缩的音频数据的数据长度为预设长度。所述预设长度可以是事先确定好的,例如通过如下公式确定的:
数据长度=((每帧的采样数/8)*比特率)/采样率;
其中,每帧的采样数为从一帧完整音频中采样的数据;采样率即采样频率可以理解为每秒钟采样的数据,通常,采样频率越高声音的还原越真实。比特率可以理解为每秒传输比特数,又称数据传输速率。在一些实施例中,比特率/采样率可以是预设值。或者,与声音音质相关,例如音质较好,比特率/采样率较高,音质较差,比特率/采样率较低。因此,终端可以基于播放器的声音音质确定比特率/采样率,然后通过上述公式确定所述预设长度,即每次压缩音频数据的数据长度的大小。
这里所述的“约定”可以有多种实现方式,包括但不限定于如下所述的方式1-方式3。
方式1,服务器确定使用静态压缩机制以及每次压缩的音频数据的数据长度之后,可以通知终端。例如,服务器向终端发送第一指示信息,例如,第一指示信息用于指示服务器使用的静态压缩机制且每次压缩的音频数据的数据长度为预设长度,例如为108字节。
方式2,终端可以要求服务器使用静态压缩机制,并要求每次压缩的音频数据的数据长度为预设长度。例如,终端向服务器发送第二指示信息,该第二指示信息用于指示服务器使用静态压缩机制,并指示服务器每次压缩的音频数据的数据长度为预设长度,例如为108字节。
方式3,服务器生产商生成服务器时,默认设置服务器使用静态压缩机制,且每次压缩的音频数据的数据长度为预设长度;例如为108字节。终端下载音频流时,默认下载数据长度为预设长度的音频流。
可以理解的是,终端播放音频流时,是以帧为单位播放。通常,音频流中会标识每一帧的帧头(或称为码头)和帧尾(或称为码尾)。终端解析音频流中每帧的帧头和帧尾,然后一帧一帧播放。
服务器和终端约定每次压缩的音频数据的数据长度相同且为预设长度,且所述数据长度中包括至少一帧完整的音频,即包括至少一帧音频对应的帧头和帧尾。因此,终端下载音频流时,可以以约定好的数据长度下载。
举例来说,假设约定好的数据长度是108字节,那么服务器每次压缩的音频数据的数据长度为108字节,终端每次下载108个字节的音频数据,该108个字节的音频数据中包括至少一帧完整的音频,即包括至少一帧音频对应的帧头和帧尾。因此,终端每次下载的108字节的音频数据都是可以播放的音频流,因为所述108字节的音频数据中包括至少一帧完整的音频。
需要说明的是,现有技术中,服务器使用动态压缩机制(variable bitrate,VBR),所谓动态压缩机制即每次压缩的音频数据的数据长度不相同。例如,服务器压缩后的音频数据包括2个压缩包,一个压缩包的音频数据的长度为100K,另一个压缩包的音频数据的长度为60K。由于动态压缩音频数据长度不固定,一个压缩包中包含的音频数据可能小于一帧,例如不包括一帧完整的音频,例如,仅包括一帧音频的帧头但是不包括帧尾。图5示出了动态压缩机制的示意图。假设服务器压缩的音频数据包括两个压缩包,第一个压缩包包括100K,第二个压缩包包括60K。第一个压缩包以及第二压缩包中的前40K属于1帧完整的音频。那么,终端仅接收到第一个压缩包时,是无法播放的。
因此,现有技术中,终端每次下载到压缩包后,需要通过解析器解析压缩包中是否包括至少1帧完整的音频,若是,则播放音频数据。在解析器解析压缩包之前,无法确定该压缩包是否可以播放。
在本申请实施例中,终端与服务器约定每次压缩的音频数据(即每次压缩得到的压缩比)的数据长度相同,且为预设长度(例如108字节),且所述数据长度包括至少一帧完整音频,即包括至少一帧音频对应的帧头和帧尾。因此,终端每次下载预设长度(例如108字节)的音频数据,所以每次下载的音频数据都是可以播放的,因为每次下载的音频数据中包括至少一帧完整音频。也就是说,终端每次下载压缩后的音频数据之后,无需等到解析器对其进行解析,也可以确定该音频数据是可以播放的。
401,终端检测到文本转语音请求。
示例性的,终端中可以包括TTS服务系统。终端中的不同应用可以调用TTS服务系统,以请求文本向语音的转换。其中,不同应用可以包括语音助手、微信等。因此,上述的用于发起文本转语音请求可以是应用调用TTS服务系统的请求。例如,以语音助手为例,用户发出“你好,华为,帮我打开微博”的声音信号。终端检测到该声音信号之后,调用 TTS服务系统,以通过TTS服务系统实现本文到语音的转换,然后输出“已打开微博”的语音信息。
402,终端启动计时器,开始计时。
应理解,终端检测到用于发起文本转语音的请求后,可以启动计时器。
403,终端向服务器发送需要转换的文本。
403和402可以同时执行,或不同时执行,二者的执行顺序不作限定。
可以理解的是,403之前还可以包括终端与服务器连接的过程。终端与服务器之间的连接例如,IP连接、TCP连接等,本申请实施例不作限定。
404,服务器基于接收到的文本进行语音转换,并基于约定的压缩策略压缩转换得到的音频数据。该音频数据的格式例如可以为Opus格式。
如前文所述,终端与服务器约定每次压缩的音频数据的数据长度相同,且所述数据长度包括至少一帧完整的音频,即包括至少一帧音频对应的帧头和帧尾。例如,约定的数据长度为108个字节,则服务器每次压缩的音频数据包括108个字节,该108个字节中包括至少一帧完整的音频。
405,终端下载音频数据。
假设约定的数据长度为108个字节,则终端每次下载108个字节的音频数据,该108个字节中包括至少一帧完整的音频。
406,终端判断在计时器的计时达到阈值之前,是否下载到首帧可以播放的音频数据;若否,执行407,若是,执行408。
其中,阈值可以事先设置好的,例如终端出厂之前默认设置好的,或者,也可以是用户自定义的,例如可以是1.5s,本申请实施例不作限定。
如前文所述,终端与服务器约定每次压缩的音频数据的数据长度相同且为预设长度,且所述数据长度包括至少一帧完整的音频,即包括至少一帧音频对应的帧头和帧尾。因此,终端每次下载预设长度的音频数据可以播放的音频数据。
因此,所述首帧可以播放的音频数据包括终端首次下载的预设长度(例如,108字节)的音频数据。或者,所述首帧可以播放的音频数据为首次下载的包括至少一帧完整音频的数据,或者,所述首帧可以播放的音频数据为首次下载的包括至少一帧完整音频对应的帧头和帧尾的数据。
以预设长度为108字节为例,终端判断是否下载到首帧可以播放的音频数据可以包括:判断是否下载了108个字节。假设在计时器阈值(例如1.5s)达到之前,成功下载了108个字节,即下载了首帧可以播放的音频数据。因此,终端可以继续下载剩余音频数据。假设在1.5s之前,未成功下载108个字节,就放弃在线合成方式,而使用离线合成方式。
407,终端采用离线合成机制。
若计时器阈值达到之前,终端未下载到首帧可以播放的音频数据,可以使用离线合成机制。其中,离线合成机制可以是终端从离线数据库中确定文本对应的语音,然后将语音合成音频数据。离线数据库可以事先存储在终端中。
408,终端继续下载音频数据。
若计时器阈值达到之前,终端下载了首帧可以播放的音频数据,可以继续下载音频数据。此时,终端可以选择关闭计时器。
因此,本申请实施例中,终端向服务器发送文本之后,若在预设时长内未下载到首帧 可以播放的音频数据,则转离线合成,以降低终端音频播放时延。
在另一些实施例中,步骤408之后还可以包括如图6所示的409-410。
409,终端缓存音频数据。
410,终端确定缓存的音频数据达到上限时,将下载的首帧可以播放的音频数据传递到解析器进行解析。其中,所述上限例如为1M,5M等,本申请实施例不作限定。
图7示出了本申请实施例提供的缓存队列的示意图。该缓存队列的上限(或称为门限)为1M,当达到该上限时,下载的首帧可以播放的音频数据(例如,首次下载的108字节的音频数据)传递到解析器。之后,终端继续下载音频数据存储到缓存队列中,该缓存队列有助于音频无卡顿的播放。
需要说明的是,本申请实施例中缓存的上限设置相对较低(小于现有技术中的10M),可以保证音频数据尽快的传递到解析器进行解析。
可以理解的是,解析器对音频数据进行解析的过程例如,对音频数据进行解压(从服务器下载的音频数据是压缩后的音频数据)、解析音频数据中的帧头、帧尾等,以便播放器识别帧头和帧尾。
411,解码器对经过解析的音频数据进行解码,得到预设音频格式的音频数据。可以理解的是,解码器对音频数据的解码过程可以理解为对音频数据进行格式转换的过程,例如将音频数据的格式转换为可以播放的音频格式,例如,PCM格式。
412,终端将经过解码的音频数据传递到播放器进行播放。
如图8所示,本申请另外一些实施例公开了一种装置800,该装置可以是上文所述的终端,该装置800可以具有音频输出接口805,比如扬声器等。该电子设备还可以包括通信接口801:一个或多个处理器802,一个或多个存储器803;一个或多个计算机程序804;上述各器件可以通过一个或多个通信总线806连接。当然,电子设备还可以包括显示屏等其它器件。
其中,一个或多个计算机程序804被存储在上述存储器803中,并被配置为被该一个或多个处理器802执行,该一个或多个计算机程序804包括指令,上述指令可以用于执行如图4对应实施例中终端的各个步骤。
示例性的,图8所示的电子设备可以是图4所示的手机。这样的话,一个或多个处理器802可以是处理器110,音频输出接口805可以是扬声器,一个或多个存储器803可以是内部存储器121。
如图9所示,本申请另外一些实施例公开了一种装置900,装置900可以是上文中的服务器,也可以是服务器中的装置,例如芯片系统。芯片系统可以由芯片构成,也可以包含芯片和其他分立器件。
装置900包括处理单元901和通信单元902。通信单元902,用于接收终端发送的本文转语音的请求,所述请求中携带文本;处理单元901,用于将所述文本转换为音频数据;对所述音频数据进行压缩,每次压缩得到的音频数据的数据长度相同;通信单元902,还用于向所述终端分别发送压缩后的语音数据。
所述处理单元901每次压缩得到的音频数据的数据长度可以为预设长度。或者所述处理单元901每次压缩得到的音频数据中可以包括至少一帧完整的音频。可选的,通信单元902还可以通知所述终端所述服务器每次压缩的音频数据的数据长度。
此外,所述处理单元901每次压缩得到的音频数据的音频格式可以但不限于为Opus 格式。
关于处理单元901、通信单元902的具体执行过程以及方案可以达到的技术效果,可参见上方法实施例中的记载。本申请实施例中对模块的划分是示意性的,仅仅为一种逻辑功能划分,实际实现时可以有另外的划分方式,另外,在本申请各个实施例中的各功能模块可以集成在一个处理器中,也可以是单独物理存在,也可以两个或两个以上模块集成在一个模块中。上述集成的模块既可以采用硬件的形式实现,也可以采用软件功能模块的形式实现。
如图10所示,本申请另外一些实施例公开了一种装置1000,装置1000可以是上文中的服务器,也可以是服务器中的装置,例如芯片系统。芯片系统可以由芯片构成,也可以包含芯片和其他分立器件。
装置1000可以包括通信接口1001:一个或多个处理器1002,一个或多个存储器1003;一个或多个计算机程序1004;上述各器件可以通过一个或多个通信总线1006连接。通信接口1001用于通过传输介质和其它设备进行通信。一个或多个计算机程序1004被存储在上述存储器1003中,并被配置为被该一个或多个处理器1002执行,该一个或多个计算机程序1004包括指令,上述指令可以用于执行如图4对应实施例中服务器的各个步骤。
通信接口1001可以是收发器、电路、总线、模块或其它类型的通信接口。处理器1002可以是通用处理器、数字信号处理器、专用集成电路、现场可编程门阵列或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件,可以实现或者执行本申请实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者任何常规的处理器等。结合本申请实施例所公开的方法的步骤可以直接体现为硬件处理器执行完成,或者用处理器中的硬件及软件模块组合执行完成。存储器1003可以是非易失性存储器,比如硬盘(hard disk drive,HDD)或固态硬盘(solid-state drive,SSD)等,还可以是易失性存储器(volatile memory),例如随机存取存储器(random-access memory,RAM)。存储器是能够用于携带或存储具有指令或数据结构形式的期望的程序代码并能够由计算机存取的任何其他介质,但不限于此。本申请实施例中的存储器还可以是电路或者其它任意能够实现存储功能的装置。存储器1003和处理器1002耦合。本申请实施例中的耦合是装置、单元或模块之间的间隔耦合或通信连接,可以是电性,机械或其它的形式,用于装置、单元或模块之间的信息交互。作为另一种实现,存储器1003还可以位于装置1000之外。处理器1002可以和存储器1003协同操作。处理器1002可以执行存储器1003中存储的程序指令。所述至少一个存储器1003中的至少一个也可以包括于处理器1002中。本申请实施例中不限定上述通信接口1001、处理器1002以及存储器1003之间的连接介质。例如,本申请实施例在图10中以存储器1003、处理器1002以及通信接口1001之间可以通过总线连接,所述总线可以分为地址总线、数据总线、控制总线等。
可以理解的,上述图9所示实施例中的装置可以以图10所示的装置1000实现。具体的,处理单元901可以由处理器1002实现,通信单元902可以由通信接口1001实现。
在上述实施例中,可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时,可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行所述计算机程序指令时,全部或部分地产生按照本发明实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。所述计算机指令可以存储在计算机可读存储介质 中,或者从一个计算机可读存储介质向另一个计算机可读存储介质传输,例如,所述计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线(DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包含一个或多个可用介质集成的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质,(例如,软盘、硬盘、磁带)、光介质(例如,DVD)、或者半导体介质(例如固态硬盘Solid State Disk(SSD))等。
需要指出的是,本专利申请文件的一部分包含受著作权保护的内容。除了对专利局的专利文件或记录的专利文档内容制作副本以外,著作权人保留著作权。
Claims (18)
- 一种文本转语音的处理方法,其特征在于,应用于终端,所述方法包括:检测到用于指示文本转语音的指令;将所述文本发送服务器;从所述服务器下载由所述文本转换而成的音频数据;判断预设时长内是否下载到首帧可以播放的音频数据;若预设时长内下载到首帧可以播放的音频数据,继续下载剩余的音频数据。
- 如权利要求1所述的方法,其特征在于,所述方法还包括:若所述预设时长内未下载到首帧可以播放的音频数据,在离线数据库中确定将所述文本对应的音频信号,并将所述音频信号合成音频数据。
- 如权利要求1或2所述的方法,其特征在于,所述首帧可以播放的音频数据为首次下载的包括预设长度的音频数据。
- 如权利要求1-3任一所述的方法,其特征在于,所述首帧可以播放的音频数据为首次下载的包括至少一帧完整音频的数据。
- 如权利要求1-4任一所述的方法,其特征在于,从所述服务器下载由所述文本转换而成的音频数据,包括:每次从所述服务器下载数据长度为预设长度的音频数据。
- 如权利要求1-5任一所述的方法,其特征在于,所述方法还包括:将下载的音频数据存储到缓存区域中;确定所述缓存区域内的音频数据的内存占用量达到门限时,播放所述首帧可以播放的音频数据。
- 如权利要求6所述的方法,其特征在于,播放所述首帧可以播放的音频数据,包括:将所述首帧可以播放的音频数据传递给解析器进行解析,得到解压后的音频数据;将经过解析器解析后的音频数据传递给解码器进行解码,得到预设音频格式的音频数据;将经过解码器解码后的音频数据传递给播放器播放。
- 如权利要求1-7任一所述的方法,其特征在于,从服务器下载的音频数据为Opus格式。
- 一种文本转语音的处理方法,其特征在于,应用于服务器,所述方法包括:接收终端发送的本文转语音的请求,所述请求中携带文本;将所述文本转换为音频数据;对所述音频数据进行压缩,每次压缩得到的音频数据的数据长度相同;向所述终端分别发送压缩后的语音数据。
- 如权利要求9所述的方法,其特征在于,所述每次压缩得到的音频数据的数据长度为预设长度。
- 如权利要求9或10所述的方法,其特征在于,所述每次压缩得到的音频数据中包括至少一帧完整的音频。
- 如权利要求9-11任一所述的方法,其特征在于,所述每次压缩得到的音频数据的 音频格式为Opus格式。
- 如权利要求9-12任一所述的方法,其特征在于,所述方法还包括:通知所述终端所述服务器每次压缩的音频数据的数据长度。
- 一种终端,其特征在于,包括:通信模块,用于与其他设备通信;一个或多个处理器;一个或多个存储器;所述一个或多个存储器中存储有一个或多个计算机程序,所述一个或多个计算机程序包括指令,当所述指令被所述一个或多个处理器执行时,使得所述终端执行如权利要求1-8任一项所述的方法步骤。
- 一种服务器,其特征在于,包括:通信模块,用于与其他设备通信;一个或多个处理器;一个或多个存储器;所述一个或多个存储器中存储有一个或多个计算机程序,所述一个或多个计算机程序包括指令,当所述指令被所述一个或多个处理器执行时,使得所述服务器执行如权利要求9-13任一项所述的方法步骤。
- 一种通信系统,其特征在于,包括如权利要求14所述的终端和如权利要求15所述的服务器。
- 一种计算机可读存储介质,其特征在于,所述计算机可读存储介质存储有计算机程序,所述计算机程序包括程序指令,当所述程序指令被计算机执行时,使所述计算机执行如权利要求1-8中任意一项所述的方法;或,当所述程序指令被计算机执行时,使所述计算机执行权利要求9-13中任一项所述的方法。
- 一种程序产品,其特征在于,所述程序产品存储有计算机程序,所述计算机程序包括程序指令,当所述程序指令被计算机执行时,使所述计算机执行如权利要求1-8中任意一项所述的方法;或,当所述程序指令被计算机执行时,使所述计算机执行权利要求9-13中任一项所述的方法。
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/790,257 US20230045631A1 (en) | 2019-12-30 | 2020-11-20 | Text to Speech Processing Method, Terminal, and Server |
EP20908489.6A EP4071752A4 (en) | 2019-12-30 | 2020-11-20 | TEXT/VOICE PROCESSING PROCESS, TERMINAL AND SERVER |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911404905.4 | 2019-12-30 | ||
CN201911404905.4A CN113129861A (zh) | 2019-12-30 | 2019-12-30 | 一种文本转语音的处理方法、终端及服务器 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2021135713A1 true WO2021135713A1 (zh) | 2021-07-08 |
Family
ID=76687264
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2020/130466 WO2021135713A1 (zh) | 2019-12-30 | 2020-11-20 | 一种文本转语音的处理方法、终端及服务器 |
Country Status (4)
Country | Link |
---|---|
US (1) | US20230045631A1 (zh) |
EP (1) | EP4071752A4 (zh) |
CN (1) | CN113129861A (zh) |
WO (1) | WO2021135713A1 (zh) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113658578A (zh) * | 2021-08-16 | 2021-11-16 | 广东省林业科学研究院 | 语音在北斗短报文碎片传输后的合并方法 |
US20230045631A1 (en) * | 2019-12-30 | 2023-02-09 | Huawei Technologies Co., Ltd. | Text to Speech Processing Method, Terminal, and Server |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2009258498A (ja) * | 2008-04-18 | 2009-11-05 | Mitsubishi Electric Corp | 音声合成装置及び音声合成方法 |
CN102568471A (zh) * | 2011-12-16 | 2012-07-11 | 安徽科大讯飞信息科技股份有限公司 | 语音合成方法、装置和系统 |
CN106504742A (zh) * | 2016-11-14 | 2017-03-15 | 海信集团有限公司 | 合成语音的传输方法、云端服务器和终端设备 |
US10186252B1 (en) * | 2015-08-13 | 2019-01-22 | Oben, Inc. | Text to speech synthesis using deep neural network with constant unit length spectrogram |
CN110223694A (zh) * | 2019-06-26 | 2019-09-10 | 百度在线网络技术(北京)有限公司 | 语音处理方法、系统和装置 |
CN110600004A (zh) * | 2019-09-09 | 2019-12-20 | 腾讯科技(深圳)有限公司 | 一种语音合成播放方法、装置和存储介质 |
Family Cites Families (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR20010099454A (ko) * | 2001-09-28 | 2001-11-09 | 김인권 | 텍스트(사용자입력)와 발신자번호 및 주소록을 이용한휴대폰음성벨소리. |
CN103916716B (zh) * | 2013-01-08 | 2017-06-20 | 北京信威通信技术股份有限公司 | 一种无线网络下视频实时传输的码率平滑方法 |
US9704477B2 (en) * | 2014-09-05 | 2017-07-11 | General Motors Llc | Text-to-speech processing based on network quality |
CN104992704B (zh) * | 2015-07-15 | 2017-06-20 | 百度在线网络技术(北京)有限公司 | 语音合成方法和装置 |
CN107039032A (zh) * | 2017-04-19 | 2017-08-11 | 上海木爷机器人技术有限公司 | 一种语音合成处理方法及装置 |
CN107105096A (zh) * | 2017-04-28 | 2017-08-29 | 努比亚技术有限公司 | 一种音频处理方法、终端及存储介质 |
CN107370814B (zh) * | 2017-07-21 | 2018-09-04 | 掌阅科技股份有限公司 | 电子书朗读处理方法、终端设备及计算机存储介质 |
JP2020529032A (ja) * | 2018-06-12 | 2020-10-01 | 深▲せん▼市合言信息科技有限公司Langogo Technology Co.,Ltd. | 音声認識翻訳方法及び翻訳装置 |
CN111105778A (zh) * | 2018-10-29 | 2020-05-05 | 阿里巴巴集团控股有限公司 | 语音合成方法、装置、计算设备和存储介质 |
CN109300467B (zh) * | 2018-11-30 | 2021-07-06 | 四川长虹电器股份有限公司 | 语音合成方法及装置 |
CN109672745A (zh) * | 2018-12-29 | 2019-04-23 | 苏州思必驰信息科技有限公司 | 用于FreeRTOS的音频在线播放控制方法和装置 |
CN113129861A (zh) * | 2019-12-30 | 2021-07-16 | 华为技术有限公司 | 一种文本转语音的处理方法、终端及服务器 |
-
2019
- 2019-12-30 CN CN201911404905.4A patent/CN113129861A/zh active Pending
-
2020
- 2020-11-20 WO PCT/CN2020/130466 patent/WO2021135713A1/zh unknown
- 2020-11-20 EP EP20908489.6A patent/EP4071752A4/en active Pending
- 2020-11-20 US US17/790,257 patent/US20230045631A1/en active Pending
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2009258498A (ja) * | 2008-04-18 | 2009-11-05 | Mitsubishi Electric Corp | 音声合成装置及び音声合成方法 |
CN102568471A (zh) * | 2011-12-16 | 2012-07-11 | 安徽科大讯飞信息科技股份有限公司 | 语音合成方法、装置和系统 |
US10186252B1 (en) * | 2015-08-13 | 2019-01-22 | Oben, Inc. | Text to speech synthesis using deep neural network with constant unit length spectrogram |
CN106504742A (zh) * | 2016-11-14 | 2017-03-15 | 海信集团有限公司 | 合成语音的传输方法、云端服务器和终端设备 |
CN110223694A (zh) * | 2019-06-26 | 2019-09-10 | 百度在线网络技术(北京)有限公司 | 语音处理方法、系统和装置 |
CN110600004A (zh) * | 2019-09-09 | 2019-12-20 | 腾讯科技(深圳)有限公司 | 一种语音合成播放方法、装置和存储介质 |
Non-Patent Citations (1)
Title |
---|
See also references of EP4071752A4 * |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20230045631A1 (en) * | 2019-12-30 | 2023-02-09 | Huawei Technologies Co., Ltd. | Text to Speech Processing Method, Terminal, and Server |
CN113658578A (zh) * | 2021-08-16 | 2021-11-16 | 广东省林业科学研究院 | 语音在北斗短报文碎片传输后的合并方法 |
CN113658578B (zh) * | 2021-08-16 | 2023-09-15 | 广东省林业科学研究院 | 语音在北斗短报文碎片传输后的合并方法 |
Also Published As
Publication number | Publication date |
---|---|
EP4071752A1 (en) | 2022-10-12 |
EP4071752A4 (en) | 2023-01-18 |
CN113129861A (zh) | 2021-07-16 |
US20230045631A1 (en) | 2023-02-09 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2020244623A1 (zh) | 一种空鼠模式实现方法及相关设备 | |
JP7235871B2 (ja) | データ伝送方法と電子デバイス | |
CN113542839B (zh) | 电子设备的投屏方法和电子设备 | |
WO2020014880A1 (zh) | 一种多屏互动方法及设备 | |
WO2022048474A1 (zh) | 一种多应用共享摄像头的方法与电子设备 | |
WO2020078336A1 (zh) | 翻译方法及终端 | |
WO2021190314A1 (zh) | 触控屏的滑动响应控制方法及装置、电子设备 | |
WO2021135713A1 (zh) | 一种文本转语音的处理方法、终端及服务器 | |
WO2022048371A1 (zh) | 跨设备音频播放方法、移动终端、电子设备及存储介质 | |
CN111316604B (zh) | 一种数据传输方法及电子设备 | |
WO2022166618A1 (zh) | 一种投屏的方法和电子设备 | |
CN113498108A (zh) | 基于业务类型调整数据传输策略的芯片、设备以及方法 | |
CN114822525A (zh) | 语音控制方法和电子设备 | |
CN114398020A (zh) | 一种音频播放方法及相关设备 | |
CN113127188A (zh) | 一种性能优化方法与电子设备 | |
CN115694598A (zh) | 一种北斗通信系统中多帧融合传输方法及相关装置 | |
WO2020078267A1 (zh) | 在线翻译过程中的语音数据处理方法及装置 | |
WO2022161006A1 (zh) | 合拍的方法、装置、电子设备和可读存储介质 | |
WO2022135254A1 (zh) | 一种编辑文本的方法、电子设备和系统 | |
WO2022170854A1 (zh) | 视频通话的方法与相关设备 | |
WO2021114950A1 (zh) | 一种多路http通道复用的方法及终端 | |
CN115525366A (zh) | 一种投屏方法及相关装置 | |
CN113473216A (zh) | 数据传输方法、芯片系统及相关装置 | |
CN114157412A (zh) | 信息验证的方法、电子设备及计算机可读存储介质 | |
CN116684036B (zh) | 数据处理方法及相关装置 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 20908489 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 2020908489 Country of ref document: EP Effective date: 20220707 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |