US20190073993A1 - Artificially generated speech for a communication session - Google Patents
Artificially generated speech for a communication session Download PDFInfo
- Publication number
- US20190073993A1 US20190073993A1 US16/177,345 US201816177345A US2019073993A1 US 20190073993 A1 US20190073993 A1 US 20190073993A1 US 201816177345 A US201816177345 A US 201816177345A US 2019073993 A1 US2019073993 A1 US 2019073993A1
- Authority
- US
- United States
- Prior art keywords
- transmitting
- remote device
- user
- audio data
- speech
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 238000004891 communication Methods 0.000 title claims abstract description 60
- 230000006870 function Effects 0.000 claims abstract description 11
- 238000000034 method Methods 0.000 claims description 14
- 230000005540 biological transmission Effects 0.000 claims description 11
- 230000005236 sound signal Effects 0.000 abstract 1
- 238000012545 processing Methods 0.000 description 11
- 230000003287 optical effect Effects 0.000 description 7
- 230000004044 response Effects 0.000 description 7
- 238000012546 transfer Methods 0.000 description 4
- 230000001413 cellular effect Effects 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 238000012937 correction Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 239000013589 supplement Substances 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000002868 homogeneous time resolved fluorescence Methods 0.000 description 1
- 239000000047 product Substances 0.000 description 1
- 230000035807 sensation Effects 0.000 description 1
- 239000004984 smart glass Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000003595 spectral effect Effects 0.000 description 1
- 230000000153 supplemental effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- PICXIOQBANWBIZ-UHFFFAOYSA-N zinc;1-oxidopyridine-2-thione Chemical class [Zn+2].[O-]N1C=CC=CC1=S.[O-]N1C=CC=CC1=S PICXIOQBANWBIZ-UHFFFAOYSA-N 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
- G10L13/02—Methods for producing synthetic speech; Speech synthesisers
- G10L13/033—Voice editing, e.g. manipulating the voice of the synthesiser
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
- G10L13/02—Methods for producing synthetic speech; Speech synthesisers
- G10L13/04—Details of speech synthesis systems, e.g. synthesiser structure or memory management
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
- G10L13/02—Methods for producing synthetic speech; Speech synthesisers
- G10L13/04—Details of speech synthesis systems, e.g. synthesiser structure or memory management
- G10L13/047—Architecture of speech synthesisers
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
- G10L13/08—Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/1066—Session management
- H04L65/1069—Session establishment or de-establishment
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/75—Media network packet handling
- H04L65/752—Media network packet handling adapting media to network capabilities
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/75—Media network packet handling
- H04L65/762—Media network packet handling at the source
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/80—Responding to QoS
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04M—TELEPHONIC COMMUNICATION
- H04M3/00—Automatic or semi-automatic exchanges
- H04M3/22—Arrangements for supervision, monitoring or testing
- H04M3/2236—Quality of speech transmission monitoring
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04M—TELEPHONIC COMMUNICATION
- H04M7/00—Arrangements for interconnection between switching centres
- H04M7/006—Networks other than PSTN/ISDN providing telephone service, e.g. Voice over Internet Protocol (VoIP), including next generation networks with a packet-switched transport layer
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04M—TELEPHONIC COMMUNICATION
- H04M7/00—Arrangements for interconnection between switching centres
- H04M7/006—Networks other than PSTN/ISDN providing telephone service, e.g. Voice over Internet Protocol (VoIP), including next generation networks with a packet-switched transport layer
- H04M7/0081—Network operation, administration, maintenance, or provisioning
- H04M7/0084—Network monitoring; Error detection; Error recovery; Network testing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04S—STEREOPHONIC SYSTEMS
- H04S7/00—Indicating arrangements; Control arrangements, e.g. balance control
- H04S7/30—Control circuits for electronic adaptation of the sound field
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
- G10L19/0018—Speech coding using phonetic or linguistical decoding of the source; Reconstruction using text-to-speech synthesis
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04M—TELEPHONIC COMMUNICATION
- H04M2201/00—Electronic components, circuits, software, systems or apparatus used in telephone systems
- H04M2201/40—Electronic components, circuits, software, systems or apparatus used in telephone systems using speech recognition
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R2227/00—Details of public address [PA] systems covered by H04R27/00 but not provided for in any of its subgroups
- H04R2227/003—Digital PA systems using, e.g. LAN or internet
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R27/00—Public address systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04S—STEREOPHONIC SYSTEMS
- H04S2420/00—Techniques used stereophonic systems covered by H04S but not provided for in its groups
- H04S2420/01—Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
Definitions
- Various systems which enable a user to send spoken audio to another user over a network, e.g. over a packet-based network such as the Internet or a private intranet, for instance in a live voice or video call such as a VoIP (voice over Internet Protocol) call.
- a network e.g. over a packet-based network such as the Internet or a private intranet
- VoIP voice over Internet Protocol
- each of the users installs a respective instance of a communication client application (e.g. VoIP client) on his or her respective user terminal.
- a communication client application e.g. VoIP client
- the users may use a web-hosted instance of the communication client.
- the call or other such communication session can also comprise video.
- a transmitting device captures voice information from a user, uses speech recognition to convert the voice information into text, and communicates packets encoding the voice information and the text to a receiving device at a remote location.
- the voice and text are sent in separate streams with different service levels.
- the receiving device receives and decodes the packets containing the voice and text information, outputs the voice information through a speaker, and outputs the text information on a visual display.
- the system provides both real-time voice communications and a reliable stream of text encoding those voice communications in case of poor network conditions.
- communications equipment can display a substantially real-time transcript of a voice communications session for reference during the conversation, to supplement the voice communications during periods of reduced transmission quality, or to save for future reference.
- a text-to-speech module at the receiving device is also able to detect a degradation in a quality of the packet-based voice communications session, and to output the transmitting user's voice information using speech synthesis to convert the remote text into an audio output.
- the voice-to-text module can supplement poor quality voice communications with synthesized speech.
- a device which includes a processor and a memory in communication with the processor.
- the memory includes executable instructions that, when executed by the processor, cause the processor to control the device to perform functions of capturing a speech by a user; generating audio data representing the captured speech by a user; generating, based on the audio data, text data representing at least a portion of the captured speech; and transmitting, via a communication channel, the audio data and text data to a remote device.
- a method in another implementation, includes capturing a speech by a user; generating audio data representing the captured speech by a user; generating, based on the audio data, text data representing at least a portion of the captured speech; and transmitting, via a communication channel, the audio data and text data to a remote device.
- a device in another implementation, includes means for capturing a speech by a user; means for generating audio data representing the captured speech by a user; means for generating, based on the audio data, text data representing at least a portion of the captured speech; and means for transmitting, via a communication channel, the audio data and text data to a remote device.
- FIG. 1 is a schematic diagram of a communication system.
- FIG. 2 is a schematic block diagram of a transmitting and receiving terminal.
- FIG. 3 a is a schematic mock-up of a user interface.
- FIG. 3 b is another schematic mock-up of a user interface.
- the following describes a system which uses automatic speech recognition (ASR) on the speech captured at the transmit side and transmits the resulting text reliability and simultaneously with the audio stream, to thereby increase the network resiliency of a live audio communication session such as a VoIP call.
- ASR automatic speech recognition
- the recognized speech is sent to the receiving end, where a user-specific text-to-speech voice can fill in the distorted audio when it is detected that audio quality is poor due to network loss or other such effects.
- the conversion from text to speech at the receive side is based on a user-specific model specific to the user from whom the speech was originally captured at the send side.
- a visual cue can be used to notify the receiving user when this is happening.
- the text-to-speech voice may be spatially separated so that it sounds separate from the rest of the conversation, e.g. by playing only to the left or right channel, or by using a head-related transfer function (HRTF) (the simplest is to).
- HRTF head-related transfer function
- the HTRF may be generic or specific to the receiving user.
- the received text may also be displayed as subtitles. This may comprise displaying the subtitles only in response to detecting that audio quality is poor at the receive side, e.g. only displaying the lost words or interrupted phrases or sentences.
- subtitles can also be displayed on-demand or by user-defined rules to allow the user to recall the text of what was recently said, e.g. to allow the receiving user to see the transcript at any time, such as to follow a meeting.
- the manual mode can be used for accessibility.
- FIG. 1 schematically illustrates a communication system 100 in accordance with embodiments disclosed herein.
- the communication system comprises a plurality of user terminals 102 each configured to connect to a computer network 101 , and each installed with a respective instance of a communication client application 103 .
- Each of the user terminals 102 is also used by at least one respective user 106 .
- two user terminals and their respective clients and users are shown for illustrative purposes: a first user terminal 102 a installed with a first communication client 103 a, and a second user terminal 102 b installed with a second communication client 103 b (in embodiments, each client 103 a, 103 being a respective instance of the same client product).
- the network 101 is preferably a packet-based network. In embodiments it may take the form of a wide-area internetwork such as that commonly referred to as the Internet. Alternatively, the network 101 may take the form of another type of network such as a company intranet, a mobile cellular network, or a wireless local area network (WLAN), or a combination of any of these and/or the Internet.
- a packet-based network In embodiments it may take the form of a wide-area internetwork such as that commonly referred to as the Internet. Alternatively, the network 101 may take the form of another type of network such as a company intranet, a mobile cellular network, or a wireless local area network (WLAN), or a combination of any of these and/or the Internet.
- WLAN wireless local area network
- Each of the user terminals 102 may take any of a variety of different forms, such as a desktop computer, laptop, tablet, smartphone, smart watch, pair of smart glasses, smart TV, or set-top box (and the different user terminals 102 need not necessarily take the same form as one another). Note therefore that the term “computer” as used herein does not restrict to a traditional desktop or laptop computer.
- the communication clients 103 are each installed on computer-readable storage of their respective user terminal 102 and arranged for execution on a respective processing apparatus of the respective user terminal 102 .
- the storage may take the form of one or more storage media (e.g. magnetic memory, electronic memory and/or optical memory) implemented in one or more memory units.
- the processing apparatus may take the form of one or more processing units.
- Each of the communications clients 103 is configured so as to be able to establish an audio communication session with one or more others of the communication clients 103 running on one or more of the other respective user terminals 102 , e.g. to establish a voice or video call such as a VoIP call.
- the user of each user terminal 102 is then able to transmit audio (and optionally video) to each other of the users of each other of the user terminals 102 participating in the session.
- the transmitted audio comprises speech spoken by the sending user 106 a, captured from an internal or external microphone at transmitting user terminal 102 a.
- the audio is transmitted live, i.e. on-the-fly, as-and-when it is captured, such as in a live VoIP call or the like.
- the transmitted content may also comprise video (e.g. live video) captured by an internal or external video camera at the transmitting user terminal 102 a, e.g. a head-and-shoulders shot of the sending user 106 a.
- the captured audio and optional video are encoded (i.e. compressed) by the sending client application 103 a and transmitted in encoded form over the network 101 to the client 103 b on the receiving terminal 102 b.
- the audio data comprises the speech of the sending user 106 a, or such like
- this will be understood to cover the typical scenario in which the audio data is transmitted in encoded (compressed) form, and similarly for any transmitted video.
- the receiving client 103 b decodes and plays out the received audio through one or more internal or external speakers at the receiving terminal 102 b, and optionally decodes and plays out any receive video through an internal or external display screen.
- the communication system 101 comprises a server 104 connected to the network 101 , arranged to provide a communication service via which the communication session is at least in part conducted.
- the audio and optional video from any given one of the users may be sent from the client 103 a on the sending user terminal 102 a to the server 104 via the network 101 , and the server 104 may be configured to forward the audio and any video onward via the network 101 to the client 103 b on the recipient user terminal 102 b (or to each of the recipient terminals in a multiparty communication session).
- a server herein refers to a logical entity which may be physically implemented by one or more server units located at one or more geographical sites.
- a server 104 may not be required at all (a pure peer-to-peer system), or the server 104 may still be included in the system to provide one or more supporting functions such as providing authentication certificates, address look-up, or hosting of supplemental information such as contact lists, user profiles, user permission settings, and/or presence statuses.
- the system need not comprise a communication client 103 installed on each user terminal 102 .
- the user terminals could instead be installed with a general-purpose web browser operable to access a web-based version of the client application (“web client”) hosted on the server 104 .
- web client web-based version of the client application
- the described functionality may be achieved by the web-client rather than the locally installed client (i.e. client installed locally on an individual user terminal 102 ).
- the functionality disclosed herein can be implemented by any combination of a local client application 103 (on each user terminal 102 ) and/or server hosted functionality (e.g. a web client).
- server hosted functionality e.g. a web client
- FIG. 3 a gives a schematic mock-up of an example of the front-end user interface (UI) 300 of the client application 103 b running on the receiving user's terminal 102 b (or of the web client accessed from the receiving user's terminal 102 b ).
- UI front-end user interface
- the receiving user 106 b sees the video of the sending user 106 a played out on screen, and the speech captured from the sending user's voice is played out audibly through the speaker(s) of the receiving terminal 102 b.
- conditions over the network 101 are poor (e.g.
- the sending client 103 a is configured with a speech-to-text converter to convert the sending user's locally captured speech to the receiving user terminal 102 b, e.g. in parallel with the audio but in separate packets.
- the receiving client 103 b is configured with a text-to-speech converter comprising a model of the sending user's voice, arranged to convert the received text to synthesized speech at the receive end based on said model, and play this out to the receiving user 106 b. This may be done in response to a detection at the receiving client 103 b that the network conditions experienced upon reception have fallen below a threshold quality according to a predetermined quality classifier.
- the text-to-speech conversion could be activated on some other basis, e.g. turned on manually by the receiving user 106 b.
- the transmitting user terminal 102 a comprises a microphone 202 , an optional video camera 204 , the transmitting client 103 a, and a transmitter 210 in the form of a transmitting network interface for connecting to the network 101 .
- the transmitting client 103 a comprises an encoder 206 and a speech-to-text converter 208 , the speech-to-text converter 208 comprises a voice model specific to the voice of the transmitting user 103 a.
- the microphone 202 may be internal or external to the housing of the transmitting user terminal 102 a. Either way, it is arranged to capture audio from its surroundings including speech from the transmitting user 106 a, and to supply the audio including the captured speech to the encoder 206 , which encodes (i.e. compresses) the audio for transmission over the network 101 via the transmitting interface 210 .
- the video camera 204 if used, may be internal or external to the housing of the transmitting user terminal 102 a. Either way it is arranged to capture video from its surroundings, e.g. a talking head shot of the transmitting user 102 a, and to supply this to the encoder 206 which encodes it for transmission over the network 101 via the transmitting interface 210 as part of the same session (e.g. call) as the transmitted audio.
- the speech-to-text converter 208 is arranged so as to also receive an instance of the captured audio, including the sending user's speech, and to convert this into a textual representation of the same speech content based on the model 209 of the sending user's voice as maintained at the transmitting user terminal 102 a.
- the model 209 at the sending side may be trained based on any known speech recognition training techniques, such as by having the sending user reading one or more passage of predetermined text, and/or by the sending user dynamically training the model 209 by correcting its speech-to-text predictions in an ongoing manner (e.g. during the call or other such session).
- the model 209 may be trained specifically for the purpose of the client application 103 (e.g. VoIP client) and its corresponding communication service (e.g. VoIP service).
- the model 209 may be trained and made available for some other or shared purpose, such as for dictation or voice control of the operating system on the transmitting terminal 102 a.
- the model 209 and speech-to-text converter 208 may be a dedicated part of the client application 103 a.
- the model 209 or indeed the speech-to-text converter 208 may be part of another application or the operating system (not shown) and merely accessed by the client application 103 a. A combination of these approaches could also be used.
- the speech-to-text converter 208 is configured to supply the converted text to the encoder 206 to synchronize this converted text with the corresponding audio from which it was generated. This may be achieved by the encoder inserting timestamps into portions of the converted text data that match the timestamps in corresponding portions of the encoded audio and optionally the video, e.g. matching the timestamps of corresponding frames, packets or other such subdivisions of the audio and optional video.
- the encoder 206 then sends the timestamped text data over the network 101 via the transmitting interface 210 as part of the session (e.g. call).
- the text data may be transmitted in uncompressed form, since it incurs a relatively very small bitrate compared to the audio, or alternatively the encoder 206 may encode the text for transmission as well as the audio and any video. Note therefore that where it is said that the transmitted content comprises text, or the like, this does not exclude the possibility that the text may be sent in encoded (compressed) form.
- the transmitting interface may take any suitable form for transmitting the encoded audio and optional video over the network (e.g. Internet) 101 , for instance a wireless interface for accessing the network 101 via a wireless router, wireless access point or a base station of a mobile cellular network; or a wired modem or such like for accessing the network 101 via wired means such as a PSTN phone line, optical cable or Ethernet connection.
- the network e.g. Internet
- the transmitting interface may take any suitable form for transmitting the encoded audio and optional video over the network (e.g. Internet) 101 , for instance a wireless interface for accessing the network 101 via a wireless router, wireless access point or a base station of a mobile cellular network; or a wired modem or such like for accessing the network 101 via wired means such as a PSTN phone line, optical cable or Ethernet connection.
- the encoder 206 , speech-to-text converter 208 and speech model 209 may be implemented in software stored in any suitable storage medium or media of the transmitting user terminal 102 a, and arranged of be fetched and run on any suitable processing apparatus of the transmitting terminal 102 a.
- suitable storage media include electronic storage media such as a solid state drive (SSD) or other EEPROM, magnetic media such as a hard disk or other magnetic drive, and/or optical memory such as a CDROM drive or other optical drive.
- suitable processing apparatus include a single or multiple core processing unit, or an arrangement of multiple such processing units.
- the receiving user terminal 102 b comprises one or more speakers 222 , a display screen 224 , the receiving client 103 b, and a receiver 212 in the form of a receiving network interface.
- the receiving client 103 b comprises a decoder 214 , a controller 216 , and a text-to-speech converter 218 , the speech-to-text converter 218 comprising a voice model 209 ′ specific to the sending user's voice maintained at the receiving user terminal 102 b.
- part or all of the voice model 209 ′ or the text-to-speech converter 218 itself could be a separate element to the client application 103 a.
- Each of the speaker(s) 222 and screen 224 may be internal or external to the housing of the receiving user terminal 102 b.
- the receiving interface 212 may take any suitable form for receiving the encoded audio and optional video over the network (e.g. Internet) 101 , for instance a wireless interface for accessing the network 101 via a wireless router, wireless access point or a base station of a mobile cellular network; or a wired interface or modem for accessing the network 101 via wired means such as a PSTN phone line, optical cable or Ethernet connection.
- the decoder 214 , controller 216 , text-to-speech speech converter 218 and speech model 209 ′ may be implemented in software stored in any suitable storage medium or media of the transmitting user terminal 102 a and arranged of be fetched and run on any suitable processing apparatus of the transmitting terminal 102 a.
- suitable storage media include electronic storage media such as a solid-state drive (SSD) or other EEPROM, magnetic media such as a hard disk or other magnetic drive, and/or optical memory such as a CDROM drive or other optical drive; and examples of suitable processing apparatus include a single or multiple core processing unit, or an arrangement of multiple such processing units.
- the transmitter 201 under control of the encoder 206 , is arranged to send the converted text data 254 , the encoded audio data 250 and (optionally) the encoded video 252 to the receiving user terminal 102 b.
- the receiving terminal 102 b receives these via the receiving interface 212 and passes the audio data 252 and (optional) video data 254 to the decoder to decoded and played out via the speaker(s) 222 and screen 224 respectively.
- the controller 216 is configured to determine when to control the text-to-speech converter 218 to convert the received text data 254 into synthesized speech and to play this synthesized speech out through the speaker(s) 222 in place of the received audio 250 .
- the controller 216 is configured detect when the quality of the network connection formed over the network 101 between the transmitting interface 210 of the transmitting terminal 102 a and the receiving interface 212 of the receiving terminal 102 b, over which the audio 250 and (optionally) the video 252 is received, falls below a predetermined level of quality. This is done based on a quality classifier evaluated by the controller 216 based on the connection quality as experienced at the receive side 102 b, 212 .
- Quality classifiers for classifying the quality of a network connection are in themselves known in the art.
- the classifier may be based on any one or a combination of network parameters that affect the receiving user's experience of the received audio, for instance: loss rate (e.g. packet loss rate); error rate (e.g. packet error rate); concealment rate (e.g. proportion of packets or frames concealed by the decoder); delay; and/or jitter.
- the controller 216 classifies whether, at the current moment in time, its experience of the network conditions is such that the received audio 250 is deemed usable or not usable (e.g. too bad to be understandable or too annoying to be practically usable). If the latter, then in response the controller 216 supplies the received text 254 to the text-to-speech converter 218 and controls it to play out the speech synthesized from this text through the speaker(s) 222 . The controller 216 also controls the decoder 214 to temporarily stop playing out the poor quality received audio 250 .
- the controller 216 detects that the connection quality is classified as good again, and then the controller 216 controls the text-to-speech converter 218 to stop playing out the synthesized speech and instead controls the decoder 214 to start playing out the received audio 250 again.
- the controller 216 uses the timestamps in the received text 254 and audio 250 to ensure that the synthesized speech supplied to the text-to-speech converter 218 correctly lines up and fills the gaps between the switched-out audio 250 , i.e. so that the periods of synthesized speech and the periods of playing out the received audio 250 are synchronized.
- the controller 216 automatically classifies the quality of the audio and switches between the received audio 250 and synthesized speech in dependence on this, this is not the only possibility. In embodiments the switching could be performed on some other alternative or additional basis, e.g. in response to a user selection input to the controller 216 through a user interface of the receiving client 103 b, selecting to manually to turn on or off the synthesized speech. Thus, the receiving user 106 b may decide for him- or herself whether the received audio 250 is unintelligible or too annoying, and in response to this select to manually switch to the artificial speech instead.
- the decoder 214 may also be arranged to play out some or all of the received text 254 in text form as subtitles.
- the controller 216 may be arranged to control the decoder 214 to output the text subtitles only when the connection is classified as poor, as above with regard to the synthesized speech.
- the decoder 214 may be arranged to display the subtitles continually throughout the session (e.g. call), even when conditions are classified as good; or the controller 216 may be configured to control the decoder 214 selectively display the subtitles on some other basis, e.g. when turned on by the receiving user 106 b.
- the controller 216 may be configured so as, when the receiving user activates an associated control in a user interface of the client application 103 b, to display a history of past-received subtitle text, e.g. all the text 254 received within a certain predefined time window or a certain user-selected time window.
- the text data 254 may be transmitted throughout the entire session (e.g. call), continually in parallel with the corresponding audio, i.e. so the transmitted content always comprises both audio 250 and text 254 representing the same periods of the session (e.g. call).
- the controller 216 is configured to control the text-to-speech converter to only play out the synthesized speech from certain selected portions of the received text during periods of the session (e.g. call), such as when conditions are classified as poor or when selected by the receiving user 106 b.
- the transmitting client 103 a may be configured to only transmit the text data 254 at certain times.
- the controller 216 may be configured to feed back a report to the transmitting client 103 a informing it when the network conditions experienced at the receive side are classified as poor.
- the speech-to-text converter 208 may only transmit the text data during such periods.
- the encoder 206 may also be configured to temporarily stop transmitting the audio 250 (and optional video 252 ) during such periods.
- the transmitted content may comprise only the audio 252 (and optionally video 252 ) but not the text data 254 during certain periods of the session (e.g. call), and only the text data 254 but not the audio 250 (nor video 252 ) during other periods of the same session.
- the text 254 is transmitted and received using a communication modality that has a higher robustness than that used for the audio data 250 (and video data 252 if present). This exploits the fact that the text data 254 will be much smaller in size than the audio data 250 (and any video data 252 ), i.e. incur a much smaller bitrate, and hence a more robust communication means can be afforded even in poor network conditions.
- this comprises transmitting and receiving the text data 254 using a transport layer protocol such as TCP that requires retransmission of unreceived packets, whereas the audio 250 (and video 252 if present) is sent and received using a transport layer protocol that does not involve retransmission (e.g. RTP or UDP).
- Retransmission can be achieved for example (as in TCP) by having the transport layer at the receiving terminal 102 b acknowledge received packets and having the transport layer at the transmitting terminal 102 a retransmit any packets that are not acknowledged as received (the packets being identified by their sequence number).
- transport layer at the receiving terminal requests retransmission of missing packets (which it identifies as missing due to missing sequence numbers).
- Another alternative or additional possibility for more robust transmission and reception is to send the text data 254 with a higher service level than the audio data 250 (and any video 252 ), e.g. a higher QoS (Quality of Service).
- the higher service level may be arranged to cause routers over the network 101 to prioritize not dropping the packets of text data 254 when there is network congestion and the router has to drop packets.
- the text data 254 may be sent with more redundant information than the audio 250 (and any video 252 ).
- the audio 250 and any video 252 .
- multiple duplicate instances of the same text packets could be sent, or a more robust checksum (incurring more bits) may be included for each packet or other such portion of text data 254 .
- the generation of the synthesized speech is based on a model of the transmit-side user's voice—i.e. specific to the voice of the transmitting user 106 a —but which is stored, maintained and applied at the received side 102 b, 218 .
- the receive-side model 209 ′ comprises one or more parameters of the transmitting user's voice, e.g. the fundamental pitch (i.e. fundamental frequency), the mean of the fundamental pitch, the variance of the fundamental pitch, and/or spectral information such as in the form of linear prediction coefficients or mel-cepstral coefficients.
- the fundamental pitch i.e. fundamental frequency
- the mean of the fundamental pitch i.e. fundamental frequency
- the variance of the fundamental pitch i.e. fundamental frequency
- spectral information such as in the form of linear prediction coefficients or mel-cepstral coefficients.
- Various parameters parametrizing the human voice are in themselves known to a person skilled in the art.
- values of one or more of these parameters may be provided to the receiving terminal 102 b “up front”, either at some point in advance of the session such as when the transmitting and receiving users 103 a, 103 b first become contacts, or at least at the beginning of the call.
- the one or more values of the one or more parameters may be
- the transmitting client 103 a may be configured to send one or more values of one or more of the voice parameters to the receiving client 103 b dynamically during the call, along with the audio stream 250 .
- the receiving client 103 b can then use these to dynamically adapt the receive-side voice model 209 ′. For instance, consider the case where, during the call or session, the transmitting user 106 a makes one or more manual corrections to the converted text generated by the speech-to-text converter 208 based on the transmit-side model 209 . This will result in the transmit-side model 209 being trained so as to update the values of one or more of the voice parameters in the transmit-side model 209 .
- the transmitting client may be configured to transmit the values of the one or more updated parameters 256 to the receiving client 103 b during the same session (e.g. call) in which those updates were made.
- the receiving client 103 b updates the values of those parameters in the receive-side model 209 ′, again during the same session (e.g. call).
- the receive-side voice model 209 ′ can be dynamically updated.
- the receiving user terminal 102 b or client 103 b receives one or more speech parameters (or indeed the audio 250 , video 252 and text 254 ) from the transmitting user terminal 102 a or client 102 b, this does not exclude that the data in question could be received via the server 104 or other intermediate element.
- the one or more parameters may be sent to the receiving client 103 b from the server 104 based on values originating from another user terminal of the sending user 106 a.
- the parameters may be trained by the sending user 106 a at some point in advance using a different user terminal other than the sending user terminal 102 a that is used during the session or call itself. I.e. the one or more parameters may be received from the transmitting user terminal, or may originate from another user terminal of the transmitting user. Or indeed a combination of such approaches could be used.
- the text-to-speech converter 218 is configured to train the receive-side voice model 209 ′ dynamically based on the received audio 250 in conjunction with the received text 254 during periods of good network conditions. This embodiment works when text 254 and audio 250 is received for the same content, i.e. same periods of text, such as when the text 254 is sent continually throughout the whole call. Since the text-to-speech converter 218 has access to the audio 250 and text 254 for the same content, it is able to train the receive side model 209 in much the same way as speech recognition software is trained based on the user reading out predetermined passages pre-known to the conversion software.
- FIG. 3 b shows a schematic mock-up of an example user interface 300 of the receiving client 103 b in circumstances when network conditions are classified as poor.
- the received audio 250 and any video 252 are no longer played out, and instead the received text 254 is displayed on the screen 224 as live subtitles 302 .
- the subtitles are played out as synthesized speech via the text-to-speech converter 218 and speaker(s) 222 .
- the controller 216 may be configured to control the user interface to also display the name 304 of the transmitting user 106 a who speech is currently being displayed as subtitles, which may be useful in multiparty call (i.e. with three or more participants).
- the controller 216 is configured to provide an indication to the receiving user 106 b that the speech being heard is synthesized speech. For instance, this may take the form of a visual indication 306 displayed on the screen 224 through the graphical user interface of the receiving client, e.g. in the form of an icon or other such graphic, and/or a written message explaining to the receiving user that the currently played out speech is synthesized due to poor network conditions.
- Another alternative or additional option for providing such an indication to the receiving user is to provide an audible indication via the speaker(s) 222 .
- One particular way of doing this in the case of two or more speakers is to use stereo or surround sound in order to place the synthesized speech at a distinct spatial position from the perspective of the receiving user 106 b, distinct compared to the position at which the regular received audio 250 is played out.
- An example of using stereo would be to play out the received audio 250 centrally, i.e. evenly through both speakers, but to play out the synthesized speech only through one of the two speakers 222 , or weighted toward one of the speakers, thus making it sound like it comes from a direction off to one side.
- HRTF head related transfer function
- the speech-to-text conversion may be implemented at a server 104 instead of at the transmitting terminal 102 a, with the converted text being forwarded to the receiving terminal 102 b.
- the text-to-speech could be performed at a server 104 and the synthesized speech forwarded to receiving user terminal 102 b.
- Such implementations are less preferred since the former does not accommodate poor network conditions between the transmitting user terminal 102 a and the server 104 , whilst the latter does not accommodate poor network conditions between the server 104 and the receiving user terminal 102 b. Nonetheless they do still provide some benefit is there are poor conditions on the other leg of the route.
- the server 104 may measure one or more factors such as loss, error rate, delay, jitter, etc. and report them to the controller 216 on the receiving terminal 102 b which performs the classification based on the reported values. Or the server 104 could perform the classification and report this to the controller 216 , in which case the controller 216 determines the network conditions simply by receiving the report.
- the server 104 may measure one or more factors such as loss, error rate, delay, jitter, etc. and report them to the controller 216 on the receiving terminal 102 b which performs the classification based on the reported values.
- the server 104 could perform the classification and report this to the controller 216 , in which case the controller 216 determines the network conditions simply by receiving the report.
- the communication session is not limited to being a call in the sense of a two-way live communication.
- the session could be a live one-way communication session such as a remotely-presented lecture or other such presentation, or a live internet TV show.
- the session could also comprise the streaming of pre-recorded audio or video, such as in an internet audio or video streaming service, e.g. internet TV, movie or radio service.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Acoustics & Sound (AREA)
- Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Computational Linguistics (AREA)
- Business, Economics & Management (AREA)
- General Business, Economics & Management (AREA)
- Environmental & Geological Engineering (AREA)
- Telephonic Communication Services (AREA)
Abstract
Description
- This application is a continuation of co-pending U.S. patent application Ser. No. 15/422,865, titled “ARTIFICIALLY GENERATED SPEECH FOR A COMMUNICATION SESSION,” filed on Feb. 2, 2017, which is incorporated hereto in its entirety. This application is related to co-pending U.S. application Ser. No. 16/147,838, titled “ARTIFICIALLY GENERATED SPEECH FOR A COMMUNICATION SESSION,” filed on Sep. 30, 2018, which is also a continuation of U.S. patent application Ser. No. 15/422,865 and incorporated hereto in its entirety.
- Various systems are known which enable a user to send spoken audio to another user over a network, e.g. over a packet-based network such as the Internet or a private intranet, for instance in a live voice or video call such as a VoIP (voice over Internet Protocol) call. To enable this, typically each of the users installs a respective instance of a communication client application (e.g. VoIP client) on his or her respective user terminal. Alternatively, one or both of the users may use a web-hosted instance of the communication client. In some scenarios the call or other such communication session can also comprise video.
- Calls and other audio or video sessions use networks that often have significant packet loss and jitter, which impair audio quality. Poor networks are the top reason why ˜5-20% of all audio calls (depending on region) are rated poor or very poor. Thus, loss and jitter can be mitigated and concealed but not eliminated. Previous solutions use forward error correction (FEC), audio concealment, or multi-path transmission techniques to mitigate network loss. However, significant loss can still result in unintelligible audio which makes communication difficult or impossible. Many calls with loss have large bursts of packet loss which makes FEC ineffective. Regarding jitter, this can be mitigated using a jitter buffer at the receive side. Increasing the length of the jitter buffer increases tolerance to jitter, but this comes at the cost of increased delay. In short, all techniques to deal with imperfect network conditions come with a limit or trade-off of one sort or another.
- In one known alternative solution, a transmitting device captures voice information from a user, uses speech recognition to convert the voice information into text, and communicates packets encoding the voice information and the text to a receiving device at a remote location. The voice and text are sent in separate streams with different service levels. The receiving device receives and decodes the packets containing the voice and text information, outputs the voice information through a speaker, and outputs the text information on a visual display. Thus, the system provides both real-time voice communications and a reliable stream of text encoding those voice communications in case of poor network conditions. In this way, communications equipment can display a substantially real-time transcript of a voice communications session for reference during the conversation, to supplement the voice communications during periods of reduced transmission quality, or to save for future reference.
- Furthermore, a text-to-speech module at the receiving device is also able to detect a degradation in a quality of the packet-based voice communications session, and to output the transmitting user's voice information using speech synthesis to convert the remote text into an audio output. Thus, the voice-to-text module can supplement poor quality voice communications with synthesized speech.
- A device is disclosed, which includes a processor and a memory in communication with the processor. The memory includes executable instructions that, when executed by the processor, cause the processor to control the device to perform functions of capturing a speech by a user; generating audio data representing the captured speech by a user; generating, based on the audio data, text data representing at least a portion of the captured speech; and transmitting, via a communication channel, the audio data and text data to a remote device.
- In another implementation, a method includes capturing a speech by a user; generating audio data representing the captured speech by a user; generating, based on the audio data, text data representing at least a portion of the captured speech; and transmitting, via a communication channel, the audio data and text data to a remote device.
- In another implementation, a device includes means for capturing a speech by a user; means for generating audio data representing the captured speech by a user; means for generating, based on the audio data, text data representing at least a portion of the captured speech; and means for transmitting, via a communication channel, the audio data and text data to a remote device.
- This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure.
- The drawing figures depict one or more implementations in accord with the present teachings, by way of example only, not by way of limitation. In the figures, like reference numerals refer to the same or similar elements. Furthermore, it should be understood that the drawings are not necessarily to scale.
-
FIG. 1 is a schematic diagram of a communication system. -
FIG. 2 is a schematic block diagram of a transmitting and receiving terminal. -
FIG. 3a is a schematic mock-up of a user interface. -
FIG. 3b is another schematic mock-up of a user interface. - The following describes a system which uses automatic speech recognition (ASR) on the speech captured at the transmit side and transmits the resulting text reliability and simultaneously with the audio stream, to thereby increase the network resiliency of a live audio communication session such as a VoIP call. The recognized speech is sent to the receiving end, where a user-specific text-to-speech voice can fill in the distorted audio when it is detected that audio quality is poor due to network loss or other such effects. The conversion from text to speech at the receive side is based on a user-specific model specific to the user from whom the speech was originally captured at the send side.
- In embodiments, a visual cue can be used to notify the receiving user when this is happening. Alternatively or additionally, the text-to-speech voice may be spatially separated so that it sounds separate from the rest of the conversation, e.g. by playing only to the left or right channel, or by using a head-related transfer function (HRTF) (the simplest is to). The HTRF may be generic or specific to the receiving user.
- In some embodiments, the received text may also be displayed as subtitles. This may comprise displaying the subtitles only in response to detecting that audio quality is poor at the receive side, e.g. only displaying the lost words or interrupted phrases or sentences. Alternatively, subtitles can also be displayed on-demand or by user-defined rules to allow the user to recall the text of what was recently said, e.g. to allow the receiving user to see the transcript at any time, such as to follow a meeting. The manual mode can be used for accessibility.
-
FIG. 1 schematically illustrates acommunication system 100 in accordance with embodiments disclosed herein. The communication system comprises a plurality of user terminals 102 each configured to connect to acomputer network 101, and each installed with a respective instance of a communication client application 103. Each of the user terminals 102 is also used by at least one respective user 106. InFIG. 1 , two user terminals and their respective clients and users are shown for illustrative purposes: afirst user terminal 102 a installed with afirst communication client 103 a, and asecond user terminal 102 b installed with asecond communication client 103 b (in embodiments, eachclient 103 a, 103 being a respective instance of the same client product). However, it will be appreciated that there may be different numbers of user terminals 102 involved in other scenarios covered by the present disclosure. Thenetwork 101 is preferably a packet-based network. In embodiments it may take the form of a wide-area internetwork such as that commonly referred to as the Internet. Alternatively, thenetwork 101 may take the form of another type of network such as a company intranet, a mobile cellular network, or a wireless local area network (WLAN), or a combination of any of these and/or the Internet. - Each of the user terminals 102 may take any of a variety of different forms, such as a desktop computer, laptop, tablet, smartphone, smart watch, pair of smart glasses, smart TV, or set-top box (and the different user terminals 102 need not necessarily take the same form as one another). Note therefore that the term “computer” as used herein does not restrict to a traditional desktop or laptop computer.
- The communication clients 103 are each installed on computer-readable storage of their respective user terminal 102 and arranged for execution on a respective processing apparatus of the respective user terminal 102. The storage may take the form of one or more storage media (e.g. magnetic memory, electronic memory and/or optical memory) implemented in one or more memory units. The processing apparatus may take the form of one or more processing units. Each of the communications clients 103 is configured so as to be able to establish an audio communication session with one or more others of the communication clients 103 running on one or more of the other respective user terminals 102, e.g. to establish a voice or video call such as a VoIP call. The user of each user terminal 102 is then able to transmit audio (and optionally video) to each other of the users of each other of the user terminals 102 participating in the session.
- By way of illustration, embodiments below may be described from the perspective of the
first user 106 a sending a message to the second user 106 b, but it will be appreciated that similar teachings may also apply in the reverse direction and/or in relation to other combinations of users. - The transmitted audio comprises speech spoken by the sending
user 106 a, captured from an internal or external microphone at transmittinguser terminal 102 a. In particularly advantageous applications of the presently disclosed techniques, the audio is transmitted live, i.e. on-the-fly, as-and-when it is captured, such as in a live VoIP call or the like. The transmitted content may also comprise video (e.g. live video) captured by an internal or external video camera at the transmittinguser terminal 102 a, e.g. a head-and-shoulders shot of the sendinguser 106 a. The captured audio and optional video are encoded (i.e. compressed) by the sendingclient application 103 a and transmitted in encoded form over thenetwork 101 to theclient 103 b on the receivingterminal 102 b. Note therefore that where it is said the audio data comprises the speech of the sendinguser 106 a, or such like, this will be understood to cover the typical scenario in which the audio data is transmitted in encoded (compressed) form, and similarly for any transmitted video. The receivingclient 103 b decodes and plays out the received audio through one or more internal or external speakers at the receivingterminal 102 b, and optionally decodes and plays out any receive video through an internal or external display screen. - In embodiments the
communication system 101 comprises aserver 104 connected to thenetwork 101, arranged to provide a communication service via which the communication session is at least in part conducted. In such embodiments, the audio and optional video from any given one of the users may be sent from theclient 103 a on the sendinguser terminal 102 a to theserver 104 via thenetwork 101, and theserver 104 may be configured to forward the audio and any video onward via thenetwork 101 to theclient 103 b on therecipient user terminal 102 b (or to each of the recipient terminals in a multiparty communication session). Note that a server herein refers to a logical entity which may be physically implemented by one or more server units located at one or more geographical sites. Distributed storage and computing techniques in themselves will be familiar to a person skilled in the art. Note also that it is not essential for the audio and/or video to be transmitted via the server. Instead such content may be transmitted directly between theterminals server 104 may not be required at all (a pure peer-to-peer system), or theserver 104 may still be included in the system to provide one or more supporting functions such as providing authentication certificates, address look-up, or hosting of supplemental information such as contact lists, user profiles, user permission settings, and/or presence statuses. - Note also, in yet further embodiments, the system need not comprise a communication client 103 installed on each user terminal 102. For instance, alternatively one, some or all of the user terminals could instead be installed with a general-purpose web browser operable to access a web-based version of the client application (“web client”) hosted on the
server 104. In such cases the described functionality may be achieved by the web-client rather than the locally installed client (i.e. client installed locally on an individual user terminal 102). Or more generally, the functionality disclosed herein can be implemented by any combination of a local client application 103 (on each user terminal 102) and/or server hosted functionality (e.g. a web client). For conciseness the various options in this respect will not be repeated each time the functionality below is described, but it will be understood that these options apply throughout. -
FIG. 3a gives a schematic mock-up of an example of the front-end user interface (UI) 300 of theclient application 103 b running on the receiving user's terminal 102 b (or of the web client accessed from the receiving user's terminal 102 b). When conditions over thenetwork 101 are good, the receiving user 106 b sees the video of the sendinguser 106 a played out on screen, and the speech captured from the sending user's voice is played out audibly through the speaker(s) of the receivingterminal 102 b. However, when conditions over thenetwork 101 are poor (e.g. high loss or error rate, long delay and/or high jitter), then there may not be sufficient bandwidth to play out the video or even to play out the audio, or at least not with a quality that allows the sending user's speech to be well understood by the receiving user 106 b, or not with a quality that is tolerable for the receiving user 106 b to listen to. - To accommodate such situations or similar, according to the present disclosure the sending
client 103 a is configured with a speech-to-text converter to convert the sending user's locally captured speech to the receivinguser terminal 102 b, e.g. in parallel with the audio but in separate packets. The receivingclient 103 b is configured with a text-to-speech converter comprising a model of the sending user's voice, arranged to convert the received text to synthesized speech at the receive end based on said model, and play this out to the receiving user 106 b. This may be done in response to a detection at the receivingclient 103 b that the network conditions experienced upon reception have fallen below a threshold quality according to a predetermined quality classifier. Alternatively, the text-to-speech conversion could be activated on some other basis, e.g. turned on manually by the receiving user 106 b. - More detail is shown in
FIG. 2 . The transmittinguser terminal 102 a comprises amicrophone 202, anoptional video camera 204, the transmittingclient 103 a, and atransmitter 210 in the form of a transmitting network interface for connecting to thenetwork 101. The transmittingclient 103 a comprises anencoder 206 and a speech-to-text converter 208, the speech-to-text converter 208 comprises a voice model specific to the voice of the transmittinguser 103 a. Alternatively part or all of thevoice model 209 or the speech-to-text converter 208 itself need not be part of theclient application 103 a, but instead theclient application 103 a could be configured to access avoice model 209 or speech-to-text converter 208 present elsewhere on the transmitting user terminal 103 for another purpose, e.g. as part of a dictation application or a speech control software of the operating system. - The
microphone 202 may be internal or external to the housing of the transmittinguser terminal 102 a. Either way, it is arranged to capture audio from its surroundings including speech from the transmittinguser 106 a, and to supply the audio including the captured speech to theencoder 206, which encodes (i.e. compresses) the audio for transmission over thenetwork 101 via the transmittinginterface 210. Similarly thevideo camera 204, if used, may be internal or external to the housing of the transmittinguser terminal 102 a. Either way it is arranged to capture video from its surroundings, e.g. a talking head shot of the transmittinguser 102 a, and to supply this to theencoder 206 which encodes it for transmission over thenetwork 101 via the transmittinginterface 210 as part of the same session (e.g. call) as the transmitted audio. - The speech-to-
text converter 208 is arranged so as to also receive an instance of the captured audio, including the sending user's speech, and to convert this into a textual representation of the same speech content based on themodel 209 of the sending user's voice as maintained at the transmittinguser terminal 102 a. Themodel 209 at the sending side may be trained based on any known speech recognition training techniques, such as by having the sending user reading one or more passage of predetermined text, and/or by the sending user dynamically training themodel 209 by correcting its speech-to-text predictions in an ongoing manner (e.g. during the call or other such session). Themodel 209 may be trained specifically for the purpose of the client application 103 (e.g. VoIP client) and its corresponding communication service (e.g. VoIP service). Alternatively, themodel 209 may be trained and made available for some other or shared purpose, such as for dictation or voice control of the operating system on the transmitting terminal 102 a. In the former case themodel 209 and speech-to-text converter 208 may be a dedicated part of theclient application 103 a. Alternatively in the latter case, themodel 209 or indeed the speech-to-text converter 208 may be part of another application or the operating system (not shown) and merely accessed by theclient application 103 a. A combination of these approaches could also be used. - The speech-to-
text converter 208 is configured to supply the converted text to theencoder 206 to synchronize this converted text with the corresponding audio from which it was generated. This may be achieved by the encoder inserting timestamps into portions of the converted text data that match the timestamps in corresponding portions of the encoded audio and optionally the video, e.g. matching the timestamps of corresponding frames, packets or other such subdivisions of the audio and optional video. Theencoder 206 then sends the timestamped text data over thenetwork 101 via the transmittinginterface 210 as part of the session (e.g. call). The text data may be transmitted in uncompressed form, since it incurs a relatively very small bitrate compared to the audio, or alternatively theencoder 206 may encode the text for transmission as well as the audio and any video. Note therefore that where it is said that the transmitted content comprises text, or the like, this does not exclude the possibility that the text may be sent in encoded (compressed) form. - The transmitting interface may take any suitable form for transmitting the encoded audio and optional video over the network (e.g. Internet) 101, for instance a wireless interface for accessing the
network 101 via a wireless router, wireless access point or a base station of a mobile cellular network; or a wired modem or such like for accessing thenetwork 101 via wired means such as a PSTN phone line, optical cable or Ethernet connection. - As described in relation to the
client 103 a, theencoder 206, speech-to-text converter 208 andspeech model 209 may be implemented in software stored in any suitable storage medium or media of the transmittinguser terminal 102 a, and arranged of be fetched and run on any suitable processing apparatus of the transmitting terminal 102 a. Examples of suitable storage media include electronic storage media such as a solid state drive (SSD) or other EEPROM, magnetic media such as a hard disk or other magnetic drive, and/or optical memory such as a CDROM drive or other optical drive. Examples of suitable processing apparatus include a single or multiple core processing unit, or an arrangement of multiple such processing units. - The receiving
user terminal 102 b comprises one ormore speakers 222, adisplay screen 224, the receivingclient 103 b, and areceiver 212 in the form of a receiving network interface. The receivingclient 103 b comprises adecoder 214, acontroller 216, and a text-to-speech converter 218, the speech-to-text converter 218 comprising avoice model 209′ specific to the sending user's voice maintained at the receivinguser terminal 102 b. Or as an alternative implementation, part or all of thevoice model 209′ or the text-to-speech converter 218 itself could be a separate element to theclient application 103 a. Each of the speaker(s) 222 andscreen 224 may be internal or external to the housing of the receivinguser terminal 102 b. Like the transmittinginterface 210, the receivinginterface 212 may take any suitable form for receiving the encoded audio and optional video over the network (e.g. Internet) 101, for instance a wireless interface for accessing thenetwork 101 via a wireless router, wireless access point or a base station of a mobile cellular network; or a wired interface or modem for accessing thenetwork 101 via wired means such as a PSTN phone line, optical cable or Ethernet connection. - As already described in relation to the receiving
client 103 b, thedecoder 214,controller 216, text-to-speech speech converter 218 andspeech model 209′ may be implemented in software stored in any suitable storage medium or media of the transmittinguser terminal 102 a and arranged of be fetched and run on any suitable processing apparatus of the transmitting terminal 102 a. Again, examples of suitable storage media include electronic storage media such as a solid-state drive (SSD) or other EEPROM, magnetic media such as a hard disk or other magnetic drive, and/or optical memory such as a CDROM drive or other optical drive; and examples of suitable processing apparatus include a single or multiple core processing unit, or an arrangement of multiple such processing units. - The transmitter 201, under control of the
encoder 206, is arranged to send the convertedtext data 254, the encodedaudio data 250 and (optionally) the encodedvideo 252 to the receivinguser terminal 102 b. The receivingterminal 102 b receives these via the receivinginterface 212 and passes theaudio data 252 and (optional)video data 254 to the decoder to decoded and played out via the speaker(s) 222 andscreen 224 respectively. - Further, the
controller 216 is configured to determine when to control the text-to-speech converter 218 to convert the receivedtext data 254 into synthesized speech and to play this synthesized speech out through the speaker(s) 222 in place of the receivedaudio 250. To make this determination, preferably thecontroller 216 is configured detect when the quality of the network connection formed over thenetwork 101 between the transmittinginterface 210 of the transmitting terminal 102 a and the receivinginterface 212 of the receivingterminal 102 b, over which the audio 250 and (optionally) thevideo 252 is received, falls below a predetermined level of quality. This is done based on a quality classifier evaluated by thecontroller 216 based on the connection quality as experienced at the receiveside - Based on any such suitable classifier, the
controller 216 classifies whether, at the current moment in time, its experience of the network conditions is such that the receivedaudio 250 is deemed usable or not usable (e.g. too bad to be understandable or too annoying to be practically usable). If the latter, then in response thecontroller 216 supplies the receivedtext 254 to the text-to-speech converter 218 and controls it to play out the speech synthesized from this text through the speaker(s) 222. Thecontroller 216 also controls thedecoder 214 to temporarily stop playing out the poor quality receivedaudio 250. This continues until such time as thecontroller 216 detects that the connection quality is classified as good again, and then thecontroller 216 controls the text-to-speech converter 218 to stop playing out the synthesized speech and instead controls thedecoder 214 to start playing out the receivedaudio 250 again. Thus, at times the receivedaudio 250 is played out, and at other times the artificially generated audio is played out (but not both at the same time). Thecontroller 216 uses the timestamps in the receivedtext 254 and audio 250 to ensure that the synthesized speech supplied to the text-to-speech converter 218 correctly lines up and fills the gaps between the switched-outaudio 250, i.e. so that the periods of synthesized speech and the periods of playing out the receivedaudio 250 are synchronized. - Note: although it may be preferred that the
controller 216 automatically classifies the quality of the audio and switches between the receivedaudio 250 and synthesized speech in dependence on this, this is not the only possibility. In embodiments the switching could be performed on some other alternative or additional basis, e.g. in response to a user selection input to thecontroller 216 through a user interface of the receivingclient 103 b, selecting to manually to turn on or off the synthesized speech. Thus, the receiving user 106 b may decide for him- or herself whether the receivedaudio 250 is unintelligible or too annoying, and in response to this select to manually switch to the artificial speech instead. - In embodiments, the
decoder 214 may also be arranged to play out some or all of the receivedtext 254 in text form as subtitles. Thecontroller 216 may be arranged to control thedecoder 214 to output the text subtitles only when the connection is classified as poor, as above with regard to the synthesized speech. Alternatively, thedecoder 214 may be arranged to display the subtitles continually throughout the session (e.g. call), even when conditions are classified as good; or thecontroller 216 may be configured to control thedecoder 214 selectively display the subtitles on some other basis, e.g. when turned on by the receiving user 106 b. In further embodiments, thecontroller 216 may be configured so as, when the receiving user activates an associated control in a user interface of theclient application 103 b, to display a history of past-received subtitle text, e.g. all thetext 254 received within a certain predefined time window or a certain user-selected time window. - In embodiments the
text data 254 may be transmitted throughout the entire session (e.g. call), continually in parallel with the corresponding audio, i.e. so the transmitted content always comprises both audio 250 andtext 254 representing the same periods of the session (e.g. call). In this case thecontroller 216 is configured to control the text-to-speech converter to only play out the synthesized speech from certain selected portions of the received text during periods of the session (e.g. call), such as when conditions are classified as poor or when selected by the receiving user 106 b. Alternatively the transmittingclient 103 a may be configured to only transmit thetext data 254 at certain times. For instance, thecontroller 216 may be configured to feed back a report to the transmittingclient 103 a informing it when the network conditions experienced at the receive side are classified as poor. In response, the speech-to-text converter 208 may only transmit the text data during such periods. Theencoder 206 may also be configured to temporarily stop transmitting the audio 250 (and optional video 252) during such periods. Thus, in some alternative embodiments, the transmitted content may comprise only the audio 252 (and optionally video 252) but not thetext data 254 during certain periods of the session (e.g. call), and only thetext data 254 but not the audio 250 (nor video 252) during other periods of the same session. - Either way, the
text 254 is transmitted and received using a communication modality that has a higher robustness than that used for the audio data 250 (andvideo data 252 if present). This exploits the fact that thetext data 254 will be much smaller in size than the audio data 250 (and any video data 252), i.e. incur a much smaller bitrate, and hence a more robust communication means can be afforded even in poor network conditions. - In embodiments, this comprises transmitting and receiving the
text data 254 using a transport layer protocol such as TCP that requires retransmission of unreceived packets, whereas the audio 250 (andvideo 252 if present) is sent and received using a transport layer protocol that does not involve retransmission (e.g. RTP or UDP). Retransmission can be achieved for example (as in TCP) by having the transport layer at the receivingterminal 102 b acknowledge received packets and having the transport layer at the transmitting terminal 102 a retransmit any packets that are not acknowledged as received (the packets being identified by their sequence number). Another possibility is that transport layer at the receiving terminal requests retransmission of missing packets (which it identifies as missing due to missing sequence numbers). - Another alternative or additional possibility for more robust transmission and reception is to send the
text data 254 with a higher service level than the audio data 250 (and any video 252), e.g. a higher QoS (Quality of Service). For instance, the higher service level may be arranged to cause routers over thenetwork 101 to prioritize not dropping the packets oftext data 254 when there is network congestion and the router has to drop packets. - In yet another alternative or additional variant, the
text data 254 may be sent with more redundant information than the audio 250 (and any video 252). E.g. multiple duplicate instances of the same text packets could be sent, or a more robust checksum (incurring more bits) may be included for each packet or other such portion oftext data 254. - Furthermore, according to the particular techniques of the present disclosure, the generation of the synthesized speech is based on a model of the transmit-side user's voice—i.e. specific to the voice of the transmitting
user 106 a—but which is stored, maintained and applied at the receivedside - The receive-
side model 209′ comprises one or more parameters of the transmitting user's voice, e.g. the fundamental pitch (i.e. fundamental frequency), the mean of the fundamental pitch, the variance of the fundamental pitch, and/or spectral information such as in the form of linear prediction coefficients or mel-cepstral coefficients. Various parameters parametrizing the human voice are in themselves known to a person skilled in the art. In embodiments, values of one or more of these parameters may be provided to the receivingterminal 102 b “up front”, either at some point in advance of the session such as when the transmitting and receivingusers client 103 a to the receivingclient 103 b based on the transmit-side model 209 maintained at the transmitting terminal 102 a. - Alternatively, or additionally, in embodiments, the transmitting
client 103 a may be configured to send one or more values of one or more of the voice parameters to the receivingclient 103 b dynamically during the call, along with theaudio stream 250. The receivingclient 103 b can then use these to dynamically adapt the receive-side voice model 209′. For instance, consider the case where, during the call or session, the transmittinguser 106 a makes one or more manual corrections to the converted text generated by the speech-to-text converter 208 based on the transmit-side model 209. This will result in the transmit-side model 209 being trained so as to update the values of one or more of the voice parameters in the transmit-side model 209. In addition, in accordance with embodiments herein, the transmitting client may be configured to transmit the values of the one or more updatedparameters 256 to the receivingclient 103 b during the same session (e.g. call) in which those updates were made. In response, the receivingclient 103 b updates the values of those parameters in the receive-side model 209′, again during the same session (e.g. call). Thus, the receive-side voice model 209′ can be dynamically updated. - Note that where it is said that the receiving
user terminal 102 b orclient 103 b receives one or more speech parameters (or indeed the audio 250,video 252 and text 254) from the transmittinguser terminal 102 a orclient 102 b, this does not exclude that the data in question could be received via theserver 104 or other intermediate element. - As another variant, the one or more parameters may be sent to the receiving
client 103 b from theserver 104 based on values originating from another user terminal of the sendinguser 106 a. For instance, the parameters may be trained by the sendinguser 106 a at some point in advance using a different user terminal other than the sendinguser terminal 102 a that is used during the session or call itself. I.e. the one or more parameters may be received from the transmitting user terminal, or may originate from another user terminal of the transmitting user. Or indeed a combination of such approaches could be used. - Another alternative or additional possibility is that the text-to-
speech converter 218 is configured to train the receive-side voice model 209′ dynamically based on the receivedaudio 250 in conjunction with the receivedtext 254 during periods of good network conditions. This embodiment works whentext 254 andaudio 250 is received for the same content, i.e. same periods of text, such as when thetext 254 is sent continually throughout the whole call. Since the text-to-speech converter 218 has access to the audio 250 andtext 254 for the same content, it is able to train the receiveside model 209 in much the same way as speech recognition software is trained based on the user reading out predetermined passages pre-known to the conversion software. -
FIG. 3b shows a schematic mock-up of anexample user interface 300 of the receivingclient 103 b in circumstances when network conditions are classified as poor. The receivedaudio 250 and anyvideo 252 are no longer played out, and instead the receivedtext 254 is displayed on thescreen 224 aslive subtitles 302. Further, the subtitles are played out as synthesized speech via the text-to-speech converter 218 and speaker(s) 222. In embodiments, thecontroller 216 may be configured to control the user interface to also display thename 304 of the transmittinguser 106 a who speech is currently being displayed as subtitles, which may be useful in multiparty call (i.e. with three or more participants). Further, preferably thecontroller 216 is configured to provide an indication to the receiving user 106 b that the speech being heard is synthesized speech. For instance, this may take the form of avisual indication 306 displayed on thescreen 224 through the graphical user interface of the receiving client, e.g. in the form of an icon or other such graphic, and/or a written message explaining to the receiving user that the currently played out speech is synthesized due to poor network conditions. - Another alternative or additional option for providing such an indication to the receiving user is to provide an audible indication via the speaker(s) 222. One particular way of doing this in the case of two or more speakers is to use stereo or surround sound in order to place the synthesized speech at a distinct spatial position from the perspective of the receiving user 106 b, distinct compared to the position at which the regular received
audio 250 is played out. An example of using stereo would be to play out the receivedaudio 250 centrally, i.e. evenly through both speakers, but to play out the synthesized speech only through one of the twospeakers 222, or weighted toward one of the speakers, thus making it sound like it comes from a direction off to one side. In the case of surround sound (more than two speakers at different positions around the receiving user 106 b ), a head related transfer function (HRTF) may be used to place the synthesized speech at a distinct position, including giving the sensation of forward or backward and/or up or down directions. HRTFs in themselves are known in the art. - It will be appreciated that the above embodiments have been described only by way of example.
- For instance, in alternative implementations the speech-to-text conversion may be implemented at a
server 104 instead of at the transmitting terminal 102 a, with the converted text being forwarded to the receivingterminal 102 b. Alternatively, the text-to-speech could be performed at aserver 104 and the synthesized speech forwarded to receivinguser terminal 102 b. Such implementations are less preferred since the former does not accommodate poor network conditions between the transmittinguser terminal 102 a and theserver 104, whilst the latter does not accommodate poor network conditions between theserver 104 and the receivinguser terminal 102 b. Nonetheless they do still provide some benefit is there are poor conditions on the other leg of the route. And in multiparty scenarios (three or more users), there may be a desired trade-off in storing and applying the voice models in the cloud (i.e. on a server). That is, for N-endpoint calls it may not be desirable to have N̂2 transfers of the voice model (everyone to everyone) but only O(N) transfers. - Further, regarding the measuring and/or classifying of the network conditions, this could also be assisted by a server. In this case the
server 104 may measure one or more factors such as loss, error rate, delay, jitter, etc. and report them to thecontroller 216 on the receivingterminal 102 b which performs the classification based on the reported values. Or theserver 104 could perform the classification and report this to thecontroller 216, in which case thecontroller 216 determines the network conditions simply by receiving the report. - In further embodiments, the communication session is not limited to being a call in the sense of a two-way live communication. Alternatively, the session could be a live one-way communication session such as a remotely-presented lecture or other such presentation, or a live internet TV show. As another possibility the session could also comprise the streaming of pre-recorded audio or video, such as in an internet audio or video streaming service, e.g. internet TV, movie or radio service.
- Other variants or applications of the techniques disclosed herein may become apparent to the person skilled in the art in light of the present disclosure. The scope of the disclosure is not limited by the described embodiments but only by the accompanying claims.
Claims (20)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/177,345 US20190073993A1 (en) | 2017-02-02 | 2018-10-31 | Artificially generated speech for a communication session |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/422,865 US10147415B2 (en) | 2017-02-02 | 2017-02-02 | Artificially generated speech for a communication session |
US16/177,345 US20190073993A1 (en) | 2017-02-02 | 2018-10-31 | Artificially generated speech for a communication session |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/422,865 Continuation US10147415B2 (en) | 2017-02-02 | 2017-02-02 | Artificially generated speech for a communication session |
Publications (1)
Publication Number | Publication Date |
---|---|
US20190073993A1 true US20190073993A1 (en) | 2019-03-07 |
Family
ID=62980166
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/422,865 Active US10147415B2 (en) | 2017-02-02 | 2017-02-02 | Artificially generated speech for a communication session |
US16/147,838 Active US10930262B2 (en) | 2017-02-02 | 2018-09-30 | Artificially generated speech for a communication session |
US16/177,345 Abandoned US20190073993A1 (en) | 2017-02-02 | 2018-10-31 | Artificially generated speech for a communication session |
Family Applications Before (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/422,865 Active US10147415B2 (en) | 2017-02-02 | 2017-02-02 | Artificially generated speech for a communication session |
US16/147,838 Active US10930262B2 (en) | 2017-02-02 | 2018-09-30 | Artificially generated speech for a communication session |
Country Status (1)
Country | Link |
---|---|
US (3) | US10147415B2 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11763157B2 (en) | 2019-11-03 | 2023-09-19 | Microsoft Technology Licensing, Llc | Protecting deep learned models |
Families Citing this family (24)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10897433B2 (en) * | 2017-03-27 | 2021-01-19 | Orion Labs | Bot group messaging using general voice libraries |
US11495232B2 (en) * | 2017-04-20 | 2022-11-08 | Telefonaktiebolaget Lm Ericsson (Publ) | Handling of poor audio quality in a terminal device |
EP3396964B1 (en) | 2017-04-25 | 2020-07-22 | Accenture Global Solutions Ltd | Dynamic content placement in a still image or a video |
EP3528196A1 (en) * | 2018-02-16 | 2019-08-21 | Accenture Global Solutions Limited | Dynamic content generation |
US10972459B2 (en) | 2018-04-24 | 2021-04-06 | Apple Inc. | Identity credential verification techniques |
US11093597B2 (en) * | 2018-04-24 | 2021-08-17 | Apple Inc. | Identity credential verification techniques |
US11398218B1 (en) * | 2018-04-26 | 2022-07-26 | United Services Automobile Association (Usaa) | Dynamic speech output configuration |
US20190332899A1 (en) * | 2018-04-26 | 2019-10-31 | Sorenson Ip Holdings, Llc | Analysis of image media corresponding to a communication session |
US11315553B2 (en) * | 2018-09-20 | 2022-04-26 | Samsung Electronics Co., Ltd. | Electronic device and method for providing or obtaining data for training thereof |
US11423073B2 (en) | 2018-11-16 | 2022-08-23 | Microsoft Technology Licensing, Llc | System and management of semantic indicators during document presentations |
JP7385381B2 (en) * | 2019-06-21 | 2023-11-22 | 株式会社日立製作所 | Abnormal sound detection system, pseudo sound generation system, and pseudo sound generation method |
US11595462B2 (en) * | 2019-09-09 | 2023-02-28 | Motorola Mobility Llc | In-call feedback to far end device of near end device constraints |
CN111916073B (en) * | 2020-06-22 | 2023-10-24 | 深圳追一科技有限公司 | Robot outbound control method and device, server, and computer-readable storage medium |
US11356492B2 (en) | 2020-09-16 | 2022-06-07 | Kyndryl, Inc. | Preventing audio dropout |
CN112202803A (en) * | 2020-10-10 | 2021-01-08 | 北京字节跳动网络技术有限公司 | Audio processing method, device, terminal and storage medium |
US11783804B2 (en) * | 2020-10-26 | 2023-10-10 | T-Mobile Usa, Inc. | Voice communicator with voice changer |
US11044287B1 (en) * | 2020-11-13 | 2021-06-22 | Microsoft Technology Licensing, Llc | Caption assisted calling to maintain connection in challenging network conditions |
US11606465B2 (en) | 2020-12-16 | 2023-03-14 | Rovi Guides, Inc. | Systems and methods to automatically perform actions based on media content |
US11595278B2 (en) * | 2020-12-16 | 2023-02-28 | Rovi Guides, Inc. | Systems and methods to automatically perform actions based on media content |
US11749079B2 (en) | 2020-12-16 | 2023-09-05 | Rovi Guides, Inc. | Systems and methods to automatically perform actions based on media content |
US11580954B2 (en) * | 2021-02-03 | 2023-02-14 | Qualcomm Incorporated | Systems and methods of handling speech audio stream interruptions |
US12003949B2 (en) * | 2022-01-19 | 2024-06-04 | Meta Platforms Technologies, Llc | Modifying audio data transmitted to a receiving device to account for acoustic parameters of a user of the receiving device |
CN114630144A (en) * | 2022-03-03 | 2022-06-14 | 广州方硅信息技术有限公司 | Audio replacement method, system and device in live broadcast room and computer equipment |
US20230394886A1 (en) * | 2022-06-05 | 2023-12-07 | Apple Inc. | Providing personalized audio |
Citations (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20020103656A1 (en) * | 2001-01-30 | 2002-08-01 | Bahler Lawrence G. | Automatic confirmation of personal notifications |
US20020156900A1 (en) * | 2001-03-30 | 2002-10-24 | Brian Marquette | Protocol independent control module |
US20030204596A1 (en) * | 2002-04-29 | 2003-10-30 | Satyendra Yadav | Application-based network quality of service provisioning |
US20040143438A1 (en) * | 2003-01-17 | 2004-07-22 | International Business Machines Corporation | Method, apparatus, and program for transmitting text messages for synthesized speech |
US20050099992A1 (en) * | 2001-12-07 | 2005-05-12 | Masanori Sato | Data communication control system, transmitter, and transmitting method |
US7236580B1 (en) * | 2002-02-20 | 2007-06-26 | Cisco Technology, Inc. | Method and system for conducting a conference call |
US20090198497A1 (en) * | 2008-02-04 | 2009-08-06 | Samsung Electronics Co., Ltd. | Method and apparatus for speech synthesis of text message |
US7706510B2 (en) * | 2005-03-16 | 2010-04-27 | Research In Motion | System and method for personalized text-to-voice synthesis |
US20110093272A1 (en) * | 2008-04-08 | 2011-04-21 | Ntt Docomo, Inc | Media process server apparatus and media process method therefor |
US8120638B2 (en) * | 2006-01-24 | 2012-02-21 | Lifesize Communications, Inc. | Speech to text conversion in a videoconference |
US20140214426A1 (en) * | 2013-01-29 | 2014-07-31 | International Business Machines Corporation | System and method for improving voice communication over a network |
US20150170651A1 (en) * | 2013-12-12 | 2015-06-18 | International Business Machines Corporation | Remedying distortions in speech audios received by participants in conference calls using voice over internet (voip) |
US20160210960A1 (en) * | 2014-08-06 | 2016-07-21 | Lg Chem, Ltd. | Method of outputting content of text data to sender voice |
Family Cites Families (35)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5561736A (en) | 1993-06-04 | 1996-10-01 | International Business Machines Corporation | Three dimensional speech synthesis |
US5724410A (en) | 1995-12-18 | 1998-03-03 | Sony Corporation | Two-way voice messaging terminal having a speech to text converter |
US6151572A (en) | 1998-04-27 | 2000-11-21 | Motorola, Inc. | Automatic and attendant speech to text conversion in a selective call radio system and method |
US6490550B1 (en) | 1998-11-30 | 2002-12-03 | Ericsson Inc. | System and method for IP-based communication transmitting speech and speech-generated text |
US6513003B1 (en) | 2000-02-03 | 2003-01-28 | Fair Disclosure Financial Network, Inc. | System and method for integrated delivery of media and synchronized transcription |
US7117152B1 (en) | 2000-06-23 | 2006-10-03 | Cisco Technology, Inc. | System and method for speech recognition assisted voice communications |
FR2814625B1 (en) | 2000-09-25 | 2003-01-03 | Prosodie | TELEPHONY SYSTEM WITH SUBTITLETING AND / OR TRANSLATION |
US20030097262A1 (en) | 2001-11-20 | 2003-05-22 | Gateway, Inc. | Handheld device having speech-to text conversion functionality |
US20030125947A1 (en) * | 2002-01-03 | 2003-07-03 | Yudkowsky Michael Allen | Network-accessible speaker-dependent voice models of multiple persons |
US7050553B2 (en) | 2003-11-20 | 2006-05-23 | Lucent Technologies Inc. | Communication device that provides enhanced services |
US7917178B2 (en) | 2005-03-22 | 2011-03-29 | Sony Ericsson Mobile Communications Ab | Wireless communications device with voice-to-text conversion |
US8542803B2 (en) | 2005-08-19 | 2013-09-24 | At&T Intellectual Property Ii, L.P. | System and method for integrating and managing E-mail, voicemail, and telephone conversations using speech processing techniques |
US7657006B2 (en) | 2005-12-15 | 2010-02-02 | At&T Intellectual Property I, L.P. | Messaging translation services |
US8160210B2 (en) | 2007-01-08 | 2012-04-17 | Motorola Solutions, Inc. | Conversation outcome enhancement method and apparatus |
US8886537B2 (en) * | 2007-03-20 | 2014-11-11 | Nuance Communications, Inc. | Method and system for text-to-speech synthesis with personalized voice |
US8180029B2 (en) * | 2007-06-28 | 2012-05-15 | Voxer Ip Llc | Telecommunication and multimedia management method and apparatus |
US20090190031A1 (en) | 2008-01-28 | 2009-07-30 | Herve Guihot | Television apparatus and method for subtitle handling |
US8345849B2 (en) | 2008-05-27 | 2013-01-01 | Avaya Inc. | Telecommunications endpoint that prompts a user to focus on a monitored call |
US20090326939A1 (en) | 2008-06-25 | 2009-12-31 | Embarq Holdings Company, Llc | System and method for transcribing and displaying speech during a telephone call |
US8649494B2 (en) | 2008-08-05 | 2014-02-11 | International Business Machines Corporation | Participant alerts during multi-person teleconferences |
US9871916B2 (en) | 2009-03-05 | 2018-01-16 | International Business Machines Corporation | System and methods for providing voice transcription |
US8265671B2 (en) | 2009-06-17 | 2012-09-11 | Mobile Captions Company Llc | Methods and systems for providing near real time messaging to hearing impaired user during telephone calls |
CN102483917B (en) | 2009-07-15 | 2016-03-16 | 谷歌公司 | For the order of display text |
EP2299440B1 (en) | 2009-09-11 | 2012-10-31 | Vodafone Holding GmbH | Method and Device for automatic recognition of given keywords and/or terms within voice data |
US8583431B2 (en) | 2011-08-25 | 2013-11-12 | Harris Corporation | Communications system with speech-to-text conversion and associated methods |
EP2566144B1 (en) | 2011-09-01 | 2017-05-03 | BlackBerry Limited | Conferenced voice to text transcription |
US9031382B1 (en) | 2011-10-20 | 2015-05-12 | Coincident.Tv, Inc. | Code execution in complex audiovisual experiences |
US9230546B2 (en) | 2011-11-03 | 2016-01-05 | International Business Machines Corporation | Voice content transcription during collaboration sessions |
EP2621140A1 (en) | 2012-01-24 | 2013-07-31 | Alcatel Lucent | Media enrichment for a call in a communication network |
AU2013221345A1 (en) * | 2012-02-16 | 2014-08-28 | Covidien Lp | Multifunctional conferencing systems and methods |
US20150255068A1 (en) * | 2014-03-10 | 2015-09-10 | Microsoft Corporation | Speaker recognition including proactive voice model retrieval and sharing features |
US9338302B2 (en) | 2014-05-01 | 2016-05-10 | International Business Machines Corporation | Phone call playback with intelligent notification |
US9558733B1 (en) * | 2014-09-29 | 2017-01-31 | Amazon Technologies, Inc. | Audibly indicating secondary content with spoken text |
US20160165350A1 (en) * | 2014-12-05 | 2016-06-09 | Stages Pcs, Llc | Audio source spatialization |
US10304463B2 (en) * | 2016-10-03 | 2019-05-28 | Google Llc | Multi-user personalization at a voice interface device |
-
2017
- 2017-02-02 US US15/422,865 patent/US10147415B2/en active Active
-
2018
- 2018-09-30 US US16/147,838 patent/US10930262B2/en active Active
- 2018-10-31 US US16/177,345 patent/US20190073993A1/en not_active Abandoned
Patent Citations (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20020103656A1 (en) * | 2001-01-30 | 2002-08-01 | Bahler Lawrence G. | Automatic confirmation of personal notifications |
US20020156900A1 (en) * | 2001-03-30 | 2002-10-24 | Brian Marquette | Protocol independent control module |
US20050099992A1 (en) * | 2001-12-07 | 2005-05-12 | Masanori Sato | Data communication control system, transmitter, and transmitting method |
US7236580B1 (en) * | 2002-02-20 | 2007-06-26 | Cisco Technology, Inc. | Method and system for conducting a conference call |
US20030204596A1 (en) * | 2002-04-29 | 2003-10-30 | Satyendra Yadav | Application-based network quality of service provisioning |
US20040143438A1 (en) * | 2003-01-17 | 2004-07-22 | International Business Machines Corporation | Method, apparatus, and program for transmitting text messages for synthesized speech |
US7706510B2 (en) * | 2005-03-16 | 2010-04-27 | Research In Motion | System and method for personalized text-to-voice synthesis |
US8120638B2 (en) * | 2006-01-24 | 2012-02-21 | Lifesize Communications, Inc. | Speech to text conversion in a videoconference |
US20090198497A1 (en) * | 2008-02-04 | 2009-08-06 | Samsung Electronics Co., Ltd. | Method and apparatus for speech synthesis of text message |
US20110093272A1 (en) * | 2008-04-08 | 2011-04-21 | Ntt Docomo, Inc | Media process server apparatus and media process method therefor |
US20140214426A1 (en) * | 2013-01-29 | 2014-07-31 | International Business Machines Corporation | System and method for improving voice communication over a network |
US20150170651A1 (en) * | 2013-12-12 | 2015-06-18 | International Business Machines Corporation | Remedying distortions in speech audios received by participants in conference calls using voice over internet (voip) |
US20160210960A1 (en) * | 2014-08-06 | 2016-07-21 | Lg Chem, Ltd. | Method of outputting content of text data to sender voice |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11763157B2 (en) | 2019-11-03 | 2023-09-19 | Microsoft Technology Licensing, Llc | Protecting deep learned models |
Also Published As
Publication number | Publication date |
---|---|
US10147415B2 (en) | 2018-12-04 |
US20180218727A1 (en) | 2018-08-02 |
US20190035383A1 (en) | 2019-01-31 |
US10930262B2 (en) | 2021-02-23 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10930262B2 (en) | Artificially generated speech for a communication session | |
US8089502B2 (en) | Adaptive video telephone system | |
US9509953B2 (en) | Media detection and packet distribution in a multipoint conference | |
US7822050B2 (en) | Buffering, pausing and condensing a live phone call | |
JP5811641B2 (en) | TV conference system, video transmission method, and video transmission apparatus | |
CN114616606A (en) | Multi-device conferencing with improved destination playback | |
US9363480B2 (en) | Obtaining replay of audio during a conference session | |
JP2005318534A (en) | Method and apparatus for holding conference for perming stream selection | |
US7619645B2 (en) | Audio visual media encoding system | |
WO2008114087A2 (en) | Method of transmitting data in a communication system | |
JP6278275B2 (en) | TRANSMISSION DEVICE, RECEPTION DEVICE, TRANSMISSION METHOD, AND RECEPTION METHOD | |
KR101597768B1 (en) | Interactive multiparty communication system and method using stereophonic sound | |
JP4218456B2 (en) | Call device, call method, and call system | |
JP2016181856A (en) | Conference system | |
Perkins et al. | Multicast audio: The next generation | |
US11741933B1 (en) | Acoustic signal cancelling | |
US20240121342A1 (en) | Conference calls | |
JP2018137614A (en) | Communication device, communication system, communication method, and program | |
JP2009055469A (en) | Transmission terminal | |
WO2023170677A1 (en) | Acoustic signal cancelling | |
CN117650932A (en) | Multilayer safe real-time audio and video method and device | |
JP2005045737A (en) | Apparatus, method and system for telephone communication | |
Yeo | Finding perceptually optimal operating points of a real time interactive video-conferencing system | |
KR20030057505A (en) | Multimedia data transfer system using Real-time Transport Protocol | |
JP2001069123A (en) | Equpment and method for multimedia data communication |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: MICROSOFT TECHNOLOGY LICENSING, LLC, WASHINGTON Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:CUTLER, ROSS G.;SRINIVASAN, SRIRAM;MEHRAN, RAMIN;AND OTHERS;SIGNING DATES FROM 20170208 TO 20170221;REEL/FRAME:047375/0370 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: APPLICATION DISPATCHED FROM PREEXAM, NOT YET DOCKETED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |