EP3533196B1 - Sicherung von audiodaten - Google Patents

Sicherung von audiodaten Download PDF

Info

Publication number
EP3533196B1
EP3533196B1 EP17757936.4A EP17757936A EP3533196B1 EP 3533196 B1 EP3533196 B1 EP 3533196B1 EP 17757936 A EP17757936 A EP 17757936A EP 3533196 B1 EP3533196 B1 EP 3533196B1
Authority
EP
European Patent Office
Prior art keywords
time
limited period
access
audio information
processing module
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP17757936.4A
Other languages
English (en)
French (fr)
Other versions
EP3533196A1 (de
Inventor
Lee Campbell
Samuel Kramer Beder
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Google LLC
Original Assignee
Google LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Google LLC filed Critical Google LLC
Publication of EP3533196A1 publication Critical patent/EP3533196A1/de
Application granted granted Critical
Publication of EP3533196B1 publication Critical patent/EP3533196B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/16—Sound input; Sound output
    • G06F3/165—Management of the audio stream, e.g. setting of volume, audio stream path
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/16—Sound input; Sound output
    • G06F3/167—Audio in a user interface, e.g. using voice commands for navigating, audio feedback
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L17/00—Speaker identification or verification techniques
    • G10L17/22—Interactive procedures; Man-machine interfaces
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00—Network architectures or network communication protocols for network security
    • H04L63/04—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks
    • H04L63/0428—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/08—Speech classification or search
    • G10L2015/088—Word spotting
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
    • G10L2015/223—Execution procedure of a spoken command
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W12/00—Security arrangements; Authentication; Protecting privacy or anonymity
    • H04W12/02—Protecting privacy or anonymity, e.g. protecting personally identifiable information [PII]

Definitions

  • the present specification relates generally to security techniques for handling audio data.
  • Patent document US2013/0095805 A1 discloses a computer-implemented method that includes detecting a current context associated with a mobile computing device and determining, based on the current context, whether to switch the mobile computing device from a current mode of operation to a second mode of operation during which the mobile computing device monitors ambient sounds for voice input that indicates a request to perform an operation. The method can further include, in response to determining whether to switch to the second mode of operation, activating one or more microphones and a speech analysis subsystem associated with the mobile computing device so that the mobile computing device receives a stream of audio data.
  • security can be enhanced by restricting the ability of a device to access audio data that the device acquires.
  • audio detected by a microphone of a device may be encrypted or otherwise made inaccessible to the device.
  • the device can be granted authorization to access audio detected during a limited period of time. During this limited period of time, audio detected by the microphone is not encrypted, allowing the device to access the audio.
  • a secure module of the device can switch between providing encrypted audio and unencrypted audio depending on the context of the device.
  • various techniques can enable the device to receive and respond to voice commands, even while the audio is encrypted so the device cannot access the audio for the command.
  • a device can include a processing module connected to a microphone.
  • the processing module can be configured to control access to the audio received at the microphone by the device. Specifically, the processing module can restrict access to detected audio by encrypting audio data in a manner that the device cannot access it.
  • the processing module can accept a token from a trusted source, e.g., a remote server that the device communicates with through a network such as the Internet.
  • the token can authorize the device to access the unencrypted audio.
  • the processing module allows the device to receive unencrypted audio data.
  • the token can indicate a predetermined amount of time during which audio data can be detected and passed unencrypted to the device.
  • the processing module can authorize access to the audio by the device for a limited period of time, and restrict access to the audio once the limited period of time has ended.
  • One innovative aspect of the subject matter described in this specification is embodied in methods that include the actions of restricting, by a processing module of a device, access by the device to audio information detected by a microphone, receiving, by the processing module, data indicating that the device is authorized to access audio information detected by the microphone during a limited period of time, and in response to receiving data indicating that the device is authorized to access audio information detected by the microphone during the limited period of time, providing audio information to the device.
  • the methods can also include the actions of monitoring, by the processing module, audio information detected by the microphone during the limited period of time for the presence of a hotword, and after the end of the limited period of time, restricting, by the processing module, access by the device to audio information detected by the microphone.
  • a method, a device, and a system for restricting access to audio data detected by a microphone of the device, to prevent unauthorized snooping on or access to the audio data from the device When the device processor receives data indicating that audio access is authorized for a limited period of time, the processor allows the device to receive the audio data. When the time period has ended, audio access is (once again) restricted. During authorized audio access, the processor also monitors the audio data for the presence of a hotword.
  • the hotword may comprise a predetermined word or phrase, indicating that a command or query is about to be provided to the device.
  • the processor may encrypt the command or query for transmission to a remote server for secure processing.
  • the device processor may then return to allowing the device to receive the audio data if the limited period of time has not expired.
  • Implementations may each optionally include one or more of the following features.
  • the methods can include restricting access to audio information detected by the microphone in response to determining that the limited period of time has ended.
  • the methods can also include receiving a token indicating (i) that the device is authorized to access the audio information detected by the microphone during the limited period of time and (ii) a duration of the limited period of time, evaluating the token to verify a validity of the token, and in response to verifying the validity of the token, extracting data of the token indicating (i) that the device is authorized to access the audio information detected by the microphone during the limited period of time and (ii) a duration of the limited period of time.
  • the methods can include at least one of encrypting the audio information detected by the microphone, or blocking access of the audio information from the processing module to the device.
  • the methods can include detecting the presence of the hotword during the limited period of time, and in response to detecting the presence of the hotword during the limited period of time, restricting, during the limited period of time, access by the device to audio information that is detected by the microphone and that follows the hotword.
  • the methods can include detecting, during the limited period of time, a presence of a command that follows the hotword, determining that the command has ended during the limited period of time, and in response to determining that the command has ended during the limited period of time, providing audio information to the device until the end of the limited period of time.
  • the methods can also include after determining that the command has ended during the limited period of time, detecting, during the limited period of time, a presence of a question that occurs before another hotword is detected, and in response to detecting the presence of the question that occurs before another hotword is detected during the limited period of time, providing audio information to the device until the end of the limited period of time. Additionally, the methods can include receiving a request to access audio information detected by the microphone, wherein the request is provided by the device in response to a trigger of the device. In this instance, the methods can include associating the request with a value, or a data string, derived from an identifier for the processing module and a time.
  • the technology enables a device to seamlessly transition between a first mode that allows access to unencrypted audio data and a second mode that restricts access by encrypting audio data.
  • the transition can be made based on a context of the device when audio is received by the device.
  • the device is able to seamlessly transition between providing unencrypted audio data and providing encrypted audio data using trusted sources such as a processing module located at the device and a remote server.
  • the processing module can be configured to control transitions based on the current context or situation.
  • the encryption of audio can be used as a default or baseline operating mode. This provides enhanced security. If a device were hacked or otherwise compromised, an attacker could potentially obtain all information on the device, including audio data being generated based on microphone input. However, by encrypting audio during standard operation, the encryption of the audio data would prevent an attacker from eavesdropping on the detected audio.
  • a device can include a secure processing module, separate from the host processor of the device, that directly receives audio information and controls access to audio data.
  • the processing module can facilitate secure processing of voice commands by detecting hotwords that signal voice actions, such as spoken commands and queries. For example, when a hotword is detected, the processing module can cause encrypted audio data to be sent to a server system.
  • the server system can process the audio and instruct the device to perform the command requested by the user. In this manner, voice commands are accepted and performed by a device, while the device cannot intercept or alter the audio of the voice command.
  • the processing module can provide raw or unencrypted audio for a limited period of time, after verifying that the device is authorized to access to the audio. At the end of the limited period of time, the processing module can transition back to encrypting audio that is subsequently received.
  • This technique increases confidentiality of received audio data in non-trusted devices through the use of a trusted processing module located at the device and through communication with a trusted, remote server. As such, the technique can be used to mitigate malicious attacks in which attackers seek to listen in on in-room audio and/or transmit malicious commands from non-trusted devices to trusted servers.
  • FIG. 1 is a diagram of a system 100 for securing audio data.
  • the system can make context-based switches between providing encrypted audio data and unencrypted audio data.
  • the system 100 enables audio to be encrypted to enhance security.
  • the system 100 also enables unencrypted audio to be provided at certain times, when appropriate for a device to carry out legitimate functions.
  • the system 100 includes a device 120 and a server 140 that communicate via a network 130.
  • the network 130 can include one or more public or private networks and may include the Internet.
  • the device 120 may communicate over the network 130 using one or more wired or wireless networks such as Ethernet, a LAN, a WAN, Wi-Fi, BLUETOOTH, and the like.
  • the device 120 can be any of various types of devices such as a desktop, a laptop, a smartphone, a tablet, and the like.
  • the device 120 is a device such as an intercom system, a baby monitor, a karaoke machine, or other device that uses microphone input during its operation.
  • the device 120 includes a processor 128 that handles the primary functions of the device 120.
  • the device 120 can also include a microphone 115 for receiving raw audio 110.
  • the microphone 115 can represent one or more microphones that are integrated in the device 120 and/or are in communication with the device 120.
  • the device 120 includes a separate processing module 122 that manages various audio-related processing tasks.
  • the processing module 122 can be a voice assistant module that enables the device 120 to accept and respond to voice actions, such as voice commands and voice queries. For example, the processing module 122 can determine that a hotword was spoken, and designate audio data occurring after the hotword for transmission to the server 140.
  • the processing module 122 can include a hotword module 123 to detect the presence of a hotword, e.g., a predetermined key word or phrase, in received audio.
  • the processing module 124 may also include a question module 124 to detect whether a spoken follow-up request has occurred without being immediately preceded by a hotword.
  • the processing module 122 can be a secured hardware module that is integrated with the host device 202.
  • the processing module 122 may be implemented as, for example, one or more semiconductor chips, a card, a circuit board, or other electronic module.
  • the operations of the processing module 122 can be performed using analog and/or digital circuitry.
  • the processing module 122 can include a general purpose processor, a digital signal processor, an analog to digital converter, and/or other processing elements.
  • the processing module 122 can include one or more data storage devices, including volatile or nonvolatile memory, which can store software, firmware, settings, data, and so on.
  • the processing module 122 is provided by a third party and operates in a different security domain from the device 120. As a result, even if the device 120 is compromised, the processing module 122 operates independently to encrypt audio, securely handle voice commands, and decide when the device 120 is permitted to receive unencrypted audio data. In some implementations, the processing module 122 directly receives audio input received through a microphone 115 of the device 120. In other words, detected audio for the device 120 may be routed through the processing module 122, and the device 120 may be blocked from accessing the detected audio unless permitted by the processing module 122.
  • the processing module 122 can include a cryptography module 125 to encrypt audio data.
  • the processing module 122 can also store a module identifier 126. In some implementations, the module identifier 126 is a unique identifier that distinguishes the processing module 122 from all other processing modules of other devices.
  • the processing module 122 does not include a direct network interface for communicating with the server 140. Rather, the processing module 122 can communicate with the processor 128 and rely on the device 120 to transmit and receive messages for the processing module 122.
  • the device 120 can include a user interface, sensor, or element that generates trigger data 121 for indicating a requested transition between encrypted and unencrypted audio.
  • an intercom system may include a button that a person may press to initiate a call. When a user presses the button, the device 120 may generate trigger data 121 indicating that the device 120 needs to access detected audio from the microphone 115.
  • the processing module 122 can encrypt audio data detected by the microphone 115 as a default or baseline operating mode. In this manner, the processing module 122 can be configured to restrict access to audio data that is received by the microphone, even from the device 120 that includes the processing module 122. In the example of Fig. 1 , as a default or baseline operating state, the processing module 122 encrypts all received audio to restrict access to the received audio by the device 120.
  • the processing module 122 can communicate with the server 140 to determine whether the device 120 should be granted access to detected audio. For example, each time the device 120 needs to access audio from the microphone, the device 120 can send a request for access to received audio to the server 140. If the device 120 is authorized, the processing module 122 can then receive a token from the server 140 indicating that device 120 is authorized to receive the audio. The processing module can be configured to verify the authenticity of the received token, and provide unencrypted audio to the device 120 in response to verifying the token. After the limited period of time ends, the processing module 122 can automatically stop providing unencrypted audio to the device 120, and resume encrypting audio that is detected.
  • audio 110 is received at the microphone 115 of the device 120.
  • the audio 110 can include raw audio that is received at the microphone 115.
  • the audio 110 can include various types of audio in the environment of the device 120.
  • the audio 110 can be transmitted directly to the processing module 122 for handling.
  • the processing module 122 does not provide audio to the device 120 or the server 140.
  • the processing module 122 can monitor audio for the presence of a hotword that signals the beginning of a voice action, such as a voice command or voice query. If a hotword is detected, the processing module 122 can encrypt the subsequent audio and have the processor 128 of the device 120 send the encrypted audio data to the server 140.
  • the server 140 then processes the audio to initiate commands requested by the user.
  • the processing module 122 can be configured to initially encrypt all received audio information that leaves the processing module 122 so that the device 120 cannot access detected audio.
  • the device 120 receives or generates trigger data 121 indicating that the device 120 needs access to audio data obtained using the microphone 115.
  • the trigger data 121 can indicate a condition or event, such as a user input to a button, a slider, a touchscreen, or other user interface element.
  • the device 120 may be an intercom device, and the trigger data 121 may indicate that a person pressed a button to initiate a call. Based on the trigger data 121, the device 120 determines that it needs access to audio from the microphone 115.
  • the processor 128 in response to the trigger data 121, the processor 128 generates a request 135 for access to detected audio.
  • the request 135 can be a request for a token that enables the device 120 to access audio data for a limited period of time.
  • the processor 128 sends a request 127 to the processing module 122 to obtain information.
  • the processing module 122 can generate and provide identifying data 129 or other data to be included in the request 135.
  • the identifying data 129 can indicate or be derived from the module identifier 126 for the processing module 122.
  • the identifying data 129 may also include or be derived from a time, such as the current time.
  • the processing module 122 may generate the identifying data 129 by encrypting a set of data that includes the module identifier 126 and other information that varies, such as time.
  • the module identifier 126 indicates to the server 140 which processing module 122 the request 135 corresponds to.
  • the use of information that varies, such as the current time, ensures that the encrypted data changes for each session. Further, providing the time of the request 135 helps to prevent replay attacks, since identifying data 129 that is later reused would show a time that is not current.
  • the identifying data 129 can be encrypted or otherwise securely encoded, e.g., using a signature or hash, so that the identifying data 129 cannot be read, or at least cannot be altered without the change being detectible. Because information from the processing module 122 is sent through the device 120, which is assumed to be insecure, communications from the processing module 122 to the server 140 generally can use encryption or other techniques so that the server 140 can identify the device 120 and avoid or detect tampering.
  • the device 120 sends the request 135, including the identifying data 129 from the processing module 122, to the server 140 over the network 130.
  • the server 140 can include one or more servers that are remote from the device 120.
  • the server 140 can be a trusted system that is configured to receive the request 135.
  • the server 140 can be configured to grant or deny the request 135 based on the identifying data 129 in the request 135, known characteristics of the device 120, and/or other information.
  • the server 140 evaluates the request 135 and generates a response that grants or denies access to received audio.
  • the response can be a secure token 144 that indicates that the device 120 is authorized to access audio from the microphone 115. If accessing the audio is not appropriate, the server 140 can provide a response that indicates that access should be denied. In some implementations, the response denying access may be provided in the form of a secure token.
  • the server 140 can perform a number of determinations to assess whether access should be granted. For example, the server 140 can verify the integrity of the request 135. The server 140 can decrypt or otherwise decode the identifying data 129 in the request 135, and extract information used for verification. As discussed above, this identifying data 129 can indicate, for example, the module identifier 126, the time the request for access is made, or other information. The encryption of this information with a key known to both the server 140 and the processing module 122 can demonstrate validity of the identifying data 129 and thus the request 135. Similarly, the identifying data may include an accompanying hash or signature that the server 140 may use to verify the authenticity of the request 135.
  • the server 140 may use information from the identifying data 129 to determine whether the request should be granted. For example, the server 140 may extract a time from the identifying data 129 or another part of the request 135. The server 140 may compare the extracted time to the time the request 135. If the time that the request 135 was received by the server 140 is within a threshold amount of time from the extracted time, e.g., within 15 seconds, within 1 minute, etc., the server 140 may determine that the request 135 is current. If the server 140 received the request 135 more than a maximum amount of time after the extracted time, than the server 140 determines that the request 135 is invalid and should be denied.
  • a threshold amount of time from the extracted time e.g., within 15 seconds, within 1 minute, etc.
  • the server 140 can extract the module identifier 126 from the identifying data 129 and use the module identifier 126 to look up information about the device 120 that includes the particular processing module 122. For example, the server 140 can access device data 132 that maps processing module identifiers to corresponding device types. Other information, such as a model number of the device 120, a manufacturer of the device, and capabilities of the device 120 may also be stored in the device data 132. The device data 132 may be populated using information from device manufacturers, e.g., to indicate that certain sets of processing modules are being associated with certain device types.
  • Devices may be required to be of a certain type in order for access to audio data to be granted.
  • Some types of devices may include the processing module 122 to permit voice assistant functionality, but have no need to use detected audio for their operation. For example, it may be desirable for a smart clothes washing machine to respond to user voice requests, but the washing machine does not require audio to operate.
  • the server 140 may deny access to audio data as a matter of course.
  • other devices such as intercoms, baby monitors, karaoke machines, and the like may require detected audio for purposes other than responding to user voice requests.
  • the server 140 can use the device data 132 to verify that a request comes from an appropriate device that has a legitimate use for audio as part of the evaluation of the request 135. In the example of FIG. 1 , the server determines that the device 120 is an intercom, and so the server 140 determines that the device 120 is an appropriate type of device to receive audio data.
  • the server 140 may perform other determinations when evaluating a request 135. For example, the server 140 may access user account data and identify a user account that the device 120 is associated with. For example, the server 140 may look up the module identifier 126 extracted from the request 135.
  • the user account may indicate various types of information, including preferences of the user, security levels or permissions granted to devices, the status or locations of other devices, access histories, and so on that the server 140 may use to evaluate the request 135.
  • the request 135 may include information indicating a state of the device 120 or a requested use of the audio data. These also may be evaluated by the server 140 to determine whether access is appropriate.
  • the server 140 determines that the request 135 is authentic, current (e.g., not delayed or repeated), and is for an appropriate type of device, the server 140 generates a token 144 that indicates that the device 120 should be granted access to audio data.
  • the token 144 can indicate a limited period of time 145 that the device 120 may obtain access to audio data. This limited period of time may be indicated as an expiration time, a length of time, a time range, or in another manner.
  • the server 140 may indicate that the device 120 should receive access for, e.g., only one minute, or two minutes. The amount of time that authorization lasts may be based on the type of device 120.
  • the server 140 may determine that a karaoke machine is authorized for a longer amount of time, such as 5 minutes, or that a baby monitor is authorized for 15 minutes.
  • the amount of time may also vary depending on a task or other context indicated by the request 135.
  • the data indicating the authorization of the device 120 to receive audio and the data indicating the limited period of time may be encrypted or otherwise securely encoded by the server 140 so that the information cannot be falsified.
  • the server 140 sends a request to a user device for user confirmation before providing the device 120 access to audio. For example, before generating and sending a token 144 in response to the request 135, the server 140 may determine that the device 120 is registered to a user account for a particular user. From user account data, the server 140 may identify a user device associated with the user account, such as the user's phone, and send a notification that the device 120 has requested access to audio data.
  • the notification may include a user interface, such as buttons or other areas displayed a touchscreen, that the user may interact with to allow or deny access. For example, the notification can ask, "Do you want to use your intercom now?" and give the user an option to select yes or no. In this manner, users can be notified when devices are attempting to access audio, and attempted eavesdropping by a compromised device can be detected and prevented.
  • step (F) the server 140 transmits the token 144 to the device 120 via the network 130.
  • the device 120 receives the token 144.
  • the token data is encrypted or otherwise securely encoded, the device 120 generally cannot read the data.
  • the processing module 122 controls access to audio data generated using the microphone 115, the device 120 does not receive audio until the processing module 122 switches to a mode that provides unencrypted audio data.
  • the device 120 provides the token to the processing module 122 and the processing module evaluates the token 144.
  • the processing module 122 can be configured to verify a validity of the token 144. As such, the processing module 122 can decrypt or otherwise decode the token 144 to determine whether or not permit the device 120 access to the received audio data. Similar to generation of the identifying data 129, the token 144 can be generated with a time or other information that allows the processing module 122 to verify that the token is current and not outdated. For example, the processing module 122 can compare an extracted time value with a time that the token 144 was received by the processing module 122. If the two times are within a threshold amount of time of each other, the request can be determined to be current.
  • the token 144 can be ignored as potentially outdated or a potential repeat of a previous token.
  • the token may include data that can be matched with data in the identifying data.
  • the token 144 may include a secret or random value encoded in the identifying data 129, demonstrating that the token 144 was generated using the secret value for the same device 120 and the same session.
  • the token 144 indicates whether the device 120 is authorized to receive audio data from the microphone 115. In some implementations, a token 144 is provided even if authorization is denied. This can provide the device 120 and/or processing module 122 at least the confirmation that the request 135 was received and answered.
  • the processing module 122 can determine, based on the token 144, a limited period of time 145 that the device 120 may access the received audio data. As noted above, this may be determined in any appropriate form, e.g., a duration of time, an expiration for the authorization, a range of time that authorization lasts, and so on. For example, if the processing module 122 determines that the device 120 may be granted permission to access the audio data, the processing module 122 can switch to an unencrypted audio mode and pass the raw or unencrypted audio 146 to the processor 128 for the limited period of time 145.
  • the unencrypted audio channel may be referred to as the unencrypted audio mode and the encrypted audio channel may be referred to as the encrypted audio mode, as discussed further herein.
  • step (H) after the token 144 is verified and the processing module 122 has determined that the device 120 is authorized to access to the audio data, the processing module 122 provides unencrypted audio to the processor 128 of the device 120.
  • the processing module 122 may begin to provide audio data in substantially real time once the authorization is verified. This access may allow the device 120 to use the audio data in substantially the same manner as if the device 120 had a direct connection to the microphone 115 or typical analog to digital audio processing hardware that is unsecured. That is, in some implementations, the processor 128 may have substantially unrestricted access to audio data detected during the limited period of time 145.
  • a device such as an intercom can provide audio for an ongoing call unimpeded and without delay.
  • the processing module 122 may continue to provide unencrypted audio as audio is received.
  • the processing module 122 automatically transitions back to restricting access to audio detected by the microphone 115.
  • the cryptography module 125 can be used to encrypt audio detected after the limited period of time 145 ends, so that the device 120 is once again restricted from accessing the audio data.
  • the device 120 may initiate a new request for authorization, for example, by detecting that audio is needed and initiating the generation of a request in the same manner discussed for the request 135.
  • the processing module 122 can be configured to detect hotwords while unencrypted audio is being provided to the device 120, e.g., during the limited period of time 145.
  • the processing module 122 may use the hotword module 123 to perform this function.
  • the hotword module 123 can implement a hotword detection model, which may include a trained machine learning model, such as a neural network, a maximum entropy classifier, and so on.
  • the hotword module 123 can use the same hotword detection model used to detect hotwords that occur while audio access is restricted, or may use a different hotword detection model.
  • one or more parameters of the hotword module 123 may be different when operating during the period that audio is restricted and during the period when audio is unrestricted.
  • the detection of a hotword or voice command may interrupt the open access of device 120 to received audio. In this manner, voice commands and other user requests that are intended for a voice assistant and not for the device 120 can be securely routed to the server 140.
  • the audio is provided to the device 120 and also monitored using the hotword module 123. If the hotword module 123 detects a hotword, e.g., with at least a minimum level of confidence, the device's access to the audio may be restricted, even if the limited period of time 145 of authorization has not ended.
  • the detection of a hotword may prompt the processing module 122 to transition from providing unencrypted audio for use by the device 120 to providing encrypted audio to be sent to the server 140.
  • the hotword module 123 can pass a signal to the processing module 122 to restrict access to the audio data by the device 120. This allows the processing module 122 to receive user voice requests in an encrypted audio mode until the end of the voice request is detected.
  • commands and other requests may be received at the device 120 without sacrificing the confidentiality of the command during the limited period of time 145 that the device 120 is authorized to access audio data.
  • the processing module 122 may include a voice activity detector or other functionality to determine the end of a user's voice request. For example, after a hotword and an immediately subsequent voice command, the voice activity detector may detect a pause in speaking of at least a predetermined minimum duration, e.g., 2 seconds, 5 seconds, etc. This can indicate to the processing module 122 that the user request corresponding to the hotword has ended. When the user request has ended, the processing module 122 may stop providing encrypted audio to be sent to the server 140. The processing module 122 also determines whether the limited period of time 145 has ended.
  • a voice activity detector may detect a pause in speaking of at least a predetermined minimum duration, e.g., 2 seconds, 5 seconds, etc. This can indicate to the processing module 122 that the user request corresponding to the hotword has ended.
  • the processing module 122 may stop providing encrypted audio to be sent to the server 140.
  • the processing module 122 also determines whether the limited period of time 145 has ended.
  • the processing module 122 continues blocking access to the audio, e.g., by providing no audio data to the device 120 or providing only encrypted audio data that is not accessible by the device 120. If the limited period of time has not ended, however, the processing module 122 returns to the unencrypted mode, and again provides unencrypted audio.
  • the processing module 122 can also be configured to handle certain commands that are not immediately preceded by a hotword. Sometimes, after a user has spoken a hotword and a first voice request, a user may pause and later make a second voice request without speaking a second hotword. For example, a user may speak a hotword, e.g., "Okay phone,” and ask “Order a pepperoni pizza.” After the command has ended and potentially after the action is completed, the user may follow-up with a second voice request, "When will the pizza arrive?", in an attempt to continue the conversation with the virtual assistant without speaking a hotword before the second request. In this instance, only one hotword may be detected for both a command and a follow up question to be received by the processing module 122. To detect these types of follow-up requests or other requests that are not preceded by a hotword, the processing module 122 can use the question module 124.
  • the processing module 122 can use the question module 124.
  • the question module 124 can operate by assessing speech characteristics during a certain duration after the end of a known user voice request. Similar to the hotword module 123, the question module 124 can include an audio processing model, for example, a trained machine learning model such as a neural network, a classifier, and so on. For example, the question module 124 may monitor audio for a potential follow-up request during a period of time, e.g., 30 seconds, 1 minute, etc., after the end of the prior request. During this time the question module 124 can monitor sound characteristics, such as a voice inflection, to generate a likelihood score indicating an estimated likelihood that a follow-up request has been made.
  • an audio processing model for example, a trained machine learning model such as a neural network, a classifier, and so on.
  • the question module 124 may monitor audio for a potential follow-up request during a period of time, e.g., 30 seconds, 1 minute, etc., after the end of the prior request.
  • the question module 124
  • the question module 124 may detect non-hotword terms that signal a possible question, e.g., who, what, where, why, when, how, etc., and use that information alone or in combination with other data to generate a likelihood score.
  • the processing module 122 can treat the audio as a potential user voice request, e.g., by transitioning to an encrypted mode and providing encrypted audio to the server 140 as a potential voice command.
  • FIG. 2 is a diagram of an example architecture 200 for context-based switches between encrypted and unencrypted audio.
  • the architecture 200 for context-based switches between encrypted and unencrypted audio includes a microphone 210, a device 220, a processing module 230, a server 240, and a speaker 250.
  • the architecture 200 illustrates a block diagram in which audio data is processed to transition between encrypted and unencrypted audio channels.
  • the audio data may transition between the encrypted and unencrypted audio channels in response to a trigger as well as in response to specific content of the received audio data.
  • the microphone 210 includes one or more microphones that persistently transmit raw audio 215.
  • the microphone 210 is connected to the processing module 230.
  • the microphone 210 passes received audio to the processing module 230.
  • the processing module 230 can be configured to encrypt the audio data via an cryptography module 233. In some implementations, all received audio 215 is initially encrypted by the cryptography module 233.
  • a trigger 222 of the device 220 may be initiated.
  • the trigger 222 can indicate a request for access to the audio data by the device 220.
  • the trigger 222 can be a button of an intercom system in which a request to transmit unencrypted audio over the intercom system is initiated.
  • the trigger 222 can initiate the generation of a request 234 by a processor 224 of the device 220.
  • the processor 224 may generate the request 234 and pass the request 234 to the processing module 230.
  • the processing module 230 can be configured to add an encrypted value to the request 234.
  • the encrypted value can include an identifier for the processing module 230 as well as a time. In some aspects, the time corresponds to the time at which the request 234 is generated. In other aspects, the time corresponds to the time at which the trigger 222 is actuated.
  • the identifier for the processing module 230 can specify an owner of the device 220 and/or a type of the device 220.
  • the encrypted value is a hash.
  • the request 234 may be passed back to the processor 224 after the encrypted value is added to the request 234.
  • the request 234 can be transmitted from the device 220 to the server 240 for validation of the request 234.
  • the server 240 can be configured to receive the request 234 and verify the integrity of the request 234. In other words, the server 240 can be configured to determine whether or not to allow the device 220 to access the audio data.
  • the server 240 can be configured to verify the request 234 based on the encrypted value of the request 234. Specifically, the server 240 can be configured to verify an integrity of the request 234 by looking up information corresponding to the encrypted value.
  • the server 240 can be configured to determine whether or not the device 220 is to be trusted based on the identifier for the processing module 230 and the time associated with the request 234. For example, the server 240 can access a lookup table including a list of verified identifiers.
  • the server 240 can be configured to compare the identifier of the encrypted value with the lookup table to verify the integrity of the request. Specifically, if the identifier of the encrypted value exists in the lookup table, and the identifier is associated with an appropriate type of device, the device 220 may be permitted access to the audio data for a limited period of time. Otherwise, if the identified of the encrypted value does not exist in the lookup table, or if an indicated device type does not qualify for receiving audio data, the server 240 will not grant access for the device 220 to access the audio data.
  • the server 240 determines that the request 234 is valid and that access authorization is appropriate, the server 240 generates a token 241 encoding a limited amount of time 242 that the device 220 may access the audio. Otherwise, if the server 240 does not determine that the request is valid, the server 240 generates a token 241 encoding a command to disregard to the request for access to the encrypted audio and to continue restricting access to the audio data by the device 220.
  • the server 240 can be configured to transmit the token 241 to the device 220.
  • the device 220 can be configured to pass the token 241 to the processing module 230 via the processor 224.
  • the processing module 230 can be configured to decode the token 241. Specifically, the processing module 230 can be configured to evaluate the token 241 to verify a validity of the token 241, to verify permission of the device 220 to access the audio data, and to extract a duration of time 242 that the device 220 may access the audio data (if the device 220 is being granted access to the audio data).
  • the processing module 230 can be configured to switch to an unencrypted audio mode using the cryptography module 233.
  • the cryptography module 233 can be used by the processing module to permit access to the audio data by the device for the limited period of time 242 (as indicated by the token 241).
  • the processing module 230 can be configured to pass the audio 215 from the microphone 210 to the device 220 as unencrypted audio data for the limited period of time.
  • the device 220 can be configured to receive the unencrypted audio data using the processor 224 via the unencrypted audio channel from the processing module 230.
  • the unencrypted audio data can be transmitted to the speaker 250 by the processor 224.
  • the unencrypted audio data may be converted to sound for emission by the speaker 250.
  • the audio received at the intercom may be broadcast by one or more speakers as a result of the verification of a request to broadcast the received audio data.
  • the processing module 230 can further include a hotword module 231 and a question module 232.
  • the processing module 230 can be configured to detect for hotwords and questions within the received audio data during the limited period of time 242.
  • the hotword module 231 can be used by the processing module 230 to detect predetermined hotwords during the limited period of time 242. If a hotword is detected by the hotword module 231, the processing module 230 can be configured to transition from the unencrypted audio mode to the encrypted audio mode during the limited period of time 242.
  • the detected hotword and the audio that follows the detected hotword is encrypted and transmitted from the device 220 to the server 240. As such, the hotword and the audio that follows the detected hotword may not be passed to the speaker 250.
  • the audio that follows the hotword can include a command.
  • the command can be encrypted by the cryptography module 233 so that access to the command is restricted by the device 220.
  • the command may be securely transmitted to the server 240 due to the transition from the unencrypted mode to the encrypted mode in response to the detection of the hotword during the limited period of time 242.
  • the audio that follows the hotword can additionally include a follow up question during the limited period of time 242.
  • the follow up question may be detected by the question module 232.
  • the question module 232 can be used by the processing module 230 to indicate that access to the audio data should remain restricted by the device 220.
  • the processing module 230 detects a hotword via the hotword module 231
  • the audio data may be encrypted for a portion of time during the limited period of time 242 in which a command and/or a follow up question may be detected, for secure transmission of the command and/or the follow up question to the server 240.
  • FIG. 3 is an event timeline 300 showing examples of context-based switches between providing encrypted audio and unencrypted audio.
  • the timeline 300 shows a channel for unencrypted audio 302 and a channel for encrypted audio 304. Based on the events that occur along the timeline 300, the unencrypted audio channel 302 or the encrypted audio channel 304 may be used to transmit audio data from a processing module to a device.
  • a trigger condition or even is detected at the device.
  • the processing module is restricting access to audio data by requiring any access to occur on the encrypted audio channel 304.
  • the trigger can prompt the device to generate a request for authorization to access unencrypted audio data, which would require a transition from using the encrypted audio channel 304 to the unencrypted audio channel 302. Examples of devices that may require unencrypted audio data include an intercom system, a karaoke machine, a baby monitor, and the like.
  • the trigger can be used to initialize the generation of a request for the transmission of unencrypted audio over the unencrypted audio channel 302.
  • the device generates a request for access to audio data.
  • the request for the token seeks to enable the device to access audio information.
  • the request for a token can include an encrypted value that is generated by the processing module and passed to the device.
  • the encrypted value can include an identifier for the processing module as well as a time at which the request is being generated.
  • the request for the token can be generated by the device in combination with the processing module, and passed to a server for verification of the request.
  • the encrypted audio channel 304 is continues to be used as the device has not yet been permitted access to audio information.
  • the device receives a token from the server.
  • the server can be configured to verify the request for the token based on the encrypted value of the request.
  • the server can be configured to transmit a generated token to the device based on the encrypted value of the request.
  • the token can include information indicating whether or not the device is authorized to access the audio information detected by the microphone. Additionally, the token can include information specifying a limited period of time in which the device may access the audio data. The device continues to use the encrypted audio channel 304 until the token itself is verified by the processing module.
  • the device passes the token to the processing module for verification.
  • the processing module can be configured to verify the token.
  • the processing module can be configured to extract the information from the token indicating whether or not the device is authorized to access the audio information. Additionally, the processing module can be configured to extract the information from the token indicating the duration of the limited period of time. As shown in FIG. 3 , if the processing module verifies the validity of the token, a transition from the encrypted audio channel 304 to the unencrypted audio channel 302 occurs. Thus, the device is permitted to access audio information once the token has been verified by the processing module.
  • the processing module detects a hotword during the limited period of time. In response to detecting the hotword, the processing module transitions from the unencrypted audio 302 channel to the encrypted audio channel 304 during the limited period of time. As such, the audio following the hotword may be encrypted so that the audio following the hotword may not be accessed by the device.
  • the processing module detects a command that follows the hotword.
  • the command is detected by the processing module during the limited period of time over the encrypted audio channel 304.
  • a follow up question may follow the command during the limited period of time, over the encrypted audio channel 304.
  • the processing module can be configured to detect whether or not a follow up question is present.
  • the processing module passes the command and the follow up question via the encrypted audio channel 304 to the device.
  • the device may be configured to securely transmit the command and the follow up question to the server.
  • the server can be configured to decrypt the command and the follow up question.
  • the command and the follow up question are restricted from being accessed by the device during the limited period of time since the audio data has transitioned from the unencrypted audio channel 302 to the encrypted audio channel 304 in response to the detection of hotword.
  • the processing module detects the end of the command. As the end of the command is detected, the processing module transitions from the encrypted audio channel 304 to the unencrypted audio channel 302 during the limited period of time. In some aspects, the processing module detects a follow up question before the end of the command. In this instance, the processing module may be configured to detect the end of the follow up question and transition from the encrypted audio channel 304 to the unencrypted audio channel 302 upon the detection of the end of the follow up question. In an example, the processing module can be configured to detect for a follow up question for a portion of time after the detection of the end of a command and during the limited period of time. Thus, after the end of the portion of time in which detection for a follow up question occurs, the processing module transitions back to the unencrypted audio channel 302.
  • the processing module detects the expiration of the limited period of time as indicated by the token. Upon the detection of the end of the limited period of time, the processing module transitions from the unencrypted audio channel 302 to the encrypted audio channel 302. As such, the processing module restricts access to the audio data by the device.
  • the limited period of time occurs prior to the end of the command.
  • the processing module transitions from the unencrypted audio channel 302 to the encrypted audio channel 302 before the end of the command.
  • the limited period of time defines a duration in which the unencrypted audio channel 302 may be used for a single verified token.
  • FIG. 4 is a flow chart illustrating an example process 400 for context-based switches between encrypted and unencrypted audio.
  • the process 400 will be described as being performed by a system of one or more computers located in one or more locations.
  • a system for context-based switches between encrypted and unencrypted audio e.g., the system 100 for context-based switches between encrypted and unencrypted audio of FIG. 1 , appropriately programmed in accordance with this specification can perform the process 400.
  • a processing module of a device restricts access by the device to audio information.
  • the audio information may be detected by a microphone in communication with the processing module.
  • the audio information may be passed to the processing module of the device where the audio information is encrypted. Access to the audio information by the device may be restricted due to the encryption of the audio information. In this instance, the encrypted audio information may still be passed to the device, however, the device may not access the audio information due to the encryption. In other aspects, access to the audio information by the device may be restricted by blocking access of the audio information from the processing module to the device entirely.
  • the processing module receives data indicating the device is authorized to access audio information during a limited period of time.
  • the processing module can receive data, such as a token, from a remote server.
  • the server can transmit a secure token to the device.
  • the device can be configured to pass the secure token to the processing module for decoding.
  • the processing module can decode the token to determine that the device is authorized to access audio information received by the microphone for a limited period of time as indicated by the token.
  • the processing module provides audio information to the device.
  • the processing module can be configured to provide audio information to the device in response to decoding the token.
  • the processing module can extract information from the token that indicates the device is authorized to access audio information for a limited period of time.
  • the processing module decodes the token to determine that the device is not authorized to access the audio information. In this instance the processing module continues to restrict access by the device to audio information.
  • the processing module monitors audio information for the presence of a hotword during the limited period of time.
  • the processing module may restrict, during the limited period of time, access by the device to audio information that is detected by the microphone and that follows the hotword.
  • the processing module may detect a presence of a command that follows the hotword during the limited period of time.
  • the processing module can be configured to determine when the command has ended during the limited period of time. After determining that the command has ended during the limited period of time, the processing module can provide audio information to the device until the end of the limited period of time.
  • the processing module can be configured to detect, during the limited period of time, a presence of a question that occurs before another hotword is detected. In response to detecting the presence of the question, the processing module can be configured to provide audio information to the device until the end of the limited period of time.
  • the processing module restricts access to audio information after the end of the limited period of time.
  • the processing module restricts access to audio information detected by the microphone in response to determining that the limited period of time has ended.
  • Access to the audio information by the device may be restricted via a transition from an unencrypted audio channel to an encrypted audio channel. In this instance, the encrypted audio information may still be passed to the device, however, the device may not access the audio information due to the encryption.
  • access to the audio information by the device may be restricted by blocking access of the audio information from the processing module to the device entirely.
  • Embodiments of the invention and all of the functional operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.
  • Embodiments of the invention can be implemented as one or more computer program products, e.g., one or more modules of computer program instructions encoded on a computer readable medium for execution by, or to control the operation of, data processing apparatus.
  • the computer readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them.
  • data processing apparatus encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers.
  • the apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
  • a propagated signal is an artificially generated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus.
  • a computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
  • a computer program does not necessarily correspond to a file in a file system.
  • a program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code).
  • a computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.
  • the processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output.
  • the processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).
  • processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer.
  • a processor will receive instructions and data from a read only memory or a random access memory or both.
  • the essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data.
  • a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks.
  • mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks.
  • a computer need not have such devices.
  • a computer can be embedded in another device, e.g., a tablet computer, a mobile telephone, a personal digital assistant (PDA), a mobile audio player, a Global Positioning System (GPS) receiver, to name just a few.
  • Computer readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.
  • the processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
  • embodiments of the invention can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer.
  • a display device e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor
  • keyboard and a pointing device e.g., a mouse or a trackball
  • Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.
  • Embodiments of the invention can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the invention, or any combination of one or more such back end, middleware, or front end components.
  • the components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet.
  • LAN local area network
  • WAN wide area network
  • the computing system can include clients and servers.
  • a client and server are generally remote from each other and typically interact through a communication network.
  • the relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
  • HTML file In each instance where an HTML file is mentioned, other file types or formats may be substituted. For instance, an HTML file may be replaced by an XML, JSON, plain text, or other types of files. Moreover, where a table or hash table is mentioned, other data structures (such as spreadsheets, relational databases, or structured files) may be used.

Landscapes

  • Engineering & Computer Science (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Security & Cryptography (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Acoustics & Sound (AREA)
  • Computational Linguistics (AREA)
  • Computer Hardware Design (AREA)
  • Signal Processing (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • General Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Telephonic Communication Services (AREA)

Claims (15)

  1. Verfahren, umfassend:
    Empfangen, an einem Verarbeitungsmodul (122) eines Geräts (120), von Audioinformationen, die durch ein Mikrofon (115) des Geräts erkannt wurden;
    Einschränken, durch das Verarbeitungsmodul, des Zugriffs eines Geräteprozessors (128) des Geräts auf die Audioinformationen, die durch das Verarbeitungsmodul empfangen wurden, wobei der Geräteprozessor von dem Verarbeitungsmodul getrennt ist;
    Empfangen, durch das Verarbeitungsmodul, von Daten, die anzeigen, dass der Geräteprozessor während eines begrenzten Zeitraums zum Zugriff auf die Audioinformationen, die durch das Mikrofon erkannt wurden, autorisiert ist;
    in Reaktion auf das Empfangen von Daten, die anzeigen, dass der Geräteprozessor während des begrenzten Zeitraums zum Zugriff auf die Audioinformationen, die durch das Mikrofon erkannt wurden, autorisiert ist, Bereitstellen der Audioinformationen an den Geräteprozessor;
    Überwachen, durch das Verarbeitungsmodul, der Audioinformationen, die während des begrenzten Zeitraums durch das Mikrofon erkannt wurden, auf das Vorhandensein eines Aktivierungsworts; und
    nach Ablauf des begrenzten Zeitraums, Einschränken, durch das Verarbeitungsmodul, des Zugriffs des Geräteprozessors auf die Audioinformationen, die durch das Mikrofon erkannt wurden.
  2. Verfahren nach Anspruch 1, ferner umfassend das Einschränken des Zugriffs auf Audioinformationen, die durch das Mikrofon erkannt wurden, in Reaktion auf das Ermitteln, dass der begrenzte Zeitraum verstrichen ist.
  3. Verfahren nach Anspruch 1 oder 2, wobei das Empfangen der Daten, die anzeigen, dass der Geräteprozessor während des begrenzten Zeitraums zum Zugriff auf Audioinformationen, die durch das Mikrofon erkannt wurden, autorisiert ist, umfasst:
    Empfangen eines Tokens, das (i) anzeigt, dass der Geräteprozessor während des begrenzten Zeitraums zum Zugriff auf die Audioinformationen, die durch das Mikrofon erkannt wurden, autorisiert ist, und (ii) eine Dauer des begrenzten Zeitraums anzeigt;
    wobei das Verfahren ferner umfasst:
    Auswerten des Tokens, um die Gültigkeit des Tokens zu verifizieren; und
    in Reaktion auf das Verifizieren der Gültigkeit des Tokens, Extrahieren von Daten des Tokens, die (i) anzeigen, dass der Geräteprozessor während des begrenzten Zeitraums zum Zugriff auf Audioinformationen, die durch das Mikrofon erkannt wurden, autorisiert ist, und (ii) eine Dauer des begrenzten Zeitraums anzeigen.
  4. Verfahren nach einem der Ansprüche 1 bis 3, wobei das Einschränken des Zugriffs des Geräteprozessors auf die Audioinformationen, die durch das Verarbeitungsmodul empfangen wurden, zumindest eines von Folgendem umfasst:
    Verschlüsseln der Audioinformationen, die durch das Mikrofon erkannt wurden; oder
    Sperren, durch das Verarbeitungsmodul, des Zugriffs des Geräteprozessors auf die Audioinformationen.
  5. Verfahren nach einem der Ansprüche 1 bis 4, ferner umfassend:
    Erkennen des Vorhandenseins des Aktivierungsworts während des begrenzten Zeitraums; und, in Reaktion auf das Erkennen des Vorhandenseins des Aktivierungsworts während des begrenzten Zeitraums, Einschränken, während des begrenzten Zeitraums, des Zugriffs des Geräteprozessors auf Audioinformationen, die durch das Mikrofon erkannt wurden und auf das Aktivierungswort folgen.
  6. Verfahren nach Anspruch 5, ferner umfassend:
    Erkennen, während des begrenzten Zeitraums, einer Sprachanforderung, die auf das Aktivierungswort folgt, wobei die Sprachanforderung ein gesprochener Befehl oder eine Anforderung ist;
    Ermitteln, dass die Sprachanforderung vor Ende des begrenzten Zeitraums geendet hat; und
    in Reaktion auf das Ermitteln, dass die Sprachanforderung vor Ende des begrenzten Zeitraums geendet hat, Bereitstellen, an den Geräteprozessor, von unverschlüsselten Audioinformationen, die nach Ende der Sprachanforderung erkannt wurden.
  7. Verfahren nach Anspruch 6, ferner umfassend:
    nach dem Ermitteln, dass die Sprachanforderung vor Ende des begrenzten Zeitraums geendet hat, und nach Bereitstellen von unverschlüsselten Audioinformationen, die nach Ende der Sprachanforderung erkannt wurden, Erkennen einer zweiten Sprachanforderung, die während des begrenzten Zeitraums erfolgt und der kein zusätzliches Aktivierungswort vorangeht; und
    in Reaktion auf das Erkennen der zweiten Sprachanforderung, die während des begrenzten Zeitraums erfolgt und der kein zusätzliches Aktivierungswort vorangeht, Bereitstellen von der zweiten Sprachanforderung entsprechenden verschlüsselten Audioinformationen an den Geräteprozessor.
  8. Verfahren nach einem der Ansprüche 1 bis 7, das ferner ein Empfangen einer Anforderung zum Zugriff auf Audioinformationen, die durch das Mikrofon erkannt wurden, umfasst, wobei die Anforderung durch das Gerät in Reaktion auf das Erkennen einer Benutzerinteraktion mit dem Gerät bereitgestellt wird.
  9. Verfahren nach Anspruch 8, wobei das Empfangen der Anforderung zum Zugriff auf Audioinformationen, die durch das Mikrofon erkannt wurden, ein Assoziieren der Anforderung mit Daten umfasst, die abgeleitet werden (i) aus einer Kennung für das Verarbeitungsmodul und (ii) aus Daten, die einen Zeitpunkt anzeigen.
  10. Gerät (120), umfassend:
    ein Mikrofon (115);
    einen Geräteprozessor (128); und
    ein Verarbeitungsmodul (122), das von dem Geräteprozessor getrennt ist und zum Durchführen von Operationen konfiguriert ist, welche Folgendes umfassen:
    Empfangen von Audioinformationen, die durch das Mikrofon erkannt wurden;
    Einschränken des Zugriffs des Geräteprozessors auf die Audioinformationen,
    die durch das Verarbeitungsmodul empfangen wurden;
    Empfangen von Daten, die anzeigen, dass der Geräteprozessor während eines begrenzten Zeitraums zum Zugriff auf Audioinformationen, die durch das Mikrofon erkannt wurden, autorisiert ist;
    in Reaktion auf das Empfangen von Daten, die anzeigen, dass der Geräteprozessor während des begrenzten Zeitraums zum Zugriff auf Audioinformationen, die durch das Mikrofon erkannt wurden, autorisiert ist, Bereitstellen der Audioinformationen, die durch das Mikrofon erkannt wurden, an den Geräteprozessor;
    Überwachen von Audioinformationen, die während des begrenzten Zeitraums durch das Mikrofon erkannt wurden, auf das Vorhandensein eines Aktivierungsworts; und
    nach Ablauf des begrenzten Zeitraums, Einschränken des Zugriffs des Geräteprozessors auf Audioinformationen, die durch das Mikrofon erkannt wurden.
  11. Gerät nach Anspruch 10,
    wobei die Operationen ferner das Einschränken des Zugriffs auf Audioinformationen, die durch das Mikrofon erkannt wurden, in Reaktion auf das Ermitteln, dass der begrenzte Zeitraum abgelaufen ist, umfasst;
    und/oder
    wobei das Empfangen der Daten, die anzeigen, dass der Geräteprozessor während des begrenzten Zeitraums zum Zugriff auf Audioinformationen, die durch das Mikrofon erkannt wurden, autorisiert ist, umfasst:
    Empfangen eines Tokens, das (i) anzeigt, dass der Geräteprozessor während des begrenzten Zeitraums zum Zugriff auf die Audioinformationen, die durch das Mikrofon erkannt wurden, autorisiert ist, und (ii) eine Dauer des begrenzten Zeitraums anzeigt;
    wobei die Operationen ferner umfassen:
    Auswerten des Tokens, um die Gültigkeit des Tokens zu verifizieren; und
    in Reaktion auf das Verifizieren der Gültigkeit des Tokens, Extrahieren von Daten des Tokens, die (i) anzeigen, dass der Geräteprozessor während des begrenzten Zeitraums zum Zugriff auf die Audioinformationen, die durch das Mikrofon erkannt wurden, autorisiert ist, und (ii) eine Dauer des begrenzten Zeitraums anzeigen;
    und/oder
    wobei das Einschränken des Zugriffs des Geräteprozessors auf die Audioinformationen, die durch das Verarbeitungsmodul empfangen wurden, zumindest eines von Folgendem umfasst:
    Verschlüsseln der Audioinformationen, die durch das Mikrofon erkannt wurden; oder
    Sperren, durch das Verarbeitungsmodul, des Zugriffs des Geräteprozessors auf die Audioinformationen.
  12. Gerät nach Anspruch 10 oder 11, wobei die Operationen ferner umfassen:
    Erkennen des Vorhandenseins des Aktivierungsworts während des begrenzten Zeitraums; und
    in Reaktion auf das Erkennen des Vorhandenseins des Aktivierungsworts während des begrenzten Zeitraums, Einschränken, während des begrenzten Zeitraums, des Zugriffs des Geräteprozessors auf Audioinformationen, die durch das Mikrofon erkannt wurden und auf das Aktivierungswort folgen.
  13. Gerät nach Anspruch 12, wobei die Operationen ferner umfassen:
    Erkennen, während des begrenzten Zeitraums, einer Sprachanforderung, die auf das Aktivierungswort folgt, wobei die Sprachanforderung ein gesprochener Befehl oder eine Anforderung ist;
    Ermitteln, dass die Sprachanforderung vor Ende des begrenzten Zeitraums geendet hat; und
    in Reaktion auf das Ermitteln, dass die Sprachanforderung vor Ende des begrenzten Zeitraums geendet hat, Bereitstellen, an den Geräteprozessor, von unverschlüsselten Audioinformationen, die nach Ende der Sprachanforderung erkannt wurden.
  14. Gerät nach einem der Ansprüche 10 bis 13, wobei die Operationen ferner das Empfangen einer Anforderung zum Zugriff auf Audioinformationen, die durch das Mikrofon erkannt wurden, umfasst, wobei die Anforderung durch das Gerät in Reaktion auf ein Erkennen einer Benutzerinteraktion mit dem Gerät bereitgestellt wird.
  15. Ein oder mehrere nichtflüchtige computerlesbare Medien, die Software speichern, welche Anweisungen umfasst, die durch einen oder mehrere Prozessoren ausführbar sind und, bei Ausführung, den einen oder die mehreren Prozessoren veranlassen, das Verfahren nach einem der Ansprüche 1 bis 9 durchzuführen.
EP17757936.4A 2016-12-07 2017-08-01 Sicherung von audiodaten Active EP3533196B1 (de)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US15/372,182 US9940930B1 (en) 2016-12-07 2016-12-07 Securing audio data
PCT/US2017/044842 WO2018106293A1 (en) 2016-12-07 2017-08-01 Securing audio data

Publications (2)

Publication Number Publication Date
EP3533196A1 EP3533196A1 (de) 2019-09-04
EP3533196B1 true EP3533196B1 (de) 2020-05-27

Family

ID=59702813

Family Applications (1)

Application Number Title Priority Date Filing Date
EP17757936.4A Active EP3533196B1 (de) 2016-12-07 2017-08-01 Sicherung von audiodaten

Country Status (4)

Country Link
US (2) US9940930B1 (de)
EP (1) EP3533196B1 (de)
CN (1) CN110036616B (de)
WO (1) WO2018106293A1 (de)

Families Citing this family (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10103872B2 (en) * 2014-09-26 2018-10-16 Intel Corporation Securing audio communications
US10237268B2 (en) * 2016-11-02 2019-03-19 Google Llc Secure passcode processing device
CN112822205B (zh) * 2016-12-21 2024-02-02 创新先进技术有限公司 跨设备的登录方法、系统及装置
KR20180084392A (ko) * 2017-01-17 2018-07-25 삼성전자주식회사 전자 장치 및 그의 동작 방법
US11183181B2 (en) * 2017-03-27 2021-11-23 Sonos, Inc. Systems and methods of multiple voice services
US10887423B2 (en) * 2017-05-09 2021-01-05 Microsoft Technology Licensing, Llc Personalization of virtual assistant skills based on user profile information
US10789949B2 (en) * 2017-06-20 2020-09-29 Bose Corporation Audio device with wakeup word detection
WO2019083500A1 (en) * 2017-10-24 2019-05-02 Visa International Service Association SYSTEM, METHOD AND APPARATUS FOR AUTOMATICALLY ENCODING DATA IN ELECTRONIC COMMUNICATION
US10916252B2 (en) 2017-11-10 2021-02-09 Nvidia Corporation Accelerated data transfer for latency reduction and real-time processing
KR102612835B1 (ko) * 2018-04-20 2023-12-13 삼성전자주식회사 전자 장치 및 전자 장치의 기능 실행 방법
US10887351B2 (en) * 2018-05-02 2021-01-05 NortonLifeLock Inc. Security for IoT home voice assistants
CN110211608B (zh) * 2019-06-11 2021-05-28 兰州理工大学 一种语音检索方法及系统
US11380312B1 (en) * 2019-06-20 2022-07-05 Amazon Technologies, Inc. Residual echo suppression for keyword detection
US11310209B2 (en) * 2019-09-10 2022-04-19 Motorola Solutions, Inc. Method and device for protecting confidentiality of audio messages
US11200328B2 (en) * 2019-10-17 2021-12-14 The Toronto-Dominion Bank Homomorphic encryption of communications involving voice-enabled devices in a distributed computing environment
US11437043B1 (en) * 2019-12-12 2022-09-06 Amazon Technologies, Inc. Presence data determination and utilization
US20220129543A1 (en) * 2020-10-27 2022-04-28 Arris Enterprises Llc Secure voice interface in a streaming media device to avoid vulnerability attacks
US11620990B2 (en) 2020-12-11 2023-04-04 Google Llc Adapting automated speech recognition parameters based on hotword properties
US11662973B2 (en) * 2021-03-16 2023-05-30 Dell Products, L.P. Systems and methods for orchestrated audio session management for modern workspaces
US11783831B1 (en) * 2021-06-29 2023-10-10 Amazon Technologies, Inc. Data protection in a multi-assistant system

Family Cites Families (48)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5267312A (en) 1990-08-06 1993-11-30 Nec Home Electronics, Ltd. Audio signal cryptographic system
AU6049298A (en) 1997-01-31 1998-08-25 T-Netix, Inc. System and method for detecting a recorded voice
US6496581B1 (en) 1997-09-11 2002-12-17 Digisonix, Inc. Coupled acoustic echo cancellation system
US7228429B2 (en) 2001-09-21 2007-06-05 E-Watch Multimedia network appliances for security and surveillance applications
US6889321B1 (en) 1999-12-30 2005-05-03 At&T Corp. Protected IP telephony calls using encryption
US6507653B1 (en) 2000-04-14 2003-01-14 Ericsson Inc. Desired voice detection in echo suppression
US6741874B1 (en) 2000-04-18 2004-05-25 Motorola, Inc. Method and apparatus for reducing echo feedback in a communication system
US6928161B1 (en) 2000-05-31 2005-08-09 Intel Corporation Echo cancellation apparatus, systems, and methods
US7447321B2 (en) 2001-05-07 2008-11-04 Harman International Industries, Incorporated Sound processing system for configuration of audio signals in a vehicle
US7158776B1 (en) 2001-09-18 2007-01-02 Cisco Technology, Inc. Techniques for voice-based user authentication for mobile access to network services
US7072354B1 (en) * 2001-10-03 2006-07-04 Cisco Technology, Inc. Token registration of managed devices
MXPA04006444A (es) 2002-01-02 2005-03-31 Sony Electronics Inc Encriptacion parcial y mapeo de pid.
WO2003079721A1 (en) 2002-03-13 2003-09-25 Harman International Industries, Incorporated Audio feedback processing system
US6904146B2 (en) 2002-05-03 2005-06-07 Acoustic Technology, Inc. Full duplex echo cancelling circuit
US7418392B1 (en) 2003-09-25 2008-08-26 Sensory, Inc. System and method for controlling the operation of a device by voice commands
US20080312934A1 (en) 2007-03-07 2008-12-18 Cerra Joseph P Using results of unstructured language model based speech recognition to perform an action on a mobile communications facility
WO2009015460A1 (en) 2007-07-27 2009-02-05 Clear-Com Research Inc. Multi-point to multi-point intercom system
US20090049307A1 (en) * 2007-08-13 2009-02-19 Authennex, Inc. System and Method for Providing a Multifunction Computer Security USB Token Device
US8099289B2 (en) 2008-02-13 2012-01-17 Sensory, Inc. Voice interface and search for electronic devices including bluetooth headsets and remote systems
US20100114573A1 (en) * 2008-10-30 2010-05-06 Motorola, Inc. Method and Device for Verifying a User
EP2545674A2 (de) * 2010-03-10 2013-01-16 SRA International, Inc. One-vault-sprachverschlüsselung
US8359020B2 (en) * 2010-08-06 2013-01-22 Google Inc. Automatically monitoring for voice input based on context
EP2451139B1 (de) * 2010-11-03 2015-01-07 BlackBerry Limited Zugriff auf gesperrte Funktionen
CN102158470B (zh) * 2011-01-30 2013-10-16 韦峻峰 一种用于移动便携式设备的声信号处理系统及其处理方法
US8713660B2 (en) * 2011-05-16 2014-04-29 Tec Solutions, Inc. Authentication platform and related method of operation
WO2013025590A1 (en) 2011-08-15 2013-02-21 Bank Of America Corporation Method and apparatus for making token-based access decisions
US8950002B2 (en) 2011-08-15 2015-02-03 Bank Of America Corporation Method and apparatus for token-based access of related resources
US8898459B2 (en) 2011-08-31 2014-11-25 At&T Intellectual Property I, L.P. Policy configuration for mobile device applications
US8996381B2 (en) 2011-09-27 2015-03-31 Sensory, Incorporated Background speech recognition assistant
US8452597B2 (en) 2011-09-30 2013-05-28 Google Inc. Systems and methods for continual speech recognition and detection in mobile computing devices
KR20130133629A (ko) 2012-05-29 2013-12-09 삼성전자주식회사 전자장치에서 음성명령을 실행시키기 위한 장치 및 방법
US9191823B2 (en) * 2012-06-29 2015-11-17 GSMK Gesellschaft für sichere mobile Kommunikation mbH Mobile device and method to monitor a baseband processor in relation to the actions on an applicaton processor
US9875741B2 (en) 2013-03-15 2018-01-23 Google Llc Selective speech recognition for chat and digital personal assistant systems
CN203180938U (zh) * 2012-12-21 2013-09-04 国民技术股份有限公司 电子令牌和电子令牌认证系统
US11393461B2 (en) 2013-03-12 2022-07-19 Cerence Operating Company Methods and apparatus for detecting a voice command
US9058805B2 (en) 2013-05-13 2015-06-16 Google Inc. Multiple recognizer speech recognition
US20140343949A1 (en) 2013-05-17 2014-11-20 Fortemedia, Inc. Smart microphone device
US9723487B2 (en) * 2013-08-19 2017-08-01 American Megatrends, Inc. Mobile device security system
US8719039B1 (en) 2013-12-05 2014-05-06 Google Inc. Promoting voice actions to hotwords
US9684778B2 (en) * 2013-12-28 2017-06-20 Intel Corporation Extending user authentication across a trust group of smart devices
US9270931B2 (en) 2014-05-05 2016-02-23 Cloudtalk Llc Intercom system utilizing Wi-Fi
WO2016049235A1 (en) * 2014-09-23 2016-03-31 UV Networks, Inc. Application host with distributed remote input and output interfaces
US10103872B2 (en) 2014-09-26 2018-10-16 Intel Corporation Securing audio communications
CN204103936U (zh) * 2014-10-31 2015-01-14 上海众人科技有限公司 一种动态令牌及动态令牌认证系统
US9891882B2 (en) 2015-06-01 2018-02-13 Nagravision S.A. Methods and systems for conveying encrypted data to a communication device
US20160365101A1 (en) * 2015-06-15 2016-12-15 Motorola Mobility Llc Enabling Event Driven Voice Interaction with a Device
US20170012982A1 (en) * 2015-07-10 2017-01-12 Google Inc. Protecting Data From Unauthorized Access
CN105933893A (zh) 2016-03-30 2016-09-07 苏州美天网络科技有限公司 一种蓝牙耳机对讲系统的数据加密系统

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
None *

Also Published As

Publication number Publication date
CN110036616A (zh) 2019-07-19
CN110036616B (zh) 2021-08-13
EP3533196A1 (de) 2019-09-04
WO2018106293A1 (en) 2018-06-14
US20180190290A1 (en) 2018-07-05
US9940930B1 (en) 2018-04-10

Similar Documents

Publication Publication Date Title
EP3533196B1 (de) Sicherung von audiodaten
US10277577B2 (en) Password-less authentication system and method
US11765158B1 (en) Multi-factor authentication systems and methods
KR102369955B1 (ko) 차량 환경에서의 다중 인자 인증 및 액세스 제어
US9641521B2 (en) Systems and methods for network connected authentication
US10445487B2 (en) Methods and apparatus for authentication of joint account login
JP2019531567A (ja) 装置認証のシステム及び方法
US10225260B2 (en) Enhanced authentication security
KR20160097323A (ko) Nfc 인증 메커니즘
CN103813334A (zh) 权限控制方法和权限控制装置
US9801061B2 (en) Multi-factor user authentication based on decoy security questions
US10129299B1 (en) Network beacon management of security policies
Gao et al. Traversing the quagmire that is privacy in your smart home
US10021092B1 (en) Systems and methods for device authentication
WO2017185683A1 (zh) 基于生物识别信息的认证方法和认证系统、电子设备
EP3651034A1 (de) Systeme und verfahren zur wasserzeichenmarkierung von audio von saas-anwendungen
US20200110859A1 (en) Controlling access to computer resources by user authentication based on unique authentication patterns
US10063592B1 (en) Network authentication beacon
TWI816017B (zh) 驗證資訊處理方法和裝置
JP2015158881A (ja) セッションハイジャック防止のためのアクセス可否管理システム、プログラム
Phipps et al. Your password is music to my ears: Cloud based authentication using sound
KR102118956B1 (ko) 인증 시스템 및 방법
Cetin Authentication and SQL-Injection Prevention Techniques in Web Applications
CN113868634A (zh) 存储系统用户验证方法、装置、电子设备及存储介质
KR20180085499A (ko) 전자서명을 이용한 모바일 기기 제어 시스템 및 방법

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20190530

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20191209

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602017017400

Country of ref document: DE

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1275752

Country of ref document: AT

Kind code of ref document: T

Effective date: 20200615

REG Reference to a national code

Ref country code: DE

Ref legal event code: R082

Ref document number: 602017017400

Country of ref document: DE

Representative=s name: VENNER SHIPLEY GERMANY LLP, DE

Ref country code: DE

Ref legal event code: R082

Ref document number: 602017017400

Country of ref document: DE

Representative=s name: VENNER SHIPLEY LLP, DE

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200827

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200527

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200527

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200928

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200927

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200527

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200828

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20200527

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200827

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200527

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200527

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200527

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1275752

Country of ref document: AT

Kind code of ref document: T

Effective date: 20200527

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200527

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200527

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200527

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200527

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200527

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200527

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200527

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200527

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200527

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200527

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200527

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200527

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602017017400

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200527

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200831

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200801

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200831

26N No opposition filed

Effective date: 20210302

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20200831

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200831

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200831

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200801

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Ref document number: 602017017400

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: H04L0029060000

Ipc: H04L0065000000

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200527

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200527

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200527

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200527

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230508

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200527

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20230828

Year of fee payment: 7

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20230829

Year of fee payment: 7