KR20230146964A - Method for processing voice data and electronic device supporting the same - Google Patents

Abstract

An electronic device including a microphone, a speaker, a memory storing at least one instruction, and a processor is disclosed. The instructions, when executed by a processor, cause the electronic device to obtain audio input, including voice data and noise, using a microphone, and, if the voice data is identified as including a specified call word, retrieve voice data from the audio input. and respectively extract noise, calculate the separation distance between the user and the electronic device based on the voice data, identify the first volume of the extracted voice data, identify the second volume of the extracted noise, and identify the first volume of the extracted noise. Calculate the SNR based on the volume, the second volume, and the separation distance, calculate a third volume corresponding to the calculated SNR, and if the difference between the preset volume and the third volume exceeds the specified value, the preset volume can be set to change to the third volume and output a response to the voice data at the third volume using a speaker.

Description

Voice data processing method and electronic device supporting the same {METHOD FOR PROCESSING VOICE DATA AND ELECTRONIC DEVICE SUPPORTING THE SAME}

Embodiments disclosed in this document relate to a voice data processing method and an electronic device supporting the same.

As electronic devices that perform various functions using voice data are rapidly spreading, various techniques are being proposed to increase the accuracy of voice data recognition technology.

Technologies for inferring the speaker's emotions based on speech emotion recognition (SER) or identifying the speaker based on speaker identification (SI) are widely used.

STT (speech to text) can be used to analyze such voice data. For example, an electronic device can obtain voice data and convert it into a string to identify the speaker's intention.

Voice recognition devices (e.g., smartphones, AI speakers) may include physical components (e.g., key buttons) that allow adjustment of output volume. The user can directly manipulate the components to adjust the output volume of the voice recognition device.

Despite advances in voice recognition technology, it may be difficult for electronic devices to operate at optimal performance due to changes in the environment in which a speaker speaks. For example, when the noise around the speaker is relatively high or the volume of the speaker is low, it may be difficult for the electronic device to adaptively provide various functions according to changes in the environment.

Electronic devices have difficulty in fluidly controlling the output signal output in response to the speaker's utterance. For example, in situations where the output volume must be increased or decreased (e.g., in a noisy environment), there is an inconvenience in that the user must directly control the volume of the electronic device.

According to various embodiments of the present disclosure, an electronic device that analyzes acquired speech, identifies it separately from surrounding noise, and flexibly controls the output volume through a signal-to-noise ratio (SNR) calculated based on various parameters. Devices can be provided.

An electronic device according to an embodiment of the present disclosure may include a microphone, a speaker, a memory that stores at least one instruction, and a processor. For example, the instructions, when executed by the processor, cause the electronic device to obtain an audio input including voice data and noise using the microphone, and the voice data is a designated call word ( wake-up word), extract the voice data and the noise from the audio input, calculate the separation distance between the user and the electronic device based on the voice data, and calculate the extracted voice Identify a first volume of data, identify a second volume of noise extracted, and determine signal-to-noise (SNR) based on the identified first volume, the second volume, and the separation distance. ratio), calculate a third volume corresponding to the calculated SNR, and if the difference between the preset volume and the third volume exceeds a specified value, change the preset volume to the third volume, , It can be set to output a response to the voice data at the third volume using the speaker.

A method of processing voice data by an electronic device according to an embodiment of the present disclosure includes obtaining audio input including voice data and noise using a microphone, and using a designated call word (wake word) for the voice data. -up word), an operation of extracting the voice data and the noise from the audio input, an operation of calculating the separation distance between the user and the electronic device based on the voice data, and the extracted An operation of identifying a first volume of voice data, an operation of identifying a second volume of the extracted noise, a signal-to-noise ratio (SNR) based on the identified first volume, the second volume, and the separation distance. An operation of calculating a to-noise ratio), an operation of calculating a third volume corresponding to the calculated SNR, if the difference between a preset volume and the third volume exceeds a specified value, the preset volume is converted to the third volume. It may include an operation of changing the volume to 3, and an operation of outputting a response to the voice data to the third volume using a speaker.

According to various embodiments of the present disclosure, usability of the voice recognition function is increased by adaptively controlling the volume of output data in overall consideration of the user's surrounding environment, the user's emotions, the volume of the user's speech input, or a combination thereof. You can do it.

In addition, various effects that can be directly or indirectly identified through this document may be provided.

Figure 1 is a block diagram showing an integrated intelligence system according to an embodiment.
Figure 2 is a block diagram showing an integrated intelligence system according to one embodiment.
Figure 3 is a block diagram of an automatic speech recognition (ASR) module according to an embodiment.
Figure 4 is a block diagram of a natural language understanding module according to an embodiment.
FIG. 5 is a block diagram illustrating components included in an electronic device, according to an embodiment.
Figure 6 illustrates a process of separating audio data, according to one embodiment.
7 is a flowchart of an operation of an electronic device according to an embodiment.
8 is a flowchart of an operation of an electronic device according to an embodiment.
Figure 9 shows an electronic device in a network environment according to various embodiments.
In relation to the description of the drawings, identical or similar reference numerals may be used for identical or similar components.

Hereinafter, various embodiments of the present invention are described with reference to the accompanying drawings. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include various modifications, equivalents, and/or alternatives to the embodiments of the present invention.

Figure 1 is a block diagram showing an integrated intelligence system according to an embodiment.

Referring to Figure 1, the integrated intelligence system of one embodiment may include a user terminal 101, an intelligent server 200, and a service server 299.

According to the illustrated embodiment, the user terminal 101 includes a communication interface 110, an input/output (I/O) interface 120, at least one processor 130, and/or a memory 140. It can be included. The components listed above may be operatively or electrically connected to each other.

The communication interface 110 can be connected to an external device (eg, the intelligent server 200 or the service server 299) through the network 119 to transmit and receive data. The I/O interface 120 uses an input/output device (e.g., microphone, speaker, and/or display) of the electronic device to receive user input, process the received user input, and/or process the processor 130. The results processed by can be output.

The processor 130 may be electrically connected to the communication interface 110, the I/O interface 120, and/or the memory 140 to perform designated operations. The processor 130 may perform a designated operation by executing a program (or one or more instructions) stored in the memory 140. For example, the processor 130 may receive a user's voice input (eg, user speech) through the I/O interface 120. The processor 130 may transmit the voice input received through the communication interface 110 to the intelligent server 200.

The processor 130 may receive a result corresponding to the voice input from the intelligent server 200. For example, the processor 130 may receive a plan corresponding to the voice input and/or a result calculated using the plan from the intelligent server 200. The processor 130 may receive a request from the intelligent server 200 to obtain information (eg, parameters) necessary to create a plan corresponding to the voice input. The processor 130 may transmit the necessary information to the intelligent server 200 in response to the request.

The processor 130 may output the results of executing a specified operation according to the plan through the I/O interface 120. For example, the processor 130 may sequentially display execution results of a plurality of operations on the display. As another example, the processor 130 may display only partial results of executing a plurality of operations (eg, the result of the last operation) on the display.

The processor 130 can recognize voice input that performs limited functions. For example, the processor 130 may execute an intelligent app (or voice recognition app) to process voice input in response to a designated voice input (e.g., wake up!). The processor 130 can provide a voice recognition service through an intelligent app. The processor 130 may transmit a voice input to the intelligent server 200 through an intelligent app and receive a result corresponding to the voice input from the intelligent server 200.

The intelligent server 200 in one embodiment may receive the user's voice input from the user terminal 101 through the network 119. The intelligent server 200 can convert audio data corresponding to the received voice input into text data. The intelligent server 200 may generate at least one plan for performing a task corresponding to the user's voice input based on the text data. The intelligent server 200 may transmit the generated plan or the results according to the generated plan to the user terminal 101 through the network 119.

The intelligent server 200 of one embodiment includes a front end 210, a natural language platform 220, a capsule database 230, an execution engine 240, and/or an end user interface (250).

The front end 210 may receive a voice input received by the user terminal 101 from the user terminal 101 . The front end 210 may transmit a response corresponding to the voice input to the user terminal 101.

The natural language platform 220 includes an automatic speech recognition module (ASR module) 221, a natural language understanding module (NLU module) 223, and a planner module 225. , a natural language generator module (NLG module) 227, and/or a text to speech module (TTS module) 229.

The automatic voice recognition module 221 can convert the voice input received from the user terminal 101 into text data. The natural language understanding module 223 may determine the user's intent and/or parameters based on text data of the voice input.

The planner module 225 may generate a plan using the intent and parameters determined by the natural language understanding module 223. The planner module 225 may determine a plurality of domains required to perform the task based on the determined intention. The planner module 225 may determine a plurality of operations included in each of the plurality of domains determined based on the intention. The planner module 225 may determine parameters required to execute the determined plurality of operations or result values output by executing the plurality of operations. The parameters and the result values may be defined as concepts of a specified type (or class). Accordingly, the plan may include a plurality of operations and/or a plurality of concepts determined by the user's intention. The planner module 225 may determine the relationship between the plurality of operations and the plurality of concepts in a stepwise (or hierarchical) manner. For example, the planner module 225 determines a plurality of operations based on the user's intention based on a plurality of concepts (e.g., parameters required for execution of a plurality of operations, and results output by execution of the plurality of operations). The execution order can be determined. The planner module 225 may generate a plan that includes association information (eg, ontology) between a plurality of operations and a plurality of concepts. The planner module 225 may create a plan using information (eg, at least one capsule) stored in the capsule database 230, which stores a set of relationships between concepts and operations.

The planner module 225 can create a plan based on an artificial intelligence (AI) system. For example, an artificial intelligence system may include a rule-based system, a neural network-based system (e.g., a feedforward neural network (FNN)), and/or a recurrent neural network ( It may be a recurrent neural network (RNN)), or a combination of the above, or it may be another artificial intelligence system. The planner module 225 may select a plan corresponding to a user request from a set of predefined plans or generate a plan in real time in response to a user request.

The natural language generation module 227 can change the specified information into text form. The information changed to the text form may be in the form of natural language speech. The text-to-speech conversion module 229 can convert information in text form into information in voice form.

The capsule database 230 may store information about the relationship between a plurality of concepts and operations corresponding to a plurality of domains (eg, applications). The capsule database 230 may store at least one capsule 231 or 233 in CAN (concept action network) format. For example, the capsule database 230 may store operations for processing tasks corresponding to the user's voice input and parameters necessary for the operations in CAN format. A capsule may include a plurality of action objects (or action information) and/or concept objects (or concept information) included in the plan.

The execution engine 240 may calculate a result using the generated plan. The end user interface 250 may transmit the calculated result to the user terminal 101.

According to one embodiment, some functions (eg, natural language platform 220) or all functions of the intelligent server 200 may be implemented in the user terminal 101. For example, the user terminal 101 includes a natural language platform separate from the intelligent server 200, or the natural language platform 220 of the intelligent server 200 (e.g., automatic speech recognition module 221, natural language understanding module 223 ), the planner module 225, the natural language generation module 227, and/or the text-to-speech conversion module 229) may be directly performed.

The service server 299 in one embodiment may provide a designated service (eg, food ordering or hotel reservation) to the user terminal 101. The service server 299 may be a server operated by a third party. The service server 299 may communicate with the intelligent server 200 and/or the user terminal 101 through the network 119. The service server 299 can communicate with the intelligent server 200 through a separate connection. The service server 299 provides information for creating a plan corresponding to the voice input received at the user terminal 101 (e.g., operation information and/or concept information for providing a designated service) to the intelligent server 200. can do. The provided information may be stored in the capsule database 230. The service server 299 may provide result information according to the plan received from the user terminal 101 to the intelligent server 200.

Figure 2 is a block diagram showing an integrated intelligence system according to an embodiment.

Referring to Figure 2, the integrated intelligence system may include a user terminal 103 and an intelligent server 200. The user terminal 103 and the intelligent server 202 are connected to each other through a network and can transmit and receive data. According to one embodiment, the integrated intelligence system may be composed of a single device (eg, user terminal 103 or intelligent server 202). According to one embodiment, the intelligent server 202 may include the entire configuration or at least a partial configuration of the intelligent server 200 shown in FIG. 1. For example, intelligent server 202 may include the natural language platform 220 and/or capsule database 230 of intelligent server 200 of FIG. 1 . However, the configuration of the intelligent server 202 is not limited to that shown in FIG. 2, and may include at least some components of the natural language platform 220 (e.g., automatic speech recognition module 221, natural language understanding module 223, planner module ( 225), natural language generation module 227, and/or text-to-speech conversion module 229) may be omitted, and some components of the intelligent server 200 of FIG. 1 (e.g., front end 210, execution engine ( 240) and/or an end user interface 250) may be further included.

The user terminal 103 may include a natural language platform 150 and/or a capsule database 160. For example, the user terminal 103 may include components of the user terminal 101 of FIG. 1 and may further include a natural language platform 150 and/or a capsule database 160.

The natural language platform 150 includes an automatic speech recognition (ASR) module 151, a natural language understanding module (NLU) 153, a planner module 155, and a natural language generation module (NLG). (natural language generator) module) 157, and/or a text to speech (TTS) module 159. The automatic speech recognition module 151, the natural language understanding module 153, the planner module 155, the natural language generation module 157, and the text-to-speech conversion module 159 are respectively the automatic speech recognition module 221 of FIG. 1 and the natural language It may perform the same or similar functions as the understanding module 223, the planner module 225, the natural language generation module 227, and the text-to-speech conversion module 229.

The capsule database 160 may perform the same or similar functions as the capsule database 230 of the intelligent servers 200 and 202. The capsule database 160 may store information about relationships between a plurality of operations and a plurality of concepts included in the plan generated by the planner module 155. For example, the capsule database 160 may store at least one capsule 161 or 163.

According to one embodiment, a user terminal 103 (e.g., natural language platform 150 and/or capsule database 160) and an intelligent server 202 (e.g., natural language platform 220 and/or capsule database 230) ) may perform at least one function (or operation) in conjunction with each other, or may perform at least one function (or operation) independently. For example, the user terminal 103 may perform voice recognition on its own without transmitting the received user's voice input to the intelligent server 202. As an example, the user terminal 103 may convert voice input received through the automatic voice recognition module 151 into text data. The user terminal 103 can transmit the converted text data to the intelligent server 202. The intelligent server 202 may determine the user's intent and/or parameters from text data through the natural language understanding module 223. The intelligent server 202 generates a plan through the planner module 225 based on the determined intent and parameters and transmits it to the user terminal 103, or transmits the determined intent and parameters to the user terminal 103 to prepare the plan for the user terminal 103. You can create a plan through the planner module 155 of ). The planner module 155 of the user terminal 103 may generate at least one plan for performing a task corresponding to a voice input using information stored in the capsule database 160.

As another example, the user terminal 103 converts the voice input received through the automatic voice recognition module 151 into text data, and determines the user's intention and/or parameters based on the text data through the natural language understanding module 153. You can decide. The user terminal 103 generates a plan through the planner module 155 based on the determined intent and parameters, or transmits the determined intent and parameters to the intelligent server 202 to use the planner module 225 of the intelligent server 202. You can create a plan through. For example, if the user terminal 103 does not include the planner module 155 and/or the capsule database 160, the user terminal 103 may create a plan through the intelligent server 202.

As another example, the user terminal 103 detects a speech pattern that is difficult to learn in the automatic speech recognition module 151 or the natural language understanding module 153, and the speech input corresponding to the detected speech pattern is sent to the intelligent server 202. It can be transmitted and processed by the automatic voice recognition module 221 or the natural language understanding module 223 of the intelligent server 202.

Embodiments of the present disclosure are not limited to the above-described examples. For example, the user terminal 103 may process the received voice input only within the terminal and even calculate a result corresponding to the voice input. As another example, the user terminal 103 and the intelligent server 202 not only divide voice input into modules and process them, but also process them collaboratively between corresponding modules. For example, the natural language understanding module 153 of the user terminal 103 and the natural language understanding module 223 of the intelligent server 202 can operate together to calculate one result value (user intent and/or parameters). there is.

Figure 3 is a block diagram of the automatic voice recognition module 221 according to one embodiment.

The automatic voice recognition module 221 generates text data using voice input. For example, the automatic voice recognition module 221 acquires voice input through an I/O interface and/or an external device (e.g., user terminals 101, 103), and processes the acquired voice input to create text. Data can be output.

The front end 310 performs preprocessing operations on voice input. For example, the front end 310 may remove the echo of the voice input using an echo cancellation module (e.g., an acoustic echo canceller (AEC) and/or a residual echo suppressor (REC)). For another example, the front end 310 may remove noise from voice input using a noise removal module (e.g., noise suppressor (NS)).

The End-Point Detector (EPD) 320 detects the end point of voice input. For example, the EPD 320 may detect the end point of a voice input and identify the start and end points of a user's speech corresponding to the voice input based on the detection result.

The wake-up module 330 selectively transmits the voice input to the automatic voice recognition module 340 based on whether a designated wake-up word (or wake-up signal) is present in the voice input. For example, the wakeup module 330 may identify whether a specified call word exists in the voice input preprocessed from the front end 310 through keyword spotting.

The automatic voice recognition model 340 obtains text data using voice input. The automatic speech recognition model 340 may extract text data from speech input and/or convert the speech input into text data based on a speech to text (STT) algorithm. For example, the speech to text (STT) algorithm may be a hidden Markov model, a Gaussian-Mixture model, a deep neural network model, an n-gram language model, and/or Other statistical models may be included.

The structure of the automatic voice recognition module 221 shown in FIG. 3 is illustrative, and embodiments of the present disclosure are not limited thereto.

As an example, the automatic speech recognition module 221 may use an end to end (E2E) model (e.g., connectionist temporal classification (CTC), recurrent neural network transducer (RNN-T), listen, attend, and spell (LAS), or hybrid Can be implemented with CTC/LAS.

As an example, the automatic voice recognition module 221 may further include other components. For example, the automatic speech recognition module 221 may further include a feature extraction module (not shown) and an encoder (not shown). The automatic speech recognition module 221 extracts features from the voice input, obtains a feature vector, and encodes the feature vector using an encoder.

Figure 4 is a block diagram of the natural language understanding module 223.

Referring to FIG. 4, the natural language understanding module 223 may include a dispatcher (410), a domain classifier (420), a built-in NLU module (430), or a combination thereof.

The natural language understanding module 223 obtains a voice recognition result (eg, data converted from one or more phonemes included in the user's utterance) from the automatic voice recognition module 221. The natural language understanding module 223 provides processing results of the voice recognition results to the planner module 225. The processing result of the voice recognition result may include an intent, target device, capsule, or a combination thereof.

The natural language understanding model 410 determines the intent by interpreting (e.g., syntactic analysis and/or semantic analysis) the speech recognition result from the automatic speech recognition module 221. Here, data converted from one or more phonemes may represent one or more words included in the user's utterance, and/or a token for each of the one or more words. The intent may be data used by the natural language platform 220 to create a plan. Intents may include goals and/or parameters. Goals are used in the planner module 225 to specify the final goal of the plan. A parameter is a value input to one or more operations included in the plan in the planner module 225.

The dispatcher 420 determines a target device and/or capsule (or domain, application) related to one or more words included in the voice recognition result.

The dispatcher 420 may include a device dispatcher 421, a named dispatcher 423, a meta command dispatcher 425, or a combination thereof.

The device dispatcher 421 determines one or more target devices based on the name of the device included in the voice recognition result. The target device may be an IoT (internet of things) device.

The named dispatcher 423 determines one or more capsules based on the name of the capsule included in the voice recognition result.

The meta command dispatcher 425 determines one or more capsules based on a specific command included in the voice recognition result. A specific command may be a command designated for a specific situation. Specific situations may include situations in which a voice recognition service prompts the user, and/or situations in which content can be selected from a content list containing one or more contents. For example, in a prompt situation, a specific command may be specified as select (e.g. 'first'), cancel ('cancel'), confirm ('ok'), or a combination thereof. For another example, in a situation where selection of content is possible, a specific command may be designated as repeat (eg, 'again'), next ('next'), previous ('previous'), or a combination thereof.

The domain classifier 430 determines one or more capsules required to perform a task according to the voice recognition result. The domain classifier 430 may determine one or more capsules using predefined classification rules and/or artificial intelligence models. Predefined classification rules and/or artificial intelligence models can be learned by a learning algorithm.

FIG. 5 is a block diagram illustrating components included in an electronic device, according to an embodiment.

According to one embodiment, the electronic device 501 includes a processor 520 (e.g., processor 130 in FIG. 1), a memory 530 (e.g., memory 140 in FIG. 1), and a microphone 530 (e.g. : May include the I/O interface 120 of FIG. 1), and a speaker 550 (e.g., the I/O interface 120 of FIG. 1). For example, the electronic device 501 may not include some of the components shown in FIG. 5 or may further include at least one component (eg, sensor) not shown in FIG. 5 .

In one embodiment, the processor 520 may execute instructions stored in the memory 530. For example, the processor 520 can control the microphone 540 and speaker 550 by executing instructions.

In one embodiment, the processor 520 may obtain various audio inputs using the microphone 540.

As an example, the processor 520 may use the microphone 540 to obtain (or capture) audio input including voice data and noise. For example, voice data may include the user's spoken voice. For example, noise may include noise around the user.

In one embodiment, processor 520 may identify specific words included in voice data.

As an example, processor 520 may identify whether voice data includes a designated call word. For example, the processor 520 may identify whether a specified call word exists in the voice data based on a keyword detection algorithm.

As an example, if voice data is identified as not containing a designated call word, processor 520 may not perform processing on the audio input.

As an example, if voice data is identified as containing a designated call word, processor 520 may perform processing on the audio input. For example, the processor 520 may extract at least one piece of data included in the audio input. For example, the processor 520 may extract voice data and noise included in the audio input. For example, the processor 520 may separate audio input into voice data and noise and extract each, based on the Hidden Markov Model. For another example, the processor 520 may extract voice data and noise from audio input, respectively, using an artificial intelligence model. The artificial intelligence model may include various models learned through deep learning.

In one embodiment, processor 520 may use some of the extracted data to determine various parameters of the audio data (e.g., volume, characteristics, and/or separation distance).

As an example, the processor 520 may calculate the separation distance between the user and the electronic device 501 based on voice data. For example, the processor 520 may calculate the separation distance using the first frequency band signal and the second frequency band signal included in the voice data. For example, the first frequency band signal may include a higher band signal than the second frequency band signal. The processor 520 may calculate the separation distance between the user and the electronic device 501 using the offset rate of the first frequency band signal and the second frequency band signal.

As an example, the processor 520 may correct the calculated separation distance. For example, the electronic device 501 may further include at least one sensor. The processor 520 may identify information about the surrounding environment of the electronic device 501 using the at least one sensor and correct the separation distance based on the identified information. For example, the at least one sensor may include a temperature sensor and a humidity sensor. For example, the information may include information about the ambient temperature and humidity of the electronic device 501.

As an example, processor 520 may identify the amplitude, frequency, and/or waveform of voice data. For example, processor 520 may identify a first volume corresponding to the intensity of the voice data based on the amplitude and/or frequency of the identified voice data. The first volume may be defined as an average volume value during a designated section of voice data acquired using the microphone 540. For example, processor 520 may identify acoustic characteristics of voice data based on the identified waveform. Acoustic characteristics may include results identified by performing emotion analysis on voice data based on deep neural networks (DNN). Acoustic characteristics may include, for example, anger, boredom, fear, happiness, and/or sadness.

As an example, processor 520 may identify the amplitude, frequency, and/or waveform of the noise. For example, processor 520 may identify a second volume corresponding to the intensity of noise based on the amplitude and/or frequency of the identified voice data. The second volume may be defined as the maximum volume value of noise during a specified section obtained using the microphone 540.

In one embodiment, the processor 520 may calculate a signal-tonoise ratio (SNR) based on at least some of the calculated parameters.

As an example, the processor 520 may calculate the SNR based on the first volume, the second volume, and the separation distance. For example, SNR may be defined as the ratio of voice data corresponding to the user's voice among audio input and components (eg, noise) excluding the voice data. For example, SNR may be proportional to the first volume. As another example, SNR may be inversely proportional to the secondary volume and separation distance. For example, as the SNR increases, the proportion of voice data may be relatively larger than the proportion of noise. For example, as the SNR is smaller, the proportion of voice data may be relatively smaller than the proportion of noise. For another example, the smaller the SNR, the larger the separation distance between the user and the electronic device 501 may be.

In one embodiment, processor 520 may calculate a third volume corresponding to the calculated SNR.

As an example, the processor 520 may use SNR to calculate a third volume corresponding to the strength of the response to voice data.

As an example, the processor 520 may calculate the third volume such that the strength of the response to voice data is proportional to the first volume and the second volume. For example, the third volume can be calculated so that the louder the user's speech, the louder the response. For example, the third volume can be calculated so that the larger the noise around the user, the larger the response.

In one embodiment, the processor 520 may determine the volume of the response to voice data based on the difference between the preset volume and the third volume.

For example, if the difference between the preset volume and the third volume exceeds a specified value, the processor 520 may change the preset volume to the third volume. That is, if the preset volume has a large difference from the third volume calculated based on the SNR, it is inappropriate to output the response of the preset volume, so the volume for the response can be changed to the third volume.

For example, if the difference between the preset volume and the third volume is less than or equal to a specified value, the processor 520 may determine to output a response at the preset volume.

In one embodiment, the processor 520 may output a response to voice data using the speaker 550 based on the determined volume.

As an example, the processor 520 may output a response to voice data at a third volume or a preset volume.

In one embodiment, the processor 520 may add the acoustic characteristics of the identified voice data to the response and output it.

For example, when the processor 520 identifies an acoustic characteristic indicating that the user speaks in a quiet voice based on voice data, the processor 520 may output a response based on the identified acoustic characteristic.

For example, when the processor 520 identifies an acoustic characteristic indicating that the user speaks in a shouting voice based on voice data, the processor 520 may output a response based on the identified acoustic characteristic.

Figure 6 illustrates a process of separating audio data, according to one embodiment.

According to one embodiment, an electronic device (e.g., electronic device 501 of FIG. 5) acquires an audio input using a microphone (e.g., microphone 540 of FIG. 5) and converts the obtained audio input into a plurality of data. Each can be extracted separately.

Reference number 610 indicates various characteristics of audio input during a designated section.

In one embodiment, the audio input may include voice data corresponding to the user's voice and noise around the user. The electronic device can extract voice data and noise from the audio input, respectively, using a Hidden Markov Model and/or an artificial intelligence model.

Reference numeral 620 represents various characteristics of voice data for a specified section extracted from audio input.

In one embodiment, the electronic device may identify various characteristics of the voice data (eg, first volume, acoustic characteristics) based on the amplitude, frequency, and/or waveform of the extracted voice data.

Reference numeral 630 represents various characteristics of noise during a specified period extracted from the audio input.

In one embodiment, the electronic device may identify various characteristics of the noise (eg, a second volume) based on the amplitude and/or frequency of the extracted noise.

7 is a flowchart of an operation of an electronic device according to an embodiment.

According to one implementation, an electronic device (eg, the user terminals 101 and 103 of FIGS. 1 and 2 or the electronic device 501 of FIG. 5) may perform the operations disclosed in FIG. 7. For example, a processor of an electronic device (e.g., processor 520 of FIG. 5) may be set to perform the operations of FIG. 7 when executing instructions stored in a memory (e.g., memory 530 of FIG. 5). .

In operation 705, the electronic device can obtain audio input.

For example, the electronic device may use a microphone (e.g., microphone 540 in FIG. 5) to obtain audio input including voice data and noise. As an example, the voice data may be audio data including the user's spoken voice. As another example, the noise may be audio data including noise around the user.

For example, the electronic device can identify specific words included in the acquired audio input. As an example, the electronic device can identify whether the audio input (or voice data) includes a designated call word. The electronic device can identify whether a specified call word exists in the voice data based on a keyword detection algorithm.

In operation 710, the electronic device may extract voice data and noise.

For example, if the electronic device determines in operation 705 that a specified call word exists in the voice data, the electronic device may extract voice data and noise from the audio input. For example, the electronic device may extract voice data and noise from the audio input using a Hidden Markov Model and/or an artificial intelligence model, respectively. Artificial intelligence models may include various models learned by deep learning.

In operation 715, the electronic device may calculate the separation distance between the user and the electronic device based on the extracted voice data.

For example, the electronic device may calculate the separation distance using the first frequency band signal and the second frequency band signal included in the voice data. For example, the first frequency band signal may include a higher band signal than the second frequency band signal. The electronic device may calculate the separation distance between the user and the electronic device using the cancellation rate of the first frequency band signal and the second frequency band signal.

For example, the electronic device can correct the calculated separation distance. As an example, the electronic device may identify information about the surrounding environment of the electronic device using at least one sensor and correct the separation distance based on the identified information. The at least one sensor may include, for example, a temperature sensor and a humidity sensor. The information may include, for example, information about the ambient temperature and humidity of the electronic device.

In operation 720, the electronic device may identify a first volume of voice data and a second volume of noise.

For example, the electronic device may identify a first volume corresponding to the intensity of the voice data based on the amplitude and/or frequency of the voice data. For example, the electronic device may identify a second volume corresponding to the intensity of the noise based on the amplitude and/or frequency of the noise. The first volume may be defined, for example, as an average volume value during a designated section of voice data. The second volume may be defined, for example, as the maximum volume value of noise during a designated period.

For example, the electronic device may use the waveform of the voice data to further identify acoustic characteristics of the voice data. Acoustic characteristics may include results identified by performing emotional analysis on voice data based on a deep artificial neural network. Acoustic characteristics may include, for example, angry, bored, scared, happy, and/or sad.

In operation 725, the electronic device may calculate the SNR based on the first volume, the second volume, and/or the separation distance.

For example, the electronic device may calculate the SNR to be proportional to the first volume. As another example, the electronic device may calculate the SNR to be inversely proportional to the second volume and the separation distance.

In operation 730, the electronic device may calculate a third volume corresponding to the SNR.

For example, the electronic device may use SNR to calculate a third volume corresponding to the strength of the response to voice data.

For example, the electronic device may calculate the third volume such that the strength of the response to voice data is proportional to the first volume and the second volume. For example, the third volume can be calculated so that the louder the user's speech, the louder the response. For example, the third volume can be calculated so that the larger the noise around the user, the larger the response.

In operation 735, the electronic device may output a response to the voice data in a third volume.

For example, the electronic device may add the acoustic characteristics of the voice data identified in operation 720 to the response to the voice data. For example, when the electronic device identifies an acoustic characteristic indicating that the user speaks in a quiet voice based on voice data, the electronic device may output a response based on the identified acoustic characteristic. As another example, when the electronic device identifies an acoustic characteristic that indicates that the user speaks with a shouting voice based on voice data, the electronic device may output a response based on the identified acoustic characteristic.

In the following description of FIG. 8, the operation of the electronic device that determines the volume for a response based on the difference between the calculated third volume and the preset volume may be described later.

8 is a flowchart of an operation of an electronic device according to an embodiment.

According to one implementation, an electronic device (eg, the user terminals 101 and 103 of FIGS. 1 and 2 or the electronic device 501 of FIG. 5) may perform the operations disclosed in FIG. 8. For example, the processor of the electronic device (e.g., the processor 520 of FIG. 5) may be set to perform the operations of FIG. 8 when executing instructions stored in the memory (e.g., the memory 530 of FIG. 5). .

In operation 805, the electronic device may calculate a third volume corresponding to the SNR. The description of operation 805 can be replaced by the description of operations 705 to 730 of FIG. 7 described above.

In operation 810, the electronic device may calculate the difference between the preset volume and the third volume.

For example, the preset volume may be defined as a volume value for a response set by the user by manipulating the electronic device before the electronic device acquires audio input.

In operation 815, if the difference is identified as exceeding a specified value (e.g., operation 815 - Yes), the electronic device may perform operation 820.

In operation 815, if the difference is identified as being less than or equal to a specified value (e.g., operation 815 - No), the electronic device may perform operation 825.

In operation 820, the electronic device may change the preset volume to the third volume.

For example, if the preset volume has a large difference from the third volume calculated based on SNR, it is inappropriate to output a response with the preset volume, so the electronic device changes the volume for the response to the third volume. You can.

In operation 825, the electronic device may maintain a preset volume.

For example, if the difference with the third volume calculated based on the preset volume and SNR is small, the usability is not significantly reduced even if a response with the preset volume is output, so the electronic device sets the volume for the response to the third volume. Instead of changing the volume, the response can be output at the preset volume.

In operation 830, the electronic device may output a response to voice data. The description of operation 830 can be replaced by the description of operation 735 of FIG. 7 described above.

Figure 9 shows an electronic device in a network environment according to various embodiments.

FIG. 9 is a block diagram of an electronic device 901 (eg, the electronic device 501 of FIG. 5 ) in a network environment 900, according to various embodiments. Referring to FIG. 9, in the network environment 900, the electronic device 901 communicates with the electronic device 902 through a first network 998 (e.g., a short-range wireless communication network) or a second network 999. It is possible to communicate with at least one of the electronic device 904 or the server 908 through (e.g., a long-distance wireless communication network). According to one embodiment, the electronic device 901 may communicate with the electronic device 904 through the server 908. According to one embodiment, the electronic device 901 includes a processor 920 (e.g., processor 520 in FIG. 5), a memory 930 (e.g., memory 530 in FIG. 5), and an input module 950 ( Example: microphone 540 in FIG. 5), sound output module 955 (e.g. speaker 550 in FIG. 5), display module 960, audio module 970, sensor module 976, interface 977 ), connection terminal 978, haptic module 979, camera module 980, power management module 988, battery 989, communication module 990, subscriber identification module 996, or antenna module 997. ) may include. In some embodiments, at least one of these components (eg, the connection terminal 978) may be omitted, or one or more other components may be added to the electronic device 901. In some embodiments, some of these components (e.g., sensor module 976, camera module 980, or antenna module 997) are integrated into one component (e.g., display module 960). It can be.

Processor 920 may, for example, execute software (e.g., program 940) to operate at least one other component (e.g., hardware or software component) of electronic device 901 connected to processor 920. It can be controlled and various data processing or calculations can be performed. According to one embodiment, as at least part of data processing or computation, the processor 920 stores commands or data received from another component (e.g., sensor module 976 or communication module 990) in volatile memory 932. The commands or data stored in the volatile memory 932 can be processed, and the resulting data can be stored in the non-volatile memory 934. According to one embodiment, the processor 920 may include a main processor 921 (e.g., a central processing unit or an application processor) or an auxiliary processor 923 that can operate independently or together (e.g., a graphics processing unit, a neural network processing unit ( It may include a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device 901 includes a main processor 921 and a auxiliary processor 923, the auxiliary processor 923 may be set to use lower power than the main processor 921 or be specialized for a designated function. You can. The auxiliary processor 923 may be implemented separately from the main processor 921 or as part of it.

The auxiliary processor 923 may, for example, act on behalf of the main processor 921 while the main processor 921 is in an inactive (e.g., sleep) state, or while the main processor 921 is in an active (e.g., application execution) state. ), together with the main processor 921, at least one of the components of the electronic device 901 (e.g., the display module 960, the sensor module 976, or the communication module 990) At least some of the functions or states related to can be controlled. According to one embodiment, co-processor 923 (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module 980 or communication module 990). there is. According to one embodiment, the auxiliary processor 923 (eg, neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. Artificial intelligence models can be created through machine learning. For example, such learning may be performed in the electronic device 901 itself on which the artificial intelligence model is performed, or may be performed through a separate server (e.g., server 908). Learning algorithms may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but It is not limited. An artificial intelligence model may include multiple artificial neural network layers. Artificial neural networks include deep neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN), restricted boltzmann machine (RBM), belief deep network (DBN), bidirectional recurrent deep neural network (BRDNN), It may be one of deep Q-networks or a combination of two or more of the above, but is not limited to the examples described above. In addition to hardware structures, artificial intelligence models may additionally or alternatively include software structures.

The memory 930 may store various data used by at least one component (eg, the processor 920 or the sensor module 976) of the electronic device 901. Data may include, for example, input data or output data for software (e.g., program 940) and instructions related thereto. Memory 930 may include volatile memory 932 or non-volatile memory 934.

The program 940 may be stored as software in the memory 930 and may include, for example, an operating system 942, middleware 944, or application 946.

The input module 950 may receive commands or data to be used in a component of the electronic device 901 (e.g., the processor 920) from outside the electronic device 901 (e.g., a user). The input module 950 may include, for example, a microphone, mouse, keyboard, keys (eg, buttons), or digital pen (eg, stylus pen).

The sound output module 955 may output sound signals to the outside of the electronic device 901. The sound output module 955 may include, for example, a speaker or receiver. Speakers can be used for general purposes such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display module 960 can visually provide information to the outside of the electronic device 901 (eg, a user). The display module 960 may include, for example, a display, a hologram device, or a projector, and a control circuit for controlling the device. According to one embodiment, the display module 960 may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of force generated by the touch.

The audio module 970 can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module 970 acquires sound through the input module 950, the sound output module 955, or an external electronic device (e.g., directly or wirelessly connected to the electronic device 901). Sound may be output through an electronic device 902 (e.g., speaker or headphone).

The sensor module 976 detects the operating state (e.g., power or temperature) of the electronic device 901 or the external environmental state (e.g., user state) and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 976 includes, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 977 may support one or more designated protocols that can be used to connect the electronic device 901 directly or wirelessly with an external electronic device (e.g., the electronic device 902). According to one embodiment, the interface 977 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 978 may include a connector through which the electronic device 901 can be physically connected to an external electronic device (eg, the electronic device 902). According to one embodiment, the connection terminal 978 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 979 can convert electrical signals into mechanical stimulation (e.g., vibration or movement) or electrical stimulation that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 979 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 980 can capture still images and moving images. According to one embodiment, the camera module 980 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 988 can manage power supplied to the electronic device 901. According to one embodiment, the power management module 988 may be implemented as at least a part of, for example, a power management integrated circuit (PMIC).

The battery 989 may supply power to at least one component of the electronic device 901. According to one embodiment, the battery 989 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

Communication module 990 provides a direct (e.g., wired) communication channel or wireless communication channel between electronic device 901 and an external electronic device (e.g., electronic device 902, electronic device 904, or server 908). It can support establishment and communication through established communication channels. Communication module 990 operates independently of processor 920 (e.g., an application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module 990 is a wireless communication module 992 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 994 (e.g., : LAN (local area network) communication module, or power line communication module) may be included. Among these communication modules, the corresponding communication module is a first network 998 (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 999 (e.g., legacy It may communicate with an external electronic device 904 through a telecommunication network such as a cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., LAN or WAN). These various types of communication modules may be integrated into one component (e.g., a single chip) or may be implemented as a plurality of separate components (e.g., multiple chips). The wireless communication module 992 uses subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 996 within a communication network such as the first network 998 or the second network 999. The electronic device 901 can be confirmed or authenticated.

The wireless communication module 992 may support 5G networks after 4G networks and next-generation communication technologies, for example, NR access technology (new radio access technology). NR access technology provides high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and access to multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low latency). -latency communications)) can be supported. The wireless communication module 992 may support high frequency bands (e.g., mmWave bands), for example, to achieve high data rates. The wireless communication module 992 uses various technologies to secure performance in high frequency bands, for example, beamforming, massive array multiple-input and multiple-output (MIMO), and full-dimensional multiplexing. It can support technologies such as input/output (FD-MIMO: full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module 992 may support various requirements specified in the electronic device 901, an external electronic device (e.g., electronic device 904), or a network system (e.g., second network 999). According to one embodiment, the wireless communication module 992 supports peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mmTC, or U-plane latency (e.g., 164 dB or less) for realizing URLLC. Example: Downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) can be supported.

The antenna module 997 may transmit or receive signals or power to or from the outside (e.g., an external electronic device). According to one embodiment, the antenna module 997 may include an antenna including a radiator made of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 997 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for the communication method used in the communication network, such as the first network 998 or the second network 999, is connected to the plurality of antennas by, for example, the communication module 990. can be selected Signals or power may be transmitted or received between the communication module 990 and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (eg, radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module 997.

According to various embodiments, antenna module 997 may form a mmWave antenna module. According to one embodiment, a mmWave antenna module includes: a printed circuit board, an RFIC disposed on or adjacent to a first side (e.g., bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., mmWave band); And a plurality of antennas (e.g., array antennas) disposed on or adjacent to the second side (e.g., top or side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( (e.g. commands or data) can be exchanged with each other.

According to one embodiment, commands or data may be transmitted or received between the electronic device 901 and the external electronic device 904 through the server 908 connected to the second network 999. Each of the external electronic devices 902 or 904 may be of the same or different type as the electronic device 901. According to one embodiment, all or part of the operations performed in the electronic device 901 may be executed in one or more of the external electronic devices 902, 904, or 908. For example, when the electronic device 901 must perform a function or service automatically or in response to a request from a user or another device, the electronic device 901 may perform the function or service instead of executing the function or service on its own. Alternatively, or additionally, one or more external electronic devices may be requested to perform at least part of the function or service. One or more external electronic devices that have received the request may execute at least part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device 901. The electronic device 901 may process the result as is or additionally and provide it as at least part of a response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology can be used. The electronic device 901 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 904 may include an Internet of Things (IoT) device. Server 908 may be an intelligent server using machine learning and/or neural networks. According to one embodiment, the external electronic device 904 or server 908 may be included in the second network 999. The electronic device 901 may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

Electronic devices according to various embodiments disclosed in this document may be of various types. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to embodiments of this document are not limited to the above-described devices.

The various embodiments of this document and the terms used herein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various changes, equivalents, or replacements of the embodiments. In connection with the description of the drawings, similar reference numbers may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the above items, unless the relevant context clearly indicates otherwise. As used herein, “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “A Each of phrases such as “at least one of , B, or C” may include any one of the items listed together in the corresponding phrase, or any possible combination thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish one component from another, and to refer to that component in other respects (e.g., importance or order) is not limited. One (e.g., first) component is said to be “coupled” or “connected” to another (e.g., second) component, with or without the terms “functionally” or “communicatively.” When mentioned, it means that any of the components can be connected to the other components directly (e.g. wired), wirelessly, or through a third component.

The term “module” used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as logic, logic block, component, or circuit, for example. It can be used as A module may be an integrated part or a minimum unit of the parts or a part thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document are one or more instructions stored in a storage medium (e.g., built-in memory 936 or external memory 938) that can be read by a machine (e.g., electronic device 901). It may be implemented as software (e.g., program 940) including these. For example, a processor (e.g., processor 920) of a device (e.g., electronic device 901) may call at least one command among one or more commands stored from a storage medium and execute it. This allows the device to be operated to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves), and this term refers to cases where data is semi-permanently stored in the storage medium. There is no distinction between temporary storage cases.

According to one embodiment, methods according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or through an application store (e.g. Play Store ^™ ) or on two user devices (e.g. It can be distributed (e.g. downloaded or uploaded) directly between smart phones) or online. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

According to various embodiments, each component (e.g., module or program) of the above-described components may include a single or plural entity, and some of the plurality of entities may be separately placed in other components. there is. According to various embodiments, one or more of the components or operations described above may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components (eg, modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each component of the plurality of components in the same or similar manner as those performed by the corresponding component of the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, or omitted. Alternatively, one or more other operations may be added.

An electronic device according to an embodiment of the present disclosure may include a microphone, a speaker, a memory that stores at least one instruction, and a processor. For example, the instructions, when executed by the processor, cause the electronic device to obtain an audio input including voice data and noise using the microphone, and the voice data is a designated call word ( wake-up word), extract the voice data and the noise from the audio input, calculate the separation distance between the user and the electronic device based on the voice data, and calculate the extracted voice Identify a first volume of data, identify a second volume of noise extracted, and determine signal-to-noise (SNR) based on the identified first volume, the second volume, and the separation distance. ratio), calculate a third volume corresponding to the calculated SNR, and if the difference between the preset volume and the third volume exceeds a specified value, change the preset volume to the third volume, , It can be set to output a response to the voice data at the third volume using the speaker.

According to one embodiment, the instructions, when executed by the processor, may be configured so that the electronic device extracts the voice data and the noise, respectively, from the audio input using an artificial intelligence model. there is.

According to one embodiment, when the instructions are executed by the processor, the electronic device transmits the voice data using the speaker when the difference between the preset volume and the third volume is less than or equal to the specified value. The response may be set to be output at the preset volume.

According to one embodiment, the instructions, when executed by the processor, may be set to calculate the third volume so that the electronic device is proportional to the first volume, the second volume, and the third volume. there is.

According to one embodiment, the instructions, when executed by the processor, may be set so that the electronic device calculates the SNR based on the amplitude, frequency, waveform, or a combination thereof of the voice data.

According to one embodiment, the instructions, when executed by the processor, set the electronic device to calculate the separation distance using a first frequency band signal and a second frequency band signal included in the voice data. It can be.

According to one embodiment, the first frequency band signal may include a signal of a higher band than the second frequency band signal.

According to one embodiment, the electronic device may further include at least one sensor. For example, the instructions, when executed by the processor, cause the electronic device to identify information about the surrounding environment of the electronic device using the at least one sensor, and based on the identified information, It can be set to correct the calculated separation distance.

According to one embodiment, the at least one sensor may include a temperature sensor and a humidity sensor. For example, the information may be information about the ambient temperature and humidity of the electronic device.

According to one embodiment, the instructions, when executed by the processor, cause the electronic device to further identify acoustic characteristics of the voice data and determine a response to the voice data corresponding to the acoustic characteristics. It can be set to output by adding characteristics.

A method of processing voice data by an electronic device according to an embodiment of the present disclosure includes obtaining audio input including voice data and noise using a microphone, and using a designated call word (wake word) for the voice data. -up word), an operation of extracting the voice data and the noise from the audio input, an operation of calculating the separation distance between the user and the electronic device based on the voice data, and the extracted An operation of identifying a first volume of voice data, an operation of identifying a second volume of the extracted noise, a signal-to-noise ratio (SNR) based on the identified first volume, the second volume, and the separation distance. An operation of calculating a to-noise ratio), an operation of calculating a third volume corresponding to the calculated SNR, if the difference between a preset volume and the third volume exceeds a specified value, the preset volume is converted to the third volume. It may include an operation of changing the volume to 3, and an operation of outputting a response to the voice data to the third volume using a speaker.

According to one embodiment, the operation of extracting the voice data and the noise from the audio input, respectively, includes extracting the voice data and the noise from the audio input using an artificial intelligence model. It can be included.

According to one embodiment, the method of processing the voice data by the electronic device includes, when the difference between the preset volume and the third volume is less than or equal to the specified value, using the speaker, The operation of outputting a response at the preset volume may further be included.

According to one embodiment, calculating the third volume may include calculating the third volume such that the first volume, the second volume, and the third volume are proportional to the third volume.

According to one embodiment, the operation of calculating the SNR may include calculating the SNR based on the amplitude, frequency, waveform, or a combination thereof of the voice data.

According to one embodiment, calculating the separation distance may further include calculating the separation distance using a first frequency band signal and a second frequency band signal included in the voice data.

According to one embodiment, the method of processing the voice data by the electronic device includes identifying information about the surrounding environment of the electronic device using at least one sensor and based on the identified information, An operation of correcting the calculated separation distance may be further included.

According to one embodiment, the at least one sensor includes a temperature sensor and a humidity sensor, and the information may include information about the ambient temperature and humidity of the electronic device.

According to one embodiment, the method of processing the voice data by the electronic device includes further identifying an acoustic characteristic of the voice data and a characteristic determined corresponding to the acoustic characteristic in response to the voice data. An output operation can be further included by adding .

Claims (20)

In electronic devices,
mike;
speaker;
a memory storing at least one instruction; and
processor; Including,
The instructions, when executed by the processor, cause the electronic device to:
Using the microphone, obtain audio input including voice data and noise,
When the voice data is identified as containing a designated wake-up word, extracting the voice data and the noise respectively from the audio input,
Calculate the separation distance between the user and the electronic device based on the voice data,
Identifying a first volume of the extracted voice data,
Identifying a second volume of the extracted noise,
Calculating a signal-to-noise ratio (SNR) based on the identified first volume, the second volume, and the separation distance,
Calculating a third volume corresponding to the calculated SNR,
If the difference between the preset volume and the third volume exceeds a specified value, change the preset volume to the third volume,
Set to output a response to the voice data at the third volume using the speaker,
Electronic devices. In claim 1,
The instructions, when executed by the processor, cause the electronic device to:
Set to extract the voice data and the noise from the audio input, respectively, using an artificial intelligence model,
Electronic devices. In claim 1,
The instructions, when executed by the processor, cause the electronic device to:
When the difference between the preset volume and the third volume is less than or equal to the specified value, a response to the voice data is set to be output at the preset volume using the speaker,
Electronic devices. In claim 1,
The instructions, when executed by the processor, cause the electronic device to:
Set to calculate the third volume such that the first volume, the second volume, and the third volume are proportional,
Electronic devices. In claim 1,
The instructions, when executed by the processor, cause the electronic device to:
configured to calculate the SNR based further on the amplitude, frequency, waveform, or combination of the voice data,
Electronic devices. In claim 1,
The instructions, when executed by the processor, cause the electronic device to:
Set to calculate the separation distance using a first frequency band signal and a second frequency band signal included in the voice data,
Electronic devices. In claim 6,
The first frequency band signal includes a signal of a higher band than the second frequency band signal,
Electronic devices. In claim 1,
Contains at least one more sensor,
The instructions, when executed by the processor, cause the electronic device to:
Using the at least one sensor, identify information about the surrounding environment of the electronic device,
Based on the identified information, set to correct the calculated separation distance,
Electronic devices. In claim 8,
The at least one sensor includes a temperature sensor and a humidity sensor,
The information is information about the surrounding temperature and humidity of the electronic device,
Electronic devices. In claim 1,
The instructions, when executed by the processor, cause the electronic device to:
further identify acoustic characteristics of the voice data,
Set to output a response to the voice data by adding a characteristic determined in response to the sound characteristic,
Electronic devices. In a method for an electronic device to process voice data,
Obtaining audio input including voice data and noise using a microphone;
extracting the voice data and the noise from the audio input, respectively, when the voice data is identified as including a designated wake-up word;
calculating a separation distance between a user and the electronic device based on the voice data;
Identifying a first volume of the extracted voice data;
identifying a second volume of the extracted noise;
calculating a signal-to-noise ratio (SNR) based on the identified first volume, the second volume, and the separation distance;
An operation of calculating a third volume corresponding to the calculated SNR;
If the difference between the preset volume and the third volume exceeds a specified value, changing the preset volume to the third volume; and
Outputting a response to the voice data to the third volume using a speaker; Including,
method. In claim 11,
The operation of extracting the voice data and the noise from the audio input, respectively,
extracting the voice data and the noise from the audio input using an artificial intelligence model; Including,
method. In claim 11,
The method for the electronic device to process the voice data includes:
If the difference between the preset volume and the third volume is less than or equal to the specified value, outputting a response to the voice data at the preset volume using the speaker; Containing more,
method. In claim 11,
The operation of calculating the third volume is,
calculating the third volume so that the first volume, the second volume, and the third volume are proportional to each other; Including,
method. In claim 11,
The operation of calculating the SNR is:
calculating the SNR based further on the amplitude, frequency, waveform, or combination of the voice data; Including,
method. In claim 11,
The operation of calculating the separation distance is,
calculating the separation distance using a first frequency band signal and a second frequency band signal included in the voice data; Containing more,
method. In claim 16,
The first frequency band signal includes a signal of a higher band than the second frequency band signal,
method. In claim 11,
The method for the electronic device to process the voice data includes:
Identifying information about the surrounding environment of the electronic device using at least one sensor; and
An operation of correcting the calculated separation distance based on the identified information; Containing more,
method. In claim 18,
The at least one sensor includes a temperature sensor and a humidity sensor,
The information is information about the surrounding temperature and humidity of the electronic device,
method. In claim 11,
The method for the electronic device to process the voice data includes:
further identifying acoustic characteristics of the voice data; and
an operation of adding and outputting characteristics determined in response to the sound characteristics in response to the voice data; Containing more,
method.

Images