KR20190119521A - Electronic apparatus and operation method thereof - Google Patents

Abstract

An electronic apparatus and an operation method thereof are provided. The operation method of the electronic apparatus may include the following steps of: receiving a first voice signal through a first channel and a second voice signal through a second channel different from the first channel; acquiring first voice data from the first voice signal and the second voice data from the second voice signal; learning, based on the second voice data, a background speaker model acquired based on voice signals for a plurality of speakers; determining whether the first voice data is voice data corresponding to a first registered speaker registered in the electronic apparatus; and if the first voice data is the voice data corresponding to the first registered speaker, learning, based on the first voice data and the background speaker model, a first registered speaker model for recognizing the voice of the first registered speaker.

Description

ELECTRICAL APPARATUS AND OPERATION METHOD THEREOF

The present disclosure relates to an electronic device and a method of operating the same, and more particularly, to an electronic device for acquiring voice data for learning a voice recognition model and a method of operating the same.

Mobile intelligent terminal speech recognition falls into two categories: semantic recognition and speaker recognition. Speaker recognition is generally referred to as voiceprint recognition, and can be classified into two types, a text-dependent method and a text-independent method.

Sentence dependent speech recognition requires the user to read a fixed sentence 2-3 times in order to record the relevant function information that the user wants to register. In speech recognition, the user must read the same fixed sentence.

On the other hand, sentence independent speech recognition does not require the user to read along with a fixed sentence. As a user inputs a large amount of voice data for machine learning, the feature information of the user can be highly refined through a large amount of data learning. The user does not need to read the same fixed sentence during speech prediction.

Since the mobile intelligent terminal according to the prior art cannot distinguish user identities and different voice characteristics of different users, the same mobile intelligent terminal may provide the same voice command service to different users, thereby providing user privacy. There is a problem of poor security.

Taking voice assistant as an example, existing mobile intelligent terminals require a fixed wake up process when starting a voice assistant service. This process is a flaw in sentence dependent speech recognition, in which the user has to read a fixed sentence to wake up the voice assistant, and the voice assistant cannot respond quickly to all the voice commands of the user.

That is, all voice commands can be used after completion of the wake up process of the voice assistant. When a registered user wakes up the voice assistant using a fixed sentence, an unregistered user may give a voice command. As described above, the method of distinguishing a speaker from the existing voice assistant is limited to distinguishing the speaker through the use of a fixed sentence based on sentence dependent speech recognition.

To obtain highly refined user characteristic information and model parameters through learning through a complete learning model and a large amount of speech data input, sentence independent speech recognition may use machine learning techniques.

Based on the learned speaker model, the user can achieve high accuracy speaker speech recognition through any speech input regardless of the fixed sentence.

However, in order to implement sentence-independent speech recognition in a mobile intelligent terminal, both registered and unregistered voice data are required. The user must invest a lot of time and effort to input voice data, and obtaining voice data from an unregistered user is also a difficult problem for the user.

Since it is difficult to obtain high recognition accuracy without sufficient training data, conventional mobile intelligent terminals generally do not use sentence independent speech recognition.

An object of the present disclosure is to provide an electronic device and a method of operating the same, which can reduce the burden on input and improve the user experience by using voice data acquired through a user's voice call for learning. .

The objects of the present disclosure are not limited to the above-mentioned objects, and other objects and advantages of the present disclosure which are not mentioned above may be understood by the following description, and will be more clearly understood by the embodiments of the present disclosure. Also, it will be readily appreciated that the objects and advantages of the present disclosure may be realized by the means and combinations thereof indicated in the claims.

As a technical means for achieving the above technical problem, a first aspect of the present disclosure, receiving a first voice signal through a first channel, and receives a second voice signal through a second channel different from the first channel To obtain first voice data from the first voice signal, and to obtain second voice data from the second voice signal, based on the second voice data, to a voice signal for a plurality of speakers. Learning a background speaker model obtained based on the information; determining whether the first voice data is voice data corresponding to a first registered speaker registered in the electronic device; and registering the first voice data with the first registration. In the case of voice data corresponding to a speaker, a first registered speaker model for recognizing a voice of the first registered speaker is studied based on the first voice data and the background speaker model. A method of operating an electronic device may be provided, including an operation of moistening.

Further, a second aspect of the present disclosure provides a first voice signal through a first channel, a second voice signal through a second channel different from the first channel, and a first voice signal from the first voice signal. Acquire voice data, obtain second voice data from the second voice signal, learn a background speaker model acquired based on voice signals for a plurality of speakers, based on the second voice data, and It is determined whether first voice data is voice data corresponding to a first registered speaker registered in the electronic device, and when the first voice data is voice data corresponding to the first registered speaker, the first voice data. And at least one processor for learning a first registered speaker model for recognizing a voice of the first registered speaker and the first voice data and the second sound based on the background speaker model. Comprising a memory for storing data, it is possible to provide an electronic device.

In addition, a third aspect of the present disclosure may provide a computer program product comprising a recording medium having stored thereon a program for performing the method of the first aspect.

According to the electronic device and the operation method of the present disclosure, by using the voice data acquired through the user's voice call for learning, there is an effect that can reduce the burden on the input of the user's voice data and improve the user experience.

1 is a diagram illustrating an example of a method of generating a registered speaker model, according to some embodiments.
2 is a flowchart of a method of operating an electronic device, according to some embodiments.
3 is a flowchart illustrating a method of learning a first registered speaker model of an electronic device according to an exemplary embodiment.
4 is a diagram schematically illustrating a method of learning, registering, and recognizing an electronic device, according to an exemplary embodiment.
5A is a diagram illustrating a distribution of a GMM and a speech feature distribution of a speaker to be learned, according to an exemplary embodiment.
5B is a diagram illustrating a distribution of a GMM model after MAP adaptation according to some embodiments.
6 is a flowchart illustrating a method of learning a second registered speaker model of an electronic device according to an exemplary embodiment.
7 is a flowchart illustrating a process of performing a voice command of an electronic device according to some embodiments.
8 is a flowchart illustrating a process of performing a voice command of an electronic device according to some embodiments.
9 is a diagram illustrating a voice processing method of a host electronic device according to some embodiments.
10 is a block diagram schematically illustrating a configuration of an electronic device according to some embodiments.
11 is a configuration diagram schematically illustrating a configuration of a server according to some embodiments.

The above objects, features, and advantages will be described in detail with reference to the accompanying drawings, whereby those skilled in the art to which the present invention pertains may easily implement the technical idea of the present invention. In describing the present invention, when it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description will be omitted. Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used to indicate the same or similar components.

1 is a diagram illustrating an example of a method of generating a registered speaker model, according to some embodiments.

Referring to FIG. 1, an electronic device 1000 according to an exemplary embodiment may receive a voice signal from a user 10.

The electronic device 1000 may be a terminal device that may be used by the user 10. The electronic device 1000 is, for example, a mobile phone or a smart phone. It may be a device that can directly receive a voice signal of the user 10, such as smart TV, smart audio, smart speaker, artificial intelligence speaker, PC (Personal Computer), notebook computer and tablet PC, but is not necessarily limited thereto. .

The electronic apparatus 1000 may determine whether the user 10 is a registered speaker or a non-registered speaker based on the input voice signal 16 received from the user 10.

For example, when the user 10 determines that the user 10 is a non-registered speaker (161), the electronic device 1000 uses a voice feature obtained from the input voice signal 16 to correspond to a background speaker model corresponding to the plurality of speakers. Can be learned 18.

In the case of a method of determining a speaker, a method using a Gaussian mixture model (GMM) is widely known. A speaker recognition system using Gaussian mixture model requires a large amount of speech for learning, and builds a background speaker model using speech for a plurality of speakers in advance, and then adapts the user using a small amount of speech. The GMM-UBM method of performing learning is used.

When the user 10 determines that the user 10 is a non-registered speaker (161), the electronic apparatus 1000 trains the background speaker model corresponding to the plurality of speakers based on the input voice signal 16, and learns the learned background speaker model. After that, it can be used for adaptive learning using a small amount of speech of the user 10. Through adaptive learning, a registered speaker model for user 10 may be generated and stored (19).

For example, when the user 10 is determined to be a registered speaker (163), the electronic apparatus 1000 may use the input voice signal 16 received from the user 10 and the previously obtained background speaker model. The speech model for (10) can be trained (18).

Meanwhile, the electronic device 1000 according to some embodiments may receive a voice signal from an external device. The received voice signal 14 received from the external device may be, for example, a voice signal generated based on the voice of the call of the counterpart 12 with whom the user 10 communicates through the electronic device 1000. It is not limited.

The electronic apparatus 1000 may learn 18 a background speaker model corresponding to the plurality of speakers based on the received voice signal 14 received from the external device. That is, unlike the input voice signal 16, the electronic apparatus 1000 does not determine whether the received voice signal 14 received from an external device is a voice signal input by a registered speaker, and corresponds to a plurality of speakers. Background can be used to train the speaker model.

As such, the electronic apparatus 1000 according to an exemplary embodiment may apply a different voice processing process to a voice signal input to the electronic apparatus 1000 through different channels, thereby allowing the user 10 and the user 10 to perform a different process. Voices such as call parties and the like can be effectively classified, and voice model training can be performed based on the classified voices.

2 is a flowchart of a method of operating an electronic device, according to some embodiments.

In operation S201, the electronic apparatus 1000 may receive a first voice signal through a first channel. In operation S202, the electronic apparatus 1000 may receive a second voice signal through a second channel different from the first channel.

In some embodiments, the user 10 may perform a voice call with the counterpart 12 through the electronic device 1000. Here, the voice call may include a video call such as VoIP and VoLTE as well as a call that only exchanges voice. For example, when a user initiates a voice call through the electronic device 1000, the operation method of FIG. 2 may be triggered.

In operation S202, the electronic apparatus 1000 may obtain first voice data from the first voice signal and may obtain second voice data from the second voice signal.

The voice of the user, that is, the voice waveform, may be input to the electronic device 1000 as an input voice signal under the assumption that all the information generated by the user's voice mechanism is included. The electronic apparatus 1000 may use only some of the information included in the voice signal corresponding to the purpose of speech recognition. Some information meeting the recognition purpose may be extracted from the speech signal in a statistical manner.

As such, the information extracted from the speech signal for use in the speech recognition system may be referred to as a speech feature. The speech feature may be extracted from the speech signal, for example, to include a plurality of components in which the spectrum distribution on the frequency is different from each other.

In the voice feature extraction process, the electronic apparatus 1000 may extract feature vectors as information that may eliminate unnecessary redundant voice information, increase consistency among the same voice signals, and increase discrimination from other voice signals. have.

Such feature vectors may include, for example, linear predictive coefficients, cepstrum, mel frequency cepstral coefficients (MFCC), energy according to frequency bands (Filter Bank Energy), and the like. It may be extracted from the voice signal through the method of, but is not necessarily limited thereto.

The voice feature extracted by the electronic apparatus 1000 may be stored, for example, in the form of voice data, and may be used for generating, registering, and recognizing a speaker.

In operation S202, the electronic apparatus 1000 may determine whether, for example, the first voice data and the second voice data meet a preset speaker model learning condition.

 The speaker model training condition may include, for example, a condition regarding whether each voice data is generated by a user's speech mechanism. That is, the electronic apparatus 1000 may determine whether the first voice data and the second voice data include non-mute features, for example.

When the first voice data and the second voice data do not include non-mute features, that is, the electronic apparatus 1000 determines that no voice by the speaker is recorded in the first voice data and the second voice data. In this case, it may be determined that the first voice data and the second voice data do not satisfy the speaker model training condition.

As a result of the determination, when the first voice data and the second voice data meet the preset speaker model learning condition, the voice data obtaining apparatus may perform step S203.

In some embodiments, voice cleaning may be further performed on the first voice data and the second voice data after step S202 and before step S203. Since voice cleaning may include denoising, noise reduction, and the like, the quality of the first voice data and the second voice data may be improved, and thus the speaker model learning effect may also be improved.

Meanwhile, when the electronic device 1000 according to some embodiments corresponds to a mobile intelligent terminal, step S202 is storing voice data, and a hardware device operation layer may be set in an operating system of the electronic device 1000. .

When a user starts a voice call through an application of the electronic device 1000, the user may input and store the input voice data into the hardware device operation layer of the system in real time. In this case, the input voice signal to the microphone of the electronic device 1000 corresponds to voice data of the mobile intelligent terminal, that is, the user of the electronic device 1000. The received voice signal to the communication unit of the electronic apparatus 1000 corresponds to voice data of the voice call counterpart.

In operation S203, the electronic apparatus 1000 may learn the background speaker model obtained based on the voice signals for the plurality of speakers based on the second voice data.

For example, the electronic apparatus 1000 may classify the second voice data used for learning the background speaker model as data for background speaker model training and store the same in the memory of the electronic apparatus 1000.

The background speaker model is a Gaussian mixture model that is generated by collecting the speech signals of as many speakers as possible and then applying an expectation maximization algorithm (hereinafter referred to as an EM algorithm) to speech features corresponding to the collected speech signals. to be.

In operation S204, the electronic device 1000 may determine whether the first voice data is voice data corresponding to the first registered speaker registered in the electronic device 1000.

In operation S205, when the first voice data is voice data corresponding to the first registered speaker, the electronic apparatus 1000 may recognize the voice of the first registered speaker based on the first voice data and the background speaker model. You can train the Speaker Model.

For example, the electronic device 1000 may classify the first voice data used for learning the first registered speaker model as data for learning the first registered speaker model and store the same in the memory of the electronic device 1000.

For example, the electronic apparatus 1000 may extract a feature vector from the first voice data to train the first registered speaker model, and may calculate a maximum posterior probability based on the extracted feature vector and the background speaker model. MAP) adaptive algorithm can be used. A detailed method of learning the registered speaker model based on the voice data and the background speaker model will be described in detail later with reference to FIGS. 4, 5A, and 5B.

The speech feature extraction technique of neural network speech recognition model requires inputting a large amount of personal speech in advance to obtain the speaker's speech feature. Existing methods allow a user to input learning voice data by a specific program, and the user needs additional time for performing personal voice function learning.

In some embodiments of the present disclosure, the user recognizes the voice recognition model by collecting the instant voice data generated while working and living on a daily basis through the electronic device 1000 and using the collected voice data for learning the voice recognition model. Accuracy can be improved continuously.

Compared with the prior art, since the user of the learning method of the present disclosure does not need to perform a tedious and complicated learning voice data input operation, there is an advantage of reducing the burden on the user and improving the user experience.

At the same time, the learning method of the present disclosure allows the accumulation of sentence independent speech recognition to be implemented on a mobile intelligent terminal, so that the mobile intelligent terminal more intelligently understands the characteristics of each user, beyond the limitations of existing sentence dependent speech recognition. can do.

In the case of the voice assistant, the sentence-independent speech recognition function can be added to enhance security by identifying a user's identity and processing only a voice command to which the voice recognition model is applied.

Meanwhile, in order to avoid waste of resources and leakage of user privacy, the electronic apparatus 1000 may remove the first voice data and the second voice data immediately after the learning is completed, and the method of FIG. 1 may end. . Alternatively, when the training of the speech recognition model is terminated, the associated speech data may be deleted, and the method of FIG. 2 may be terminated.

3 is a flowchart illustrating a method of learning a first registered speaker model of an electronic device according to an exemplary embodiment.

Referring to FIG. 3, the electronic apparatus 1000 according to an exemplary embodiment receives the first voice signal 301 from the user 10 and the first voice data 303 from the received first voice signal 301. Can be obtained.

Meanwhile, the electronic apparatus 1000 receives a second voice signal 302 based on the call voice of the counterpart 12 from the external device, and obtains second voice data 304 from the received second voice signal 302. can do.

The electronic apparatus 1000 may learn the background speaker model 306 obtained based on the voice signals for the plurality of speakers using the acquired second voice data 304.

The electronic device 1000 may determine whether the acquired first voice data 303 is voice data corresponding to the first registered speaker registered in the electronic device 1000 (305).

For example, the electronic apparatus 1000 may receive separate authentication data from a user and determine whether the first voice data 303 is voice data corresponding to the first registered speaker registered in the electronic apparatus 1000. Decision 305 may be made.

The electronic apparatus 1000 may, for example, based on at least one of a face recognition based on an image photographed through a camera and a fingerprint recognition using a fingerprint recognition sensor, a voice in which the first voice data corresponds to the first registered speaker. It may be determined whether the data is 305.

In relation to the determination based on face recognition, the electronic apparatus 1000 may obtain and store a face image photographed in advance for the first registered speaker.

For example, the user inputs his or her face image through a camera included in the electronic apparatus 1000 simultaneously with the input of the first voice signal, whereby the user inputs the first voice signal corresponding to the first voice data. You can authenticate that you are a user.

As another example, the user inputs a first voice signal to the electronic device 1000 and then responds to a request for inputting a face image displayed on a screen of the electronic device 1000 through a camera included in the electronic device 1000. By inputting a face image of one's own face, the user may authenticate that the user inputs a first voice signal corresponding to the first voice data.

In relation to the determination based on fingerprint recognition, the electronic apparatus 1000 may store fingerprint data acquired for the first registered speaker in advance.

For example, the user inputs his or her fingerprint data through the fingerprint recognition sensor included in the electronic apparatus 1000 simultaneously with the input of the first voice signal, so that the user receives the first voice signal corresponding to the first voice data. The user can be authenticated.

As another example, the user inputs a first voice signal to the electronic apparatus 1000 and then uses the fingerprint recognition sensor included in the electronic apparatus 1000 in response to a fingerprint data request displayed on the screen of the electronic apparatus 1000. By inputting a fingerprint image of the user through the user, the user may authenticate that the user inputs the first voice signal corresponding to the first voice data.

As a result of the determination 305, when the first voice data 303 is voice data corresponding to the first registered speaker registered in the electronic device 1000, the electronic device 1000 may determine the first voice data 303 and the background speaker. The model 306 can be used to train the first registered speaker model 307.

In contrast, when the determination 305 determines that the first voice data 303 is not voice data corresponding to the first registered speaker registered in the electronic device 1000, the electronic device 1000 may transmit the second voice data 304. As in the case of), the first voice data 303 may be used to train the background speaker model 306 obtained based on the voice signals for the plurality of speakers.

As another example, the electronic apparatus 1000 may perform voice recognition based on a preset first registered speaker model to determine whether the first voice data is voice data corresponding to the first registered speaker (305). That is, the electronic apparatus 1000 may use the voice recognition model itself for user authentication without performing user authentication by face recognition or fingerprint recognition, and such authentication may cause a determination error in consideration of initial learning of the voice recognition model. As such, manual identity authentication by the user may be accompanied by assistance.

For example, the electronic apparatus 1000 may receive a speaker registration voice signal generated through utterance of a preset sentence to perform voice recognition based on a preset first registered speaker model, and receive the received speaker registration voice signal. Register the first registered speaker model in advance. In this case, when the determination 305 indicates that the first voice data 303 is voice data corresponding to the first registered speaker registered in the electronic apparatus 1000, the electronic apparatus 1000 may determine the first voice data 303 and The background speaker model 306 may be used to update the preset first registered speaker model to obtain a new first registered speaker model 307.

For example, the electronic apparatus 1000 may perform user authentication through at least one of a user authentication method using the above-described face recognition or fingerprint recognition and a user authentication method using the user voice recognition model itself.

For example, the electronic apparatus 1000 may perform user authentication by simultaneously or sequentially using the above-described user authentication method using face recognition or fingerprint recognition and a user authentication method using the user voice recognition model itself. However, the present invention is not limited thereto, and any authentication scheme capable of determining whether the first voice data corresponds to voice data corresponding to the first registered speaker may be used.

As such, the electronic apparatus 1000 according to an exemplary embodiment may use the acquired voice data for learning a background speaker model even when the voice data acquired through the first channel is determined to be voice data of a user rather than a registered speaker. In addition, it has the advantage of collecting voice data for background speaker model training more efficiently.

4 is a diagram schematically illustrating a method of learning, registering, and recognizing an electronic device, according to an exemplary embodiment.

Referring to FIG. 4, in the learning step, the registration step, and the recognition step, the electronic apparatus 1000 may extract voice features 402, 432, and 452 for the input or received voice signals 401, 431, and 451. have.

In the learning step, the electronic apparatus 1000 applies the expectation maximization algorithm (EM algorithm) 403 to the speech feature 402 extracted from the input speech signal 401 to the background speaker model 404. Can be generated.

Meanwhile, in the registration step, the electronic apparatus 1000 presets a maximum a posteriori adaptation for the voice feature 432 extracted from the input voice signal 431 and the background speaker model 404 generated in the learning step. algorithm, MAP adaptation algorithm) may be applied to generate a GMM, ie, a speaker model 434, adapted to a spoken user.

In the recognition step, the electronic apparatus 1000 calculates a similarity score 453 using the speech feature 452 and the speaker model 434 extracted from the input speech signal 451 and based on the calculated score, Recognition of the spoken user may be performed (454).

5A is a diagram illustrating a distribution of a GMM and a speech feature distribution of a speaker to be learned, according to an exemplary embodiment.

Referring to FIG. 5A, there is shown a background speaker model, that is, a distribution 501 of a Gaussian mixture model obtained by applying an EM algorithm to a plurality of speakers, and a distribution 511 of a voice feature of a user to be trained. .

5B is a diagram illustrating a distribution of a GMM model after MAP adaptation according to some embodiments.

Referring to FIG. 5B, with respect to the generated background speaker model, the electronic apparatus 1000 may shift a parameter of the background speaker model so that a posteriori probability is maximized with respect to voice data of a user to be trained using a MAP adaptive algorithm. Can be shifted. As a result of the shifting, a distribution 502 of the Gaussian mixture model for the user to be learned may be generated, and such a generation method of the Gaussian mixture model may be referred to as a GMM-UBM method.

In the GMM-UBM method, when Gaussian mixture models for multiple users are obtained, the parameters of each Gaussian mixture model can be obtained by shifting the same background speaker model. It has the advantage of being easy.

6 is a flowchart illustrating a method of learning a second registered speaker model of an electronic device according to an exemplary embodiment.

Referring to FIG. 6, the electronic apparatus 1000 according to an exemplary embodiment receives a first voice signal 601 from a user 10, and receives first voice data 603 from the received first voice signal 601. Can be obtained.

Meanwhile, the electronic apparatus 1000 receives a second voice signal 602 based on the call voice of the counterpart 12 from the external device, and obtains second voice data 604 from the received second voice signal 602. can do.

The electronic apparatus 1000 may learn the background speaker model 607 obtained based on the voice signals for the plurality of speakers using the acquired second voice data 604.

The electronic device 1000 may determine whether the acquired first voice data 603 is voice data corresponding to the first registered speaker registered in the electronic device 1000 (605).

For example, the electronic apparatus 1000 may receive separate authentication data from a user and determine whether the first voice data 603 is voice data corresponding to the first registered speaker registered in the electronic apparatus 1000. Decision 605 may be made.

As another example, the electronic apparatus 1000 may perform voice recognition based on a preset first registered speaker model to determine whether the first voice data is voice data corresponding to the first registered speaker (605).

As a result of the determination 605, when the first voice data 603 is voice data corresponding to the first registered speaker registered in the electronic device 1000, the electronic device 1000 may determine the first voice data 603 and the background speaker. The model 607 may be used to train the first registered speaker model 608.

On the contrary, when the determination 605 indicates that the first voice data 603 is not voice data corresponding to the first registered speaker registered in the electronic device 1000, the electronic device 1000 may transmit the second voice data 604. As in the case of), the first voice data 603 may be used to train the background speaker model 607 obtained based on the voice signals for the plurality of speakers.

The electronic device 1000 may determine whether the acquired first voice data 603 is voice data corresponding to the second registered speaker registered in the electronic device 1000 (606).

For example, the electronic apparatus 1000 may receive separate authentication data from a user and determine whether the first voice data 603 is voice data corresponding to the second registered speaker registered in the electronic apparatus 1000. Decision 606 may be made.

As another example, the electronic apparatus 1000 may perform voice recognition based on a preset second registered speaker model to determine whether the first voice data is voice data corresponding to the second registered speaker (606).

As a result of the determination 606, when the first voice data 603 is voice data corresponding to the first registered speaker registered in the electronic device 1000, the electronic device 1000 may determine the first voice data 603 and the background speaker. The model 607 may be used to train the second registered speaker model 609.

In contrast, when the determination 606 determines that the first voice data 603 is not voice data corresponding to the first registered speaker registered in the electronic apparatus 1000, the electronic apparatus 1000 may determine the second voice data 604. As in the case of), the first voice data 603 may be used to train the background speaker model 607 obtained based on the voice signals for the plurality of speakers.

That is, even when the electronic apparatus 1000 is owned by the first user or a second user who is not the first user rents the electronic apparatus 1000, the electronic apparatus 1000 may be input by the second user. The first voice data extracted from the first voice signal may be determined as voice data corresponding to the second registered speaker.

FIG. 6 illustrates whether or not the voice data corresponding to the first registered speaker for the first voice data 603 corresponds to the voice data corresponding to the second registered speaker for the first voice data 603; 606) Although the process is shown in parallel, the determination process 605, 606 may be performed simultaneously or sequentially, and those skilled in the art may apply equally to the method of FIG. 6 even when there are a plurality of registered speakers. It will be easy to understand.

As such, when it is determined that the voice data corresponding to any one of the plurality of registered speakers is the voice data acquired through the first channel, the electronic apparatus 1000 according to the exemplary embodiment may obtain the acquired voice data. By using the background speaker model for learning, the voice data for training the background speaker model can be collected more efficiently for a plurality of users of the electronic apparatus 1000.

In particular, the method according to some embodiments of FIG. 6 not only does not only the user of the electronic device 1000 be the owner of the electronic device 1000 but also voices of a plurality of users simultaneously or through the microphone of the electronic device 1000. Since the same applies to a multi-voice call that is sequentially input, the electronic apparatus 1000 according to an exemplary embodiment has an advantage of collecting voice data for background speaker model training more efficiently for a plurality of users.

7 is a flowchart illustrating a process of performing a voice command of an electronic device according to some embodiments.

Referring to FIG. 7, the electronic apparatus 1000 may use a registered speaker model for a plurality of users including a father 701, a son 703, and a mother 705. For example, father 701, son 703, and mother 705 may all input input voice signals corresponding to the same command phrase 72, such as “Call Mom” to electronic device 1000. have. The input voice signal may be processed by the electronic apparatus 1000 and / or the server 71 and used to perform an operation corresponding to a user authentication and a voice command.

At this time, the electronic apparatus 1000 detects a registered speaker model corresponding to each user with respect to an input voice signal corresponding to the same command phrase 72 input by each user, and each user included in the detected registered speaker model. By analyzing the information about, it is possible to perform a specific function appropriate for each user.

For example, the electronic apparatus 1000 may obtain voice data from an input voice signal input by the father 701, and compare the acquired voice data with a registered speaker model corresponding to the father 701. As a result of the comparison, when it is determined that the input voice signal is input by the father 701, the electronic apparatus 1000 determines that the authentication of the father 701 is successful, and from the registered speaker model corresponding to the father 701. Information about the father's mother 731 can be obtained. The electronic apparatus 1000 that obtains information about the father's mother 731 may perform a voice call request function for the father's mother 731 in response to the input voice signal of the father 701.

As another example, the electronic apparatus 1000 may obtain voice data from an input voice signal input by the son 703, and compare the acquired voice data with a registered speaker model corresponding to the son 703. As a result of the comparison, when it is determined that the input voice signal is input by the son 703, the electronic apparatus 1000 determines that authentication of the son 703 is successful, and from the registered speaker model corresponding to the son 703. Information may be obtained about the mother (705) (son). The electronic apparatus 1000 having obtained the information about the mother 705 may perform a voice call request function for the mother 705 in response to an input voice signal of the son 703.

As another example, the electronic apparatus 1000 may obtain voice data from an input voice signal input by the mother 705, and compare the acquired voice data with a registered speaker model corresponding to the mother 705. As a result of the comparison, when it is determined that the input voice signal is input by the mother 705, the electronic apparatus 1000 determines that authentication of the mother 705 is successful, and from the registered speaker model corresponding to the mother 705, Information about the mother 735 of the mother may be obtained. The electronic apparatus 1000 that obtains information about the mother 735 of the mother may perform a voice call request function for the mother 735 of the mother in response to the input voice signal of the mother 705.

8 is a flowchart illustrating a process of performing a voice command of an electronic device according to some embodiments.

Referring to FIG. 8, similar to FIG. 7, the electronic apparatus 1000 may use a registered speaker model for a plurality of users including a father 701, a son 703, and a mother 705. However, unlike FIG. 7, the electronic device 1000 corresponds to a smart TV according to some embodiments of FIG. 8, and the electronic device 1000 may receive a registered speaker model for a plurality of users from an external device.

That is, when the electronic device 1000 is not a device that uses voice signals of the user and the call counterpart, such as a smart TV, the electronic device 1000 may use external devices 711 and 713 that use voice signals of the user and the call counterpart. 715, corresponding to user authentication and voice command without receiving training on the direct registered speaker model by receiving registered speaker models for a plurality of users generated by the external devices 711, 713, and 715. Can be performed.

For example, the father 701, the son 703, and the mother 705 all input the input voice signal corresponding to the same command phrase 82, such as “Please turn on my favorite channel” to the electronic device 1000. Can be. The input voice signal may be processed by the electronic apparatus 1000 and / or the server 81 and used to perform an operation corresponding to a user authentication and a voice command.

At this time, the electronic apparatus 1000 detects a registered speaker model corresponding to each user with respect to an input voice signal corresponding to the same command phrase 82 input by each user, and each user included in the detected registered speaker model. By analyzing the information about, it is possible to perform a specific function appropriate for each user.

For example, the electronic apparatus 1000 may obtain voice data from an input voice signal input by the father 701, and compare the acquired voice data with a registered speaker model corresponding to the father 701. The registered speaker model corresponding to the father 701 may be, for example, a registered speaker model received from the first external device 711 owned by the father 701. As a result of the comparison, when it is determined that the input voice signal is input by the father 701, the electronic apparatus 1000 determines that the authentication of the father 701 is successful, and from the registered speaker model corresponding to the father 701. Information about the father's preferred channel 831 may be obtained. The electronic apparatus 1000 that has obtained the information on the father's preferred channel 831 corresponds to an input voice signal of the father 701 and changes the channel of the electronic apparatus 1000 to a sports channel which is the father's preferred channel 831. Can perform the function to switch.

As another example, the electronic apparatus 1000 may obtain voice data from an input voice signal input by the son 703, and compare the acquired voice data with a registered speaker model corresponding to the son 703. The registered speaker model corresponding to the son 703 may be, for example, a registered speaker model received from the second external device 713 owned by the son 703. As a result of the comparison, when it is determined that the input voice signal is input by the son 703, the electronic apparatus 1000 determines that authentication of the son 703 is successful, and from the registered speaker model corresponding to the son 703. Obtain information about. The electronic apparatus 1000 that obtains information about the son's preferred channel 833 may change the channel of the electronic apparatus 1000 into an animation channel which is the son's favorite channel 833 in response to the input voice signal of the son 703. Can perform the function to switch.

As another example, the electronic apparatus 1000 may obtain voice data from an input voice signal input by the mother 705, and compare the acquired voice data with a registered speaker model corresponding to the mother 705. The registered speaker model corresponding to the mother 705 may be, for example, a registered speaker model received from the third external device 715 owned by the mother 705. As a result of the comparison, when it is determined that the input voice signal is input by the mother 705, the electronic apparatus 1000 determines that authentication of the mother 705 is successful, and from the registered speaker model corresponding to the mother 705, Information about the mother 735 of the mother may be obtained. The electronic apparatus 1000 that obtains information about the mother 735 of the mother, changes the channel of the electronic apparatus 1000 to a drama channel, which is the mother's preferred channel 835, in response to the input voice signal of the mother 705. To perform the function.

9 is a diagram illustrating a voice processing method of a host electronic device according to some embodiments.

Referring to FIG. 9, a method of operating an electronic device according to an embodiment of the present disclosure may be performed by the host electronic device 901. The host electronic device 901 refers to an electronic device capable of generating a recording file and learning a speaker model corresponding to each of a plurality of speakers with respect to a voice signal inputted by a plurality of speakers during a conference.

Before proceeding with the conference, the conference attendants 971 may share 903 their registered speaker models with the host electronic device 901 and the conference attendees 972. The host electronic device 901 which has received the registered speaker models from the meeting attendants 971 may learn and update each registered speaker model in a GMM-UBM manner based on a voice input during the conference. have. Registered speaker models from which learning and updating have been performed may be re-shared between the host electronic device 901, the meeting attendants 971, and the meeting attendees 972.

Meanwhile, the host electronic device 901 may generate a recording file 905 for the voice signal input during the conference, and share the generated recording file with the non-participants 972. After the meeting, non-attendees 972 who received the generated recording file may also learn and update each registered speaker model in the GMM-UBM method based on the voice signal included in the received recording file. have.

 As such, the electronic device according to an embodiment of the present disclosure has an advantage of more efficiently learning the registered speaker model without limiting the time and place where the user's speech occurs by sharing the registered speaker models for the plurality of users.

10 is a block diagram schematically illustrating a configuration of an electronic device according to some embodiments.

Referring to FIG. 10, an electronic device 1000 according to some embodiments may include a processor 1001, a memory 1003, a microphone 1005, a communication unit 1007, and a camera 1009.

The electronic device 1000 may include, for example, at least one processor 1001.

The processor 1001 may typically control overall operations of the electronic apparatus 1000. For example, the processor 1001 may control the memory 1003, the microphone 1005, the communication unit 1007, and the camera 1009 by executing programs stored in the memory 1003.

The processor 1001 may receive a first voice signal through a first channel and receive a second voice signal through a second channel different from the first channel.

The processor 1001 may receive, for example, a first voice signal through the microphone 1005 and a second voice signal from an external device through the communication unit 1007. The processor 1001 may obtain first voice data from the first voice signal and obtain second voice data from the second voice signal.

The processor 1001 may train the background speaker model obtained based on the voice signals for the plurality of speakers based on the second voice data.

The processor 1001 may determine whether the first voice data is voice data corresponding to the first registered speaker registered in the electronic apparatus 1000.

When the first voice data is voice data corresponding to the first registered speaker, the processor 1001 may generate a first registered speaker model that recognizes the voice of the first registered speaker based on the first voice data and the background speaker model. I can learn.

The processor 1001 may extract, for example, a feature vector corresponding to the first speech data and train the first registered speaker model based on a maximum posterior probability adaptive algorithm using the extracted feature vector and the background speaker model. Can be.

In contrast, when the first voice data is not voice data corresponding to the first registered speaker, the processor 1001 may train the background speaker model based on the first voice data.

The processor 1001 may determine whether the first voice data corresponds to voice data corresponding to a second registered speaker registered in the electronic apparatus 1000.

For example, when the first voice data is voice data corresponding to the second registered speaker, the processor 1001 recognizes a voice of the second registered speaker based on the first voice data and the background speaker model. 2 You can train the registered speaker model.

In contrast, when the first voice data is not voice data corresponding to the second registered speaker, the processor 1001 may train the background speaker model based on the first voice data.

The memory 1003 may store a program for controlling the operation of the electronic apparatus 1000. The memory 1003 may include at least one instruction for controlling the operation of the electronic apparatus 1000. The memory 1003 may be, for example, a voice signal input by a user, a registered speaker model including information about a user feature vector corresponding to a specific user, and a voice signal for extracting voice data by the user's speech from the voice signal. An acoustic model, voice data extracted from a voice signal, a background speaker model, and the like may be stored. Programs stored in the memory 1003 may be classified into a plurality of modules according to their functions.

The memory 1003 may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg, SD or XD memory, etc.), RAM Random Access Memory (RAM) Static Random Access Memory (SRAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Read-Only Memory (PROM), Magnetic Memory, Magnetic Disk It may include at least one type of storage medium of the optical disk.

The microphone 1005 may receive an input voice signal from a user. The microphone 1005 may receive a user voice from the vicinity of the electronic device 1000 in real time while the electronic device 1000 is operating.

The communication unit 1007 may receive a voice signal from an external device. The communication unit 1007 may receive a voice signal generated by the call counterpart of the user in real time from the external device while the electronic device 1000 is operating.

The communication unit 1007 may include one or more communication modules for communicating with a server. For example, the communication unit 1007 may include a short range communication unit and a mobile communication unit. The short-range wireless communication unit includes a Bluetooth communication unit, a Bluetooth low energy (BLE) communication unit, a near field communication unit (Near Field Communication unit), a WLAN (Wi-Fi) communication unit, a Zigbee communication unit, an infrared ray (IrDA) It may include, but is not limited to, a Data Association (W Association) communication unit, a WFD (Wi-Fi Direct) communication unit, an ultra wideband (UWB) communication unit, an Ant + communication unit, and the like.

The camera 1009 may capture a face of the user of the electronic device 1000 to obtain a user face image. For example, the camera 1009 may acquire a user's face image in real time for authentication of the user while the electronic apparatus 1000 is operating.

11 is a configuration diagram schematically illustrating a configuration of a server according to some embodiments.

The user authentication method according to some embodiments of the present disclosure may be performed by the server 1100 connected to the electronic device 1000 and the electronic device 1000 through wired or wireless communication.

Referring to FIG. 11, a server 1100 according to some embodiments may include a processor 1101, a communication unit 1105, and a memory 1103. The server 1100 receives the registered speaker model, the acquired voice signal, the voice data, and the like from the electronic device 1000 to receive the user's authentication and / or the learning of the registered speaker model without directly receiving data necessary for user authentication from the user. Can be done.

The memory 1103 may store a program for controlling the operation of the server 1100. The memory 1103 may include at least one instruction for controlling the operation of the server 1100. The memory 1103 may be, for example, a voice signal input by a user, a registered speaker model including information about a user feature vector corresponding to a specific user, and a voice signal for extracting voice data of the user's speech from the voice signal. An acoustic model, voice data extracted from a voice signal, a background speaker model, and the like may be stored. Programs stored in the memory 1103 may be classified into a plurality of modules according to their functions.

The memory 1103 may include a plurality of DBs to integrate and manage registered speaker models for various users, data of various electronic devices, and the like.

The communication unit 1105 may include one or more communication modules for communicating with the electronic device 1000. For example, the communication unit 1105 may include a short range communication unit and a mobile communication unit. The short-range wireless communication unit includes a Bluetooth communication unit, a Bluetooth low energy (BLE) communication unit, a near field communication unit (Near Field Communication unit), a WLAN (Wi-Fi) communication unit, a Zigbee communication unit, an infrared ray (IrDA) It may include, but is not limited to, a Data Association (W Association) communication unit, a WFD (Wi-Fi Direct) communication unit, an ultra wideband (UWB) communication unit, an Ant + communication unit, and the like. The mobile communication unit transmits and receives a radio signal with at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signal may include various types of data according to transmission and reception of a voice call signal, a video call call signal, or a text / multimedia message.

The electronic device 1000 may include, for example, at least one processor 1101.

The processor 1101 may typically control overall operations of the server 1100. For example, the processor 1101 may control the memory 1103 and the communication unit 1105 by executing programs stored in the memory 1103.

The processor 1001 of the server 1100 may perform the same operation as that of the processor 1001 of the electronic device 1000 according to the exemplary embodiment of FIG. 10, and accordingly, the description of some exemplary embodiments may include the processor 1001. The description of the operation of the) will be omitted. However, those skilled in the art will appreciate that each step of the method of operating the electronic device of the present disclosure may be performed by the processor 1001 of the electronic device 1000 or the processor of the server 1100 for the same voice signal, voice data, background speaker model, and registered speaker model. It will be readily understood that it may be performed by at least one of 1001.

Some embodiments may also be embodied in the form of a recording medium containing instructions executable by a computer, such as program modules executed by the computer. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. In addition, the computer readable medium may include a computer storage medium. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

Further, in this specification, “unit” may be a hardware component such as a processor or a circuit, and / or a software component executed by a hardware component such as a processor.

The foregoing description of the disclosure is provided by way of example, and it will be understood by those skilled in the art that the present disclosure may be easily modified into other specific forms without changing the technical spirit or essential features of the present disclosure. will be. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present disclosure is indicated by the appended claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present disclosure. do.

Claims (20)

In the operating method of the electronic device,
Receiving a first voice signal through a first channel and receiving a second voice signal through a second channel different from the first channel;
Acquiring first voice data from the first voice signal and acquiring second voice data from the second voice signal;
Learning a background speaker model obtained based on voice signals for a plurality of speakers based on the second voice data;
Determining whether the first voice data is voice data corresponding to a first registered speaker registered in the electronic device; And
If the first voice data is voice data corresponding to the first registered speaker, learning a first registered speaker model for recognizing the voice of the first registered speaker based on the first voice data and the background speaker model. Causing; to include.
The method of claim 1,
Receiving a first voice signal through the first channel, and receiving a second voice signal through a second channel different from the first channel
Receiving the first voice signal through a microphone included in the electronic device; And
Receiving the second voice signal from an external device through a communication unit.
The method of claim 1,
The operation of determining whether the first voice data is voice data corresponding to the first registered speaker registered in the electronic device is performed.
Determining whether the first voice data is voice data corresponding to the first registered speaker based on at least one of face recognition based on an image photographed through a camera and fingerprint recognition using a fingerprint recognition sensor. How to.
The method of claim 1,
If the first voice data is not voice data corresponding to the first registered speaker, further comprising learning the background speaker model based on the first voice data.
The method of claim 1,
Receiving a speaker registration voice signal generated through speech of a preset sentence; And
And registering the first registered speaker with the electronic device in advance by using the speaker registered voice signal.
The method of claim 5,
The operation of determining whether the first voice data is voice data corresponding to the first registered speaker registered in the electronic device is performed.
And determining whether the first voice data is voice data corresponding to the first registered speaker based on voice recognition based on the first registered speaker model.
The method of claim 1,
Learning a first registered speaker model for recognizing the voice of the first registered speaker
Extracting a feature vector corresponding to the first voice data; And
Training the first registered speaker model based on a maximum posterior probability (MAP) adaptive algorithm using the feature vector and the background speaker model.
The method of claim 1,
Determining whether the first voice data is voice data corresponding to a second registered speaker registered in the electronic device; And
If the first voice data is voice data corresponding to the second registered speaker, learning a second registered speaker model for recognizing the voice of the second registered speaker based on the first voice data and the background speaker model. Further comprising;
The method of claim 1,
Receiving a third registered speaker model for recognizing a voice of the first registered speaker from an external device; And
And performing voice recognition of the first registered speaker using the first registered speaker model and the third registered speaker model.
In an electronic device,
Receive a first voice signal through a first channel, receive a second voice signal through a second channel different from the first channel, obtain first voice data from the first voice signal, and perform the second voice signal. Acquire second voice data from the signal, learn a background speaker model obtained based on the voice signal for a plurality of speakers based on the second voice data, and register the first voice data with the electronic device; It is determined whether the voice data corresponds to the first registered speaker, and when the first voice data is voice data corresponding to the first registered speaker, based on the first voice data and the background speaker model, At least one processor for learning a first registered speaker model that recognizes a voice of the first registered speaker; And
And a memory configured to store the first voice data and the second voice data.
The method of claim 10,
The at least one processor
And receiving the first voice signal through a microphone included in the electronic device, and receiving the second voice signal from an external device through a communication unit.
The method of claim 10,
The at least one processor
The electronic device determines whether the first voice data is voice data corresponding to the first registered speaker based on at least one of face recognition based on an image captured by a camera and fingerprint recognition using a fingerprint recognition sensor. .
The method of claim 10,
The at least one processor
And if the first voice data is not voice data corresponding to the first registered speaker, training the background speaker model based on the first voice data.
The method of claim 10,
The at least one processor
And receiving a speaker registration voice signal generated through speech of a preset sentence, and registering the first registered speaker using the speaker registration voice signal in advance.
The method of claim 14,
The at least one processor
And determining whether the first voice data is voice data corresponding to the first registered speaker based on voice recognition based on the first registered speaker model.
The method of claim 10,
The at least one processor
Extracting a feature vector corresponding to the first speech data and training the first registered speaker model based on a maximum posterior probability (MAP) adaptive algorithm using the feature vector and the background speaker model, Electronic devices.
The method of claim 10,
The at least one processor
It is determined whether the first voice data is voice data corresponding to a second registered speaker registered in the electronic device, and when the first voice data is voice data corresponding to the second registered speaker, the first voice data. And a second registered speaker model for recognizing a voice of the second registered speaker based on data and the background speaker model.
The method of claim 10,
The at least one processor
Receiving a third registered speaker model for recognizing a voice of the first registered speaker from an external device, and performing voice recognition of the first registered speaker using the first registered speaker model and the third registered speaker model , Electronic device.
Receiving a first voice signal through a first channel and receiving a second voice signal through a second channel different from the first channel;
Acquiring first voice data from the first voice signal and acquiring second voice data from the second voice signal;
Learning a background speaker model obtained based on voice signals for a plurality of speakers based on the second voice data;
Determining whether the first voice data is voice data corresponding to a first registered speaker registered in the electronic device; And
If the first voice data is voice data corresponding to the first registered speaker, learning a first registered speaker model for recognizing the voice of the first registered speaker based on the first voice data and the background speaker model. And a recording medium having a program stored thereon, the recording medium storing the program.
The method of claim 19,
Receiving a first voice signal through the first channel, and receiving a second voice signal through a second channel different from the first channel
Receiving, via a microphone, the first voice signal; And
And receiving the second voice signal through a communication unit.

Images