KR20210031344A - Computer programs for providing startup language recognition technology - Google Patents

Abstract

According to one embodiment of the present disclosure to solve the above-stated issue, a computer program stored in a computer-readable storage medium can be executed by at least one processor. The computer program enables the at least one processor to perform operations for providing startup language recognition technology. The operations can include: extracting at least one feature corresponding to voice data of a user received through a user terminal by using the voice data as input into a voice signal processing module; calculating a matching score of a predetermined keyword and the voice data by using the at least one feature as input into a keyword recognition model; and generating a control signal for the user terminal to call a voice-based user interface to determine whether to transmit the signal to the user terminal, based on the matching score. Therefore, the present invention is capable of providing startup language recognition technology which is powerful in a game environment.

Description

Computer program for providing startup word recognition technology {COMPUTER PROGRAMS FOR PROVIDING STARTUP LANGUAGE RECOGNITION TECHNOLOGY}

The present disclosure relates to a speech recognition technology, and more particularly, to a computer program for providing a startup word recognition technology in a game environment.

With the recent rapid development of IT technology, the game industry is growing together. As the game industry grows, game companies that provide game services are also explosively increasing. With the increase of game companies providing game services, competition in the game service market is intensifying. Accordingly, game companies are constantly making efforts to acquire the competitiveness of their respective games. In particular, game companies are constantly making a lot of efforts to provide convenience or interesting elements to maintain the continuity of the game to game users.

On the other hand, due to the expansion of the field of deep learning technology application, a breakthrough in keyword spottiong technology is being made. Accordingly, various products of keyword recognition technology (for example, Amazon echo, Apple homepod, Google home, etc.) are taking place in real life, providing convenience to users, giving high satisfaction.

Accordingly, the demand for a voice-based user interface is rapidly increasing in various environments in which a mobile game is played (eg, while driving, while cooking, or in a busy situation).

However, when a game user wants to play a game in a space with a lot of ambient noise, voice data for voice recognition may contain a lot of noise, which is a problem of lack of accuracy in recognition in providing a voice-based user interface. Can cause.

Accordingly, there may be a demand in the game industry for a computer program to provide convenience to game users through robust keyword recognition technology in a mobile game environment.

There may be a demand in the game industry for computer programs to provide convenience.

Korean registered patent KR1935585

The present disclosure is conceived in response to the above-described background technology, and relates to a computer program for providing a robust keyword recognition technology in a game environment.

In one embodiment of the present disclosure for solving the above-described problem, a computer program stored in a computer-readable medium executable by one or more processors is disclosed. The computer program causes the one or more processors to perform operations for providing a startup word recognition technology, and the operations include: the user's voice data received through the user terminal as input to the voice signal processing module, and the voice data is converted into the voice data. An operation of extracting one or more corresponding features, an operation of calculating a matching score between the voice data and a predetermined keyword by using the one or more features as an input of a keyword recognition model, and the user terminal based on the matching score It may include an operation of generating a control signal for calling an interface and determining to transmit it to the user terminal.

Alternatively, the operation of performing pre-processing on the voice data of the user is further included, and the pre-processing on the voice data is to remove noise included in the voice data, and the frequency is uniform in the voice data. It may be a pre-processing for removing at least one of white noise, random frequency noise with irregular frequency, and noise that may occur in a signal processing process.

Alternatively, the operation of extracting one or more features corresponding to the voice data by using the voice data of the user received through the user terminal as an input of the voice signal processing module may include a fast Fourier transform (STFT: Outputting a spectrogram through Short-Time Fourier Transform), obtaining a Mel-Spectrogram through a Mel-Filter Bank for the spectrogram, and the above It may include an operation of extracting one or more features by processing the Mel-Spectrogram with Discrete Cosine Transform (DCT).

Alternatively, the at least one feature may be a Mel Frequency Cepstral Coefficient (MFCC) feature obtained by feature vectorization of the speech data.

Alternatively, the keyword recognition model may include a recycling pooling-based convolution network as a model for determining whether the voice data is appropriate as a starting word for calling the voice-based user interface. .

Alternatively, the keyword recognition model may include at least one convolutional layer performing a convolution operation and at least one recycling pooling layer performing a recycling operation.

Alternatively, the recycling operation may be an operation of including each record included in a pooling window in a pooling feature map of a different channel with respect to the sampling feature map to which the convolution operation is applied.

Alternatively, the convolution layer generates a sampling feature map set through the convolution operation through each of an MFCC feature and one or more convolution filters, and the recycling pooling layer comprises the recycling of the sampling feature map set. A pooling feature map set can be created based on the operation.

Alternatively, the pooling feature map set includes a feature map of a size reduced than that of the feature map of the sampling feature map set, and includes feature maps equal to or greater than the number of feature maps included in the sampling feature map set. I can.

Alternatively, further comprising an operation of training the keyword recognition model through training data, and the training data includes training input data including a plurality of MFCC features and a plurality of start words corresponding to each of the MFCC features. It can be composed of learning result data.

Alternatively, the voice-based user interface is a user interface for allowing a game-related control operation based on the user's voice data, and the game-related control operation is at least one of a play control operation and an application control operation. It may include a control operation of.

In another embodiment of the present disclosure, a computing device for providing a startup word recognition technique is disclosed. The computing device includes a processor including one or more cores, a memory including program codes executable in the processor, and a network unit for transmitting and receiving data to and from a user terminal, and the processor is Extracting one or more features corresponding to the voice data by using voice data as an input of the voice signal processing module, calculating a matching score between the voice data and a predetermined keyword by using the one or more features as input of a keyword recognition model, Further, based on the matching score, it may be determined to generate a control signal for causing the user terminal to call a voice-based user interface and transmit it to the user terminal.

In yet another embodiment of the present disclosure, a method for providing a startup word recognition technique is disclosed. The method comprises the steps of extracting one or more features corresponding to the voice data by using voice data of a user received through a user terminal as an input of a voice signal processing module, and extracting one or more features corresponding to the voice data, and the one or more features as an input of a keyword recognition model. Calculating a matching score between the voice data and a predetermined keyword, and determining to transmit the control signal to the user terminal by generating a control signal for causing the user terminal to call a voice-based user interface based on the matching score. have.

The present disclosure can provide a computer program for providing a robust startup word recognition technology in a game environment.

Various aspects are now described with reference to the drawings, wherein like reference numbers are used collectively to refer to like elements. In the examples that follow, for illustrative purposes, a number of specific details are presented to provide a comprehensive understanding of one or more aspects. However, it will be apparent that such aspect(s) may be practiced without these specific details.
1 is a diagram illustrating a block diagram of a computing device that performs an operation for providing a startup word recognition technology according to an embodiment of the present disclosure.
FIG. 2 exemplarily illustrates a structure of a convolutional neural network based on a recycling pooling layer according to an embodiment of the present disclosure.
3 is a comparison of the accuracy of recognition measured through each of the max pooling-based convolutional neural network and the recycling pooling layer-based convolutional neural network for clean data and voice data including noise according to an embodiment of the present disclosure. It is an exemplary diagram for the experimental results for.
4 is a diagram exemplarily illustrating a voice-based user interface called to a user terminal when an initial word related to an embodiment of the present disclosure is recognized.
5 is a schematic diagram showing a network function related to an embodiment of the present disclosure.
6 is a flowchart illustrating an exemplary method for providing a startup word recognition technology related to an embodiment of the present disclosure.
7 is a diagram illustrating logic for implementing a method for providing a startup word recognition technology related to an embodiment of the present disclosure.
8 shows a simplified and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

Various embodiments are now described with reference to the drawings. In this specification, various descriptions are presented to provide an understanding of the present disclosure. However, it is clear that these embodiments may be implemented without this specific description.

As used herein, the terms "component", "module", "system" and the like refer to a computer-related entity, hardware, firmware, software, a combination of software and hardware, or execution of software. For example, a component may be, but is not limited to, a process executed on a processor, a processor, an object, a thread of execution, a program, and/or a computer. For example, both an application running on a computing device and a computing device may be components. One or more components may reside within a processor and/or thread of execution. A component can be localized within a single computer. A component can be distributed between two or more computers. In addition, these components can execute from a variety of computer readable media having various data structures stored therein. Components can be, for example, through a signal with one or more data packets (e.g., data from one component interacting with another component in a local system, a distributed system, and/or a signal through another system and a network such as the Internet. Depending on the data being transmitted), it may communicate via local and/or remote processes.

In addition, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or”. That is, unless specified otherwise or is not clear from the context, "X employs A or B" is intended to mean one of the natural inclusive substitutions. That is, X uses A; X uses B; Or when X uses both A and B, "X uses A or B" can be applied to either of these cases. In addition, the term “and/or” as used herein should be understood to refer to and include all possible combinations of one or more of the listed related items.

In addition, the terms "comprises" and/or "comprising" should be understood to mean that the corresponding features and/or components are present. However, it is to be understood that the terms "comprising" and/or "comprising" do not exclude the presence or addition of one or more other features, elements, and/or groups thereof. In addition, unless otherwise specified or when the context is not clear to indicate a singular form, the singular in the specification and claims should be interpreted as meaning "one or more" in general.

Those of skill in the art would further describe the various illustrative logical blocks, configurations, modules, circuits, means, logics, and algorithm steps described in connection with the embodiments disclosed herein, including electronic hardware, computer software, or a combination of both. It should be recognized that it can be implemented as To clearly illustrate the interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented in hardware or as software depends on the specific application and design restrictions imposed on the overall system. Skilled technicians can implement the described functionality in various ways for each particular application. However, such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

Description of the presented embodiments is provided so that a person of ordinary skill in the art of the present disclosure can use or implement the present invention. Various modifications to these embodiments will be apparent to those of ordinary skill in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present invention is not limited to the embodiments presented herein. The invention is to be accorded the widest scope consistent with the principles and novel features presented herein.

The scope of rights to the method in the claims of the present disclosure is generated by the functions and features described in each step, and unless the precedence of the order is specified in each step constituting the method, the claim It is not affected by the order of description of each step in the range. For example, in the claims described in a manner that includes steps A and B, even if step A is described before step B, the scope of rights is not limited to that step A must precede step B.

According to an embodiment of the present disclosure, the computing device 100 and one or more user terminals may transmit and receive data through wireless and/or wired interconnection.

The user terminal may include a personal computer (PC), a notebook (note book), a mobile terminal, a smart phone, a tablet PC, etc., and can access a wired/wireless network. It can include all types of terminals that are present.

The user terminal may access the game server to play a game provided by the game server, and obtain voice data uttered from the user of the user terminal during the game. In addition, the user terminal may acquire voice data related to the user's utterance and transmit it to the computing device.

According to an embodiment of the present disclosure, when receiving voice data related to a user's utterance while playing a game from a user terminal, the computing device 100 determines whether the corresponding voice data is a starting word for calling a voice-based user interface. can do. The computing device 100 of the present disclosure may include all kinds of computing devices capable of calculating data in electronic form, and for example, general computing devices such as personal computers and server computers, and mobile terminals (smartphones). It may include a computing device having limited computing power, such as a smartphone) or a tablet). In addition, the computing device can utilize TensorFlow Lite of the Google firebase SDK as an inference engine for work optimized for mobile devices.

In addition, the computing device 100 may provide a voice-based user interface to allow a user of a user terminal to control a game through voice in connection with recognition of a starter word. Specifically, the computing device 100 may provide a voice-based user interface based on robust keyword recognition in various environments (eg, driving, cooking, or busy situations) in which a mobile game is played. Specifically, the processor 120 may provide an effective method for performing a keyword search operation through a deep neural network in a mobile game environment. To this end, the present disclosure proposes a new neural network layer of the recycling pooling layer. The structure of a convolutional network based on a recycling pooling layer of the present disclosure can improve a keyword search function in all voice data including clean or noise in a mobile game environment.

In the present disclosure, a detailed description of the process of determining the voice data received from one or more user terminals by the computing device 100 as a starting word and the voice-based user interface provided to the user terminal when the starting word is recognized will be described with reference to the following drawings. It will be described later.

1 is a block diagram of a computing device that performs an operation for providing a startup word recognition technology according to an embodiment of the present disclosure. The computing device 100 may provide a method of providing a startup word recognition technology or a method of learning a keyword recognition model.

1, the computing device 100 may include a network unit 110, a processor 120, and a memory 130. The above-described components are exemplary, and the scope of the present disclosure is not limited to the above-described components. That is, additional components may be included or some of the above-described components may be omitted according to implementation aspects of the embodiments of the present disclosure.

According to an embodiment of the present disclosure, the computing device 100 may include a network unit 110 that transmits and receives data to and from one or more user terminals. Specifically, the network unit 110 may transmit/receive voice data according to an embodiment of the present disclosure and information on one or more features acquired through the corresponding voice data with one or more user terminals.

For example, the computing device 100 may receive voice data related to a user's utterance from each of one or more user terminals through the network unit 110. For another example, the computing device 100 may transmit a control signal for causing the user terminal to display a voice-based user interface to the user terminal through the network unit 110. The detailed description of the information transmitted and received by the network unit described above is only an example, and the present disclosure is not limited thereto.

The memory 130 of the computing device 100 may store information in any form generated or determined by the processor 120 and information in any form received by the network unit 110. In addition, the memory 130 may store data supporting various functions of the computing device 100. The memory 130 may store a plurality of application programs and instructions driven by the computing device 100. At least one of these application programs may exist in the computing device 100 from the time of release. For example, the memory 130 may store application programs for extracting features by inputting user's voice data. For another example, the memory 130 may store a pre-trained neural network in order to perform a method of recognizing a startup word of the computing device 100. The description of the information stored in the above-described memory is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the processor 120 may be composed of one or more cores, and a central processing unit (CPU) and a general purpose graphics processing unit (GPGPU) of a computing device , A tensor processing unit (TPU) may include a processor for data analysis and deep learning. The processor 120 may read a computer program stored in the memory 130 to process data for machine learning according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the processor 120 may perform an operation for learning a neural network. The processor 120 processes input data for learning in deep learning (DL), extracts features from the input data, calculates errors, and uses backpropagation to update the weights of the neural network. You can perform calculations. At least one of the CPU, GPGPU, and TPU of the processor 120 may process learning of the network function. For example, the CPU and GPGPU can work together to learn network functions and classify data using network functions. In addition, in an embodiment of the present disclosure, learning of a network function and data classification using a network function may be processed by using processors of a plurality of computing devices together. In addition, the computer program executed in the computing device according to an embodiment of the present disclosure may be a CPU, a GPGPU, or a TPU executable program.

The processor 120 may provide a keyword recognition model according to an embodiment of the present disclosure by reading a computer program stored in the memory 130. According to an embodiment of the present disclosure, the processor 120 may perform calculation for recognizing an initial word based on voice data. According to an embodiment of the present disclosure, the processor 120 may perform calculation for training a keyword recognition model.

According to an embodiment of the present disclosure, the processor 120 may typically process the overall operation of the computing device 100. The processor 120 can provide or process appropriate information or functions to the user terminal by processing signals, data, information, etc. that are input or output through the above-described components, or by running an application program stored in the memory 130. have.

According to an embodiment of the present disclosure, the processor 120 may receive the user's voice data received through the user terminal. The voice data is voice data related to the user's utterance, and may be voice data related to the user's utterance including various noises in a game playing environment through the user terminal.

The processor 120 may receive voice data through the network unit 110. Receiving voice data according to an embodiment of the present disclosure may be receiving or loading voice data stored in the memory 130. Receiving the voice data may be receiving or loading voice data from another storage medium, another computing device, or a separate processing module in the same computing device based on the existence/any communication means.

According to an embodiment of the present disclosure, the processor 120 may perform preprocessing on voice data received from a user terminal. The processor 120 may perform preprocessing to increase a recognition rate for voice data. The pre-processing of the voice data may be for removing noise (ie, ambient noise) included in the voice data received by the user terminal. That is, the processor 120 may remove sounds other than speech by analyzing the waveform of the voice included in the voice data. Specifically, the processor 120 may remove at least one specific frequency by analyzing the frequency of the voice data. The processor 120 may include white noise having a uniform frequency, random frequency noise having an irregular frequency, or various noises generated according to a recording device or a signal processing process, as the voice data received from the user terminal. That is, the processor 120 may perform preprocessing of removing noise having various frequencies included in each of the voice data. For a specific example, the processor 120 classifies using a machine learning algorithm such as SVM (Supporting Vector Machine) to determine the type of noise included in the voice data, and responds to each noise including different frequencies. Noise can be removed through a noise removal algorithm. The above-described noise removal algorithm is only an example, and the present disclosure is not limited thereto.

That is, the processor 120 removes noise included in the voice data through pre-processing of the user's voice data received from the user terminal, thereby improving recognition accuracy related to the starter word recognition, which will be described later.

According to an embodiment of the present disclosure, the processor 120 may extract one or more features corresponding to the voice data by using voice data of the user received through the user terminal as input to the voice signal processing module. The voice signal processing module may be a module for extracting features from voice data of a user. Specifically, the voice signal processing module may be a module for extracting one or more features corresponding to the voice data through processing of the voice data. The processing of speech data may refer to a process of processing the speech data in a format for processing by one or more network functions included in the keyword recognition model. For example, processing of voice data may include deleting, converting, or converting at least some data of voice data into a format for input into a keyword recognition model. Further, processing of the voice data may mean partially correcting the waveform of the voice data. Hereinafter, a process of processing the voice data received by the voice signal processing module into a format for processing by one or more network functions (ie, a keyword recognition model) will be described in detail.

According to an embodiment of the present disclosure, the voice signal processing module may output a spectrogram corresponding to voice data through a short-time Fourier transform (STFT) for voice data. Specifically, the speech signal processing module applies a hann function (or Hanning function) to speech data as a window function, and performs fast Fourier transform for each 512 sample window having a hop size of 128 samples. Can be performed to convert the voice data into a spectrogram and output it. The spectrogram is for visualizing and grasping a sound or wave, and may be a combination of characteristics of a waveform and a spectrum. The spectrogram may represent a difference in amplitude according to a change in a time axis and a frequency axis as a difference in a print density or a display color.

In addition, the voice signal processing module may acquire a Mel-Spectrogram through a Mel-Filter Bank for the spectrogram. In general, the human cochlea may have different vibrating parts depending on the frequency of the voice data. In addition, the human cochlea has a characteristic that it detects a frequency change well in a low frequency band and does not detect a frequency change well in a high band. Accordingly, it is possible to obtain a mel-spectrogram from the spectrogram by using the mel-filter bank so as to have a recognition ability similar to the characteristics of the human cochlea with respect to voice data. That is, the mel-filter bank may be a method of applying a smaller filter bank in a low frequency band and applying a wider filter bank toward a higher band. In other words, the voice signal processing module may obtain a mel-spectrogram by applying the mel-filter bank to the spectrogram in order to recognize voice data so as to resemble the characteristics of a human cochlea. That is, the mel-spectrogram may include a frequency component reflecting human auditory characteristics.

In addition, the speech signal processing module may extract one or more features by processing the Mel-Spectrogram with Discrete Cosine Transform (DCT). Specifically, the coefficient may be determined to be 40 (ie, 40 frequency bands) from the result of the discrete cosine transform of the mel spectrogram. That is, one or more features extracted through the speech signal processing module may be Mel-frequency cepstral coefficients (MFCC) features. The MFCC feature may be related to a characteristic value indicating what form the spectrum is composed of.

That is, the voice signal processing module may extract one or more features corresponding to the voice data through processing of the user's voice data, and one or more features are MFCC features that feature vectorized voice data received through the user terminal. Can be For example, the MFCC feature extracted through the voice signal processing module may have a resolution of 122x40 for one second of voice data. The detailed description of the above-described MFCC feature is only an example, and the present disclosure is not limited thereto.

According to an exemplary embodiment of the present disclosure, the processor 120 may determine whether voice data is appropriate as a starting word for calling a voice-based user interface by using one or more features as input of a keyword recognition model. Specifically, the processor 120 calculates a matching score between the voice data and a predetermined keyword by processing one or more features extracted by processing the user's voice data with the voice signal processing module as an input of the keyword recognition model, so that the corresponding voice data is It is possible to determine whether it is appropriate as a starting word for calling a voice-based user interface. The keyword recognition model may be a deep neural network model for calculating a score corresponding to speech data based on an MFCC feature corresponding to speech data.

That is, the keyword recognition model is a neural network model trained to calculate a high matching score when the matching similarity between the voice data received through the user terminal and a predetermined keyword is high, and when the matching similarity is low, a low matching score ( Alternatively, it may be one or more network functions). The keyword recognition model may be a neural network model learned by the processor 120. The processor 120 may train a keyword recognition model through the training data. The learning data may be composed of learning input data including a plurality of MFCC features and learning result data including a plurality of start words corresponding to each of the MFCC features.

Hereinafter, a method of determining whether voice data is an initial word by inputting one or more features (ie, MFCC features) in the keyword recognition model will be described in detail.

The keyword recognition model may be a model for determining whether voice data is appropriate as a starting word for calling a voice-based user interface, and may include a recycling pooling-based convolutional network. Specifically, the keyword recognition model may include one or more convolution layers performing a convolution operation and one or more recycling pooling layers performing a recycling operation.

In detail, the keyword recognition model may generate a sampling feature map set through a convolution operation of each of the MFCC feature 210 and one or more convolution filters through a convolution layer. One or more convolution filters may mean one or more filters (or kernels) for performing convolution with an MFCC feature. For example, a first sampling feature map may be generated as a result of performing convolution between an MFCC feature and a first convolution filter, and a second sampling feature map is generated as a result of performing convolution between the MFCC feature and the second convolution filter. Can be. In this case, the first sampling feature map and the second sampling feature map may constitute a sampling feature map set. The detailed description of the above-described convolution filter and sampling feature map is merely an example, and the present disclosure is not limited thereto.

That is, the number of sampling feature maps included in the sampling feature map set (that is, the number of channels of the feature map) may correspond to the number of one or more convolution filters that perform convolution with the MFCC feature.

For a specific example, referring to FIG. 2, the convolution layer may perform convolution with each of the MFCC feature 210 and one or more convolution filters, and a sampling feature map set including 64 sampling feature maps ( 220). In this case, the number of one or more convolution filters convolved with the MFCC feature 210 may be 64. That is, as a result of convolution between the MFCC feature 210 and each of the 64 convolution filters, a sampling feature map set 220 including 64 feature maps may be generated. The description of specific values of the above-described convolution filter and sampling feature map is only an example, and the present disclosure is not limited thereto.

In addition, the key word recognition model may generate a pooled feature map set by performing a recycling operation on the sampled feature map set through the recycling pooling layer. The recycling operation performed in the recycling pooling layer included in the keyword recognition model may be an operation of including each record included in the pooling window in the pooling feature map of a different channel with respect to the sampling feature map to which the convolution operation is applied.

A neural network model based on a general convolutional neural network utilizes the Max-pooling method. The Max-pooling method is a method that is widely used in deep learning-based classification and detection in computer vision tasks, and can prevent concerns of overfitting due to a large number of features in classification and detection of images. However, when a feature corresponding to voice data is to be extracted through a general Max-pooling CNN, there is a concern that the accuracy of recognition may be reduced as the main feature corresponding to the voice data is removed. Specifically, since Max-pooling extracts as features having the maximum value among records included in a specific kernel size, feature values corresponding to the remaining records may be lost.

For a specific example, when a 2x2 kernel of Max-pooling is applied using stride 2 to the first feature map, Max-pooling is one of the maximum values among four records corresponding to the kernel size in the first feature map. By extracting only the features of, 75% of the features (i.e., the remaining three features) can be removed.

Accordingly, the present disclosure proposes a new convolutional network structure incorporating a recycling pooling layer. According to the present disclosure, the recycling pooling layer may generate a pooling feature map set through a recycling operation for a sampling feature map set (ie, a combination of sampling feature maps to which a convolution operation is applied).

In more detail with reference to FIG. 2, the sampling feature map set 220 to which the convolution operation is applied may generate the pooling feature map set 240 through a recycling operation of a recycling pooling layer. As shown in FIG. 2, the sampling feature map set 220 has a resolution of 122x40 and may include 64 feature maps. In addition, each feature map included in the sampling feature map set 220 may include a plurality of records. In this case, each of a plurality of records included in the sampling feature map may be rearranged in a pooling feature map of a different channel through a recycling operation corresponding to the pooling window (ie, kernel size 2, starride 2). Specifically, as the pooling window is sequentially moved (that is, moved by two spaces corresponding to stride 2) corresponding to each feature map, the same record values may be rearranged into respective channel spaces. That is, as shown in FIG. 2, records corresponding to '1' are rearranged in the first pooling feature map of the first channel 231 according to the movement of the pooling window in the first sampling feature map of the sampling feature map set. , Records corresponding to '2' are rearranged in the first pooling feature map of the second channel 232, records corresponding to '3' are rearranged in the first pooling feature map of the third channel 233, and Records corresponding to '4' may be rearranged in the first pooling feature map of the fourth channel 234. That is, as the above-described recycling operation is performed in the recycling pooling layer, the pooling feature map set 240 may be generated. In this case, the pooling feature map set 240 includes feature maps of a size reduced than that of the feature maps of the sampled feature map set, and contains a larger number of feature maps than the number of feature maps included in the sampled feature map set. Can include.

Specifically, the resolution of the generated pooling feature map set 240 may be reduced by half according to a recycling operation based on a pooling window (ie, kernel size 2, starride 2). In addition, by rearranging the records included in each of the sampling feature maps (that is, records related to 1, 2, 3, and 4) into different channels, four channels are formed and included in the pooling feature map set 240 The number of feature maps can be increased by four times the number of feature maps included in the sampled feature map set. That is, as shown in FIG. 2, the recycling pooling layer may reconstruct a 122x44x64 feature map to 61x20x256.

In addition, the keyword recognition model may output the matching score 250 by performing convolution on the pooled feature map set again and processing the fully connected layer as a fully connected layer.

Therefore, recognition accuracy can be improved by utilizing all the features without loss of features in the pooling process. In other words, as the recycling pooling layer operates as a multi-resolution operation, the accuracy of the output may be improved. Detailed descriptions of the operation method and the number of each layer constituting the keyword recognition model described above with reference to FIG. 2 are only examples, and each of the layers included in the keyword recognition model may exist in plural. It will be apparent to those skilled in the art that the size and number of channels in a layer are not limited to the above description.

3 illustrates the accuracy of recognition measured through a max pooling-based convolutional neural network and a recycling pooling layer-based convolutional neural network for each of clean voice data and voice data including noise according to an embodiment of the present disclosure. It is an exemplary diagram for the experimental results for comparison.

To compare the performance between the existing Max-pooling-based convolutional neural network and the recycling pooling-based convolutional neural network, a performance experiment was conducted by constructing data sets in various environments. Specifically, a positive data set was constructed from data collected from offices, homes, restaurants, and unjeong vehicles, and a voice data set was constructed from data collected from radio, television, and conversation. In addition, voice data related to speech including only a part of the sentence (ie, incomplete sentences) was secured and added to the voice data set.

For the experiment on comparing the performance of each network function, a training data set was constructed from 50,000 positive data sets and 80,000 negative data sets from 109 users in various environments. Thereafter, training for each neural network model was performed with 60 epochs of 256 batch sizes. During the training period, learning was performed by setting the learning speed to the power of 10 for the first 56 epochs, and setting the learning speed to the -4 power of 10 for the remaining epochs (ie, three epochs). In addition, the validity was verified by randomly selecting 10% data as a validation data set for validation.

In addition, about 400 utterances from 13 users are collected in a clean environment (i.e., an environment where noise is not included in the negative data) and noise is present, and a positive test data set and a negative test data set are constructed with the same size. I did.

As a result of testing each convolutional neural network through a test data set including a positive test data set and a negative test data set, a receiver operating characteristics (ROC) curve as shown in FIG. 3 was obtained.

Specifically, the performance of the model is calculated as an ROC curve based on a false positive rate (FPR) and a true positive rate (TPR), and the higher the area under the curve (AUC), the better the performance. I can.

As shown in FIG. 3, it can be seen that the recycling pooling-based convolutional neural network exhibits improved performance compared to the existing convolutional neural network (ie, Max-pooling) in both a clean environment and an environment in which noise exists. In particular, it can be seen that the accuracy of keyword recognition for data acquired in a noisy environment is much improved.

That is, when using a keyword recognition model including a recycling pooling layer-based convolutional neural network according to an embodiment of the present disclosure, calculation of a matching score that is a criterion for discriminating starter word recognition in both a clean environment and an environment containing noise Accuracy can be improved.

According to an embodiment of the present disclosure, the processor 120 may provide a voice-based user interface based on voice data of a user. Specifically, the processor 120 may extract one or more features corresponding to the voice data received from the user terminal, and calculate a matching score with a predetermined keyword by using the one or more features as an input of the keyword recognition model. The processor 120 may provide a voice-based user interface to the user terminal by generating a control signal for causing the user terminal to call a voice-based user interface based on the calculated matching score and transmitting it to the user terminal.

The voice-based user interface may be a user interface for allowing a game-related control operation based on user voice data. Specifically, when the voice-based user interface is called to the user terminal as the user's voice data is determined as a starting word, the voice-based user interface may allow the user of the user terminal to control the game through voice. The control operation related to the game may include at least one of a play control operation and an application control operation.

The play control operation may be an operation of controlling the play of a character related to a game provided to the user terminal. For example, when the voice data corresponding to the user's utterance of'move the character to the hunting ground' is received after the voice-based user interface is called as the starting word is recognized, a separate input from the user through the user terminal Without manipulation, a control operation of moving the corresponding character to the hunting ground can be performed. For another example, when the voice data corresponding to the user's utterance of'start automatic hunting' is received after the voice-based user interface is called as the starting word is recognized, the user can initiate automatic hunting. Without a separate input manipulation through, a control operation for causing the corresponding character to start automatic hunting for a monster may be performed. The detailed description of the above-described play control operation is only an example, and the present disclosure is not limited thereto.

In the case of receiving voice data corresponding to the user's utterance of'Turn off background music' after the voice-based user interface is called as the starting word is recognized, the user's separate input operation through the user terminal Without, the background music provided by the game may be stopped. For another example, when the user's utterance'End the game' is received after the voice-based user interface is called as the starting word is recognized, the game is terminated without a user's separate input manipulation through the user terminal. A control operation to be performed can be performed. The detailed description of the above-described application control operation is only an example, and the present disclosure is not limited thereto.

4 is an exemplary diagram of a user interface implemented through a keyword search system based on a recycling pooling layer in a game environment of the present disclosure. As shown in FIG. 4, the processor 120 controls to call the voice-based user interface 310 on at least one area of the game screen 300 displayed on the user terminal as the user's voice data is recognized as a startup word. It can be determined to generate a signal and transmit it to the user terminal. In other words, by recognizing the user's voice data as a starting word, a voice recognition-based user interface is provided in a mobile game environment to perform a game in various environments (for example, while driving, cooking, or in a busy situation). User convenience can be increased.

5 is a schematic diagram showing a network function related to an embodiment of the present disclosure.

Throughout this specification, a computational model, a neural network, a network function, and a neural network may be used with the same meaning. A neural network may consist of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. The neural network is composed of at least one or more nodes. Nodes (or neurons) constituting neural networks may be interconnected by one or more links.

In a neural network, one or more nodes connected through a link may relatively form a relationship between an input node and an output node. The concepts of an input node and an output node are relative, and any node in an output node relationship with respect to one node may have an input node relationship in a relationship with another node, and vice versa. As described above, the input node to output node relationship can be created around the link. One or more output nodes may be connected to one input node through a link, and vice versa.

In the relationship between the input node and the output node connected through one link, the value of the output node may be determined based on data input to the input node. Here, a node interconnecting the input node and the output node may have a weight. The weight may be variable and may be changed by a user or an algorithm in order for a neural network to perform a desired function. For example, when one or more input nodes are interconnected to one output node by respective links, the output node includes values input to input nodes connected to the output node and a link corresponding to each input node. The output node value may be determined based on the weight.

As described above, in a neural network, one or more nodes are interconnected through one or more links to form an input node and an output node relationship in the neural network. The characteristics of the neural network may be determined according to the number of nodes and links in the neural network, the association relationship between the nodes and the links, and a weight value assigned to each of the links. For example, when there are the same number of nodes and links, and two neural networks having different weight values between the links exist, the two neural networks may be recognized as being different from each other.

A neural network may be configured including one or more nodes. Some of the nodes constituting the neural network may configure one layer based on distances from the initial input node. For example, a set of nodes having a distance n from the initial input` node may constitute n layers. The distance from the first input node may be defined by the minimum number of links that must be traversed to reach the node from the first input node. However, the definition of such a layer is arbitrary for explanation, and the order of the layer in the neural network may be defined in a different way than that described above. For example, the layer of nodes may be defined by the distance from the last output node.

The first input node may mean one or more nodes to which data is directly input without going through a link in a relationship with other nodes among nodes in the neural network. Alternatively, it may mean nodes that other input nodes connected by a link do not have in a relationship between nodes in a neural network network based on a link. Similarly, the final output node may mean one or more nodes that do not have an output node in a relationship with other nodes among nodes in the neural network. In addition, the hidden node may refer to nodes constituting a neural network other than the first input node and the last output node. In the neural network according to an embodiment of the present disclosure, the number of nodes in the input layer may be the same as the number of nodes in the output layer, and the number of nodes decreases and then increases again as the input layer proceeds to the hidden layer. I can. In addition, the neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes of an input layer is less than the number of nodes of an output layer, and the number of nodes decreases as the number of nodes progresses from the input layer to the hidden layer. have. In addition, the neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes of an input layer may be greater than the number of nodes of an output layer, and the number of nodes increases as the number of nodes progresses from the input layer to the hidden layer. I can. The neural network according to another embodiment of the present disclosure may be a neural network in the form of a combination of the aforementioned neural networks.

A deep neural network (DNN) may mean a neural network including a plurality of hidden layers in addition to an input layer and an output layer. Deep neural networks can be used to identify potential structures in data. In other words, it is possible to understand the potential structures of photos, texts, videos, voices, and music (for example, what objects are in photos, what are the content and emotions of the text, what are the contents and emotions of the voice, etc.). . Deep neural networks include convolutional neural networks (CNNs), recurrent neural networks (RNNs), auto encoders, Generative Adversarial Networks (GANs), and restricted Boltzmann machines (RBMs). boltzmann machine), deep belief network (DBN), Q network, U network, and Siam network. The description of the above-described deep neural network is only an example, and the present disclosure is not limited thereto.

In an embodiment of the present disclosure, the network function may include an auto encoder. The auto encoder may be a kind of artificial neural network for outputting output data similar to input data. The auto encoder may include at least one hidden layer, and an odd number of hidden layers may be disposed between input/output layers. The number of nodes of each layer may be reduced from the number of nodes of the input layer to an intermediate layer called a bottleneck layer (encoding), and then reduced and expanded symmetrically from the bottleneck layer to an output layer (symmetric with the input layer). In this case, in the example of FIG. 2, although the dimensionality reduction layer and the dimensional restoration layer are illustrated as being symmetrical, the present disclosure is not limited thereto, and nodes of the dimensionality reduction layer and the dimensional restoration layer may or may not be symmetrical. Auto encoders can perform nonlinear dimensionality reduction. The number of input layers and output layers may correspond to the number of sensors remaining after pre-processing of input data. In the auto-encoder structure, the number of nodes of the hidden layer included in the encoder may decrease as the distance from the input layer increases. If the number of nodes in the bottleneck layer (the layer with the fewest nodes located between the encoder and the decoder) is too small, a sufficient amount of information may not be delivered, so more than a certain number (e.g., more than half of the input layer, etc.) ) May be maintained.

The neural network may be learned in at least one of supervised learning, unsupervised learning, and semi supervised learning. Learning of neural networks is to minimize output errors. In learning of a neural network, iteratively inputs training data to the neural network, calculates the output of the neural network for the training data and the error of the target, and transfers the error of the neural network from the output layer of the neural network to the input layer in a direction to reduce the error. This is a process of updating the weight of each node of the neural network by backpropagation in the direction. In the case of teacher learning, learning data in which the correct answer is labeled in each learning data is used (ie, labeled learning data), and in the case of non-satellite learning, the correct answer may not be labeled in each learning data. That is, for example, in the case of teacher learning related to data classification, the learning data may be data in which a category is labeled with each learning data. Labeled training data is input to a neural network, and an error may be calculated by comparing an output (category) of the neural network with a label of the training data. As another example, in the case of comparative history learning regarding data classification, an error may be calculated by comparing the input training data with the neural network output. The calculated error is backpropagated in the neural network in the reverse direction (ie, from the output layer to the input layer), and the connection weight of each node of each layer of the neural network may be updated according to the backpropagation. A change amount may be determined according to a learning rate in the connection weight of each node to be updated. The computation of the neural network for the input data and the backpropagation of the error can constitute a learning cycle (epoch). The learning rate may be applied differently according to the number of repetitions of the learning cycle of the neural network. For example, a high learning rate is used in the early stages of training of a neural network, so that the neural network quickly secures a certain level of performance to increase efficiency, and a low learning rate can be used in the later stages of training to increase accuracy.

In the learning of a neural network, in general, the training data may be a subset of actual data (that is, data to be processed using the learned neural network), and thus, errors in the training data decrease, but errors in the actual data occur. There may be increasing learning cycles. Overfitting is a phenomenon in which errors in actual data increase due to over-learning on learning data. For example, a neural network learning a cat by showing a yellow cat may not recognize that it is a cat when it sees a cat other than yellow, which may be a kind of overfitting. Overfitting can cause an increase in errors in machine learning algorithms. Various optimization methods can be used to prevent such overfitting. In order to prevent overfitting, methods such as increasing training data, regularization, and dropout in which some nodes of the network are omitted during the training process may be applied.

A computer-readable medium storing a data structure according to an embodiment of the present disclosure is disclosed.

Data structure can refer to the organization, management, and storage of data that enables efficient access and modification of data. The data structure may refer to an organization of data for solving a specific problem (eg, data search, data storage, data modification in the shortest time). Data structures may be defined as physical or logical relationships between data elements, designed to support specific data processing functions. The logical relationship between data elements may include a connection relationship between data elements that a user thinks. The physical relationship between the data elements may include an actual relationship between the data elements physically stored in a computer-readable storage medium (eg, hard disk). The data structure may specifically include a set of data, a relationship between data, and a function or instruction applicable to the data. Through an effectively designed data structure, the computing device can perform calculations while using the resources of the computing device to a minimum. Specifically, computing devices can increase the efficiency of computation, reading, insertion, deletion, comparison, exchange, and search through an effectively designed data structure.

The data structure can be classified into a linear data structure and a non-linear data structure according to the shape of the data structure. The linear data structure may be a structure in which only one data is connected after one data. The linear data structure may include a list, a stack, a queue, and a deck. The list may mean a series of data sets in which order exists internally. The list may include a linked list. The linked list may be a data structure in which data is linked in a way that each data has a pointer and is linked in a single line. In the linked list, the pointer may include connection information with the next or previous data. The linked list can be expressed as a single linked list, a double linked list, or a circular linked list. The stack can be a data listing structure that allows limited access to data. The stack may be a linear data structure that can process (eg, insert or delete) data only at one end of the data structure. The data stored in the stack may be a data structure (LIFO-Last in First Out) that comes out sooner as it enters the stack. A queue is a data arrangement structure that allows limited access to data, and unlike a stack, a queue may be a data structure (FIFO-First in First Out) that comes out later as data stored later. A deck can be a data structure that can process data at either end of the data structure.

The nonlinear data structure may be a structure in which a plurality of data is connected behind one data. The nonlinear data structure may include a graph data structure. The graph data structure may be defined as a vertex and an edge, and the edge may include a line connecting two different vertices. The graph data structure may include a tree data structure. The tree data structure may be a data structure in which a path connecting two different vertices among a plurality of vertices included in the tree is one. That is, it may be a data structure that does not form a loop in the graph data structure.

Throughout this specification, a computational model, a neural network, a network function, and a neural network may be used with the same meaning. (Hereinafter, it will be unified and described as a neural network.) The data structure may include a neural network. In addition, the data structure including the neural network may be stored in a computer-readable medium. The data structure including the neural network may also include data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, an active function associated with each node or layer of the neural network, and a loss function for learning the neural network. have. The data structure including the neural network may include any of the components disclosed above. In other words, the data structure including the neural network is the data input to the neural network, the weight of the neural network, the hyper parameter of the neural network, the data obtained from the neural network, the active function associated with each node or layer of the neural network, the loss function for training the neural network, etc. It may be configured to include any combination of. In addition to the above-described configurations, the data structure including the neural network may include any other information that determines the characteristics of the neural network. In addition, the data structure may include all types of data that are used or generated in a neural network operation process, and is not limited to the above-described matters. Computer-readable media may include computer-readable recording media and/or computer-readable transmission media. A neural network may consist of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. The neural network is composed of at least one or more nodes.

The data structure may include data input to the neural network. A data structure including data input to the neural network may be stored in a computer-readable medium. The data input to the neural network may include training data input during the neural network training process and/or input data input to the neural network on which the training has been completed. Data input to the neural network may include data that has undergone pre-processing and/or data to be pre-processed. Pre-processing may include a data processing process for inputting data into a neural network. Accordingly, the data structure may include data to be pre-processed and data generated by pre-processing. The above-described data structure is only an example, and the present disclosure is not limited thereto.

The data structure may include weights of the neural network. (In the present specification, weights and parameters may have the same meaning.) In addition, a data structure including a weight of a neural network may be stored in a computer-readable medium. The neural network may include a plurality of weights. The weight may be variable and may be changed by a user or an algorithm in order for a neural network to perform a desired function. For example, when one or more input nodes are interconnected to one output node by respective links, the output node includes values input to input nodes connected to the output node and a link corresponding to each input node. The output node value may be determined based on the parameter. The above-described data structure is only an example, and the present disclosure is not limited thereto.

As an example and not a limitation, the weight may include a weight variable in a neural network training process and/or a weight on which neural network training has been completed. The variable weight in the neural network learning process may include a weight at a time point at which the learning cycle starts and/or a weight variable during the learning cycle. The weight at which the neural network training is completed may include the weight at which the learning cycle is completed. Accordingly, the data structure including the weight of the neural network may include a data structure including a weight variable during the neural network training process and/or a weight on which the neural network training is completed. Therefore, it is assumed that the above-described weight and/or a combination of each weight are included in the data structure including the weight of the neural network. The above-described data structure is only an example, and the present disclosure is not limited thereto.

The data structure including the weights of the neural network may be serialized and then stored in a computer-readable storage medium (eg, memory, hard disk). Serialization may be a process of storing data structures on the same or different computing devices and converting them into a form that can be reconstructed and used later. The computing device serializes the data structure to transmit and receive data through a network. The data structure including the weights of the serialized neural network can be reconstructed in the same or different computing devices through deserialization. The data structure including the weights of the neural network is not limited to serialization. Furthermore, the data structure including the weight of the neural network is a data structure to increase the efficiency of operation while using the resources of the computing device to a minimum (e.g., B-Tree, Trie, m-way search tree, AVL tree, in nonlinear data structure, Red-Black Tree). The foregoing is only an example, and the present disclosure is not limited thereto.

The data structure may include a hyper-parameter of a neural network. In addition, the data structure including the hyper parameter of the neural network may be stored in a computer-readable medium. The hyper parameter may be a variable that is variable by a user. Hyper parameters include, for example, learning rate, cost function, number of iterations of learning cycles, weight initialization (e.g., setting the range of weight values to be weighted to be initialized), Hidden Unit It may include the number (eg, the number of hidden layers, the number of nodes of the hidden layer). The above-described data structure is only an example, and the present disclosure is not limited thereto.

6 is a flowchart illustrating an exemplary method for providing a startup word recognition technology related to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the computing device 100 may extract one or more features corresponding to the voice data by using the user's voice data received through the user terminal as an input of the voice signal processing module (410). .

According to an embodiment of the present disclosure, the computing device 100 may calculate a matching score between voice data and a predetermined keyword by using one or more features as input of a keyword recognition model (420 ).

According to an embodiment of the present disclosure, the computing device 100 may determine to generate a control signal for causing the user terminal to call a voice-based user interface based on the matching score and transmit it to the user terminal (430).

The order of the steps illustrated in FIG. 6 may be changed as necessary, and at least one or more steps may be omitted or added. That is, the above-described steps are only one embodiment of the present disclosure, and the scope of the present disclosure rights is not limited thereto.

7 is a diagram illustrating logic for implementing a method for providing a startup word recognition technology related to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the computing device 100 may be implemented by the following logic.

According to an embodiment of the present disclosure, the computing device 100 uses the user's voice data received through the user terminal as an input of the voice signal processing module and extracts one or more features corresponding to the voice data. , Logic 520 for calculating a matching score between speech data and a predetermined keyword by using at least one feature as an input of a keyword recognition model, and a control signal for causing a user terminal to call a speech-based user interface based on the matching score. Thus, it may include a logic 530 for determining the transmission to the user terminal.

Alternatively, a logic for performing preprocessing on the user's voice data is further included, and the preprocessing on the voice data is to remove noise included in the voice data, and the frequency is uniform in the voice data. It may be a preprocessing to remove at least one of white noise, random frequency noise having an irregular frequency, and noise that may occur in a signal processing process.

Alternatively, the logic for extracting one or more features corresponding to the voice data by using the voice data of the user received through the user terminal as an input of the voice signal processing module is a fast Fourier transform (STFT) for the voice data. : Logic for outputting a spectrogram through Short-Time Fourier Transform), for obtaining a Mel-Spectrogram through a Mel-Filter Bank for the spectrogram It may include logic and logic for extracting one or more features by processing the Mel-Spectrogram with Discrete Cosine Transform (DCT).

Alternatively, the at least one feature may be a Mel Frequency Cepstral Coefficient (MFCC) feature obtained by feature vectorization of the speech data.

Alternatively, the keyword recognition model may include a recycling pooling-based convolution network as a model for determining whether the voice data is appropriate as a starting word for calling the voice-based user interface. .

Alternatively, the keyword recognition model may include at least one convolutional layer performing a convolution operation and at least one recycling pooling layer performing a recycling operation.

Alternatively, the recycling operation may be logic for including each record included in the pooling window in the pooling feature map of a different channel with respect to the sampling feature map to which the convolution operation is applied.

Alternatively, the convolution layer generates a sampling feature map set through the convolution operation through each of an MFCC feature and one or more convolution filters, and the recycling pooling layer comprises the recycling of the sampling feature map set. A pooling feature map set can be created based on the operation.

Alternatively, the pooling feature map set includes a feature map of a size reduced than that of the feature map of the sampling feature map set, and includes feature maps equal to or greater than the number of feature maps included in the sampling feature map set. I can.

Alternatively, it further includes logic for training the keyword recognition model through training data, and the training data includes training input data including a plurality of MFCC features and a plurality of starter words corresponding to each of the MFCC features. It may be composed of included learning result data.

Alternatively, the voice-based user interface is a user interface for allowing a game-related control operation based on the user's voice data, and the game-related control operation is at least one of a play control operation and an application control operation. It may include a control operation of.

According to an embodiment of the present disclosure, logic for implementing a computing device may be implemented by a module, circuit or means for implementing the computing device.

Those of skill in the art would further describe the various illustrative logical blocks, configurations, modules, circuits, means, logics and algorithm steps described in connection with the embodiments disclosed herein, in electronic hardware, computer software, or combinations of both. It should be recognized that it can be implemented as To clearly illustrate the interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented in hardware or as software depends on the specific application and design restrictions imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

8 shows a simplified and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented in association with an embodiment of the present disclosure.

While the present disclosure has generally been described above with respect to computer-executable instructions that can be executed on one or more computers, those skilled in the art will appreciate that the present disclosure may be implemented in combination with other program modules and/or as a combination of hardware and software. will be.

Generally, program modules include routines, procedures, programs, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Further, to those skilled in the art, the method of the present disclosure is not limited to single-processor or multiprocessor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable household appliances, etc. (each of which is It will be appreciated that other computer system configurations may be implemented, including one that may operate in connection with one or more associated devices.

The described embodiments of the present disclosure may also be practiced in a distributed computing environment where certain tasks are performed by remote processing devices that are connected through a communication network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

Computers typically include a variety of computer-readable media. Any medium accessible by the computer may be a computer-readable medium, and the computer-readable medium may include a computer-readable storage medium and a computer-readable transmission medium. Such computer-readable storage media include volatile and nonvolatile media, removable and non-removable media. Computer-readable storage media includes volatile and nonvolatile media, removable and non-removable media implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Computer-readable storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage. Devices, or any other medium that can be accessed by a computer and used to store desired information.

Computer-readable transmission media typically implement computer-readable instructions, data structures, program modules or other data on a modulated data signal such as a carrier wave or other transport mechanism. Includes information delivery media. The term modulated data signal refers to a signal in which one or more of the characteristics of the signal is set or changed so as to encode information in the signal. By way of example and not limitation, computer-readable transmission media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above-described media are also intended to be included within the scope of computer-readable transmission media.

An exemplary environment 1100 is shown that implements various aspects of the present disclosure, including a computer 1102, which includes a processing device 1104, a system memory 1106, and a system bus 1108. do. System bus 1108 couples system components, including but not limited to, system memory 1106 to processing device 1104. The processing unit 1104 may be any of a variety of commercially available processors. Dual processors and other multiprocessor architectures may also be used as processing unit 1104.

The system bus 1108 may be any of several types of bus structures that may be additionally interconnected to a memory bus, a peripheral bus, and a local bus using any of a variety of commercial bus architectures. System memory 1106 includes read-only memory (ROM) 1110 and random access memory (RAM) 1112. The basic input/output system (BIOS) is stored in non-volatile memory 1110 such as ROM, EPROM, EEPROM, etc. This BIOS is a basic input/output system that helps transfer information between components in the computer 1102, such as during startup. Includes routines. RAM 1112 may also include high speed RAM such as static RAM for caching data.

The computer 1102 also includes an internal hard disk drive (HDD) 1114 (e.g., EIDE, SATA)-this internal hard disk drive 1114 can also be configured for external use within a suitable chassis (not shown). Yes--, magnetic floppy disk drive (FDD) 1116 (for example, to read from or write to removable diskette 1118), and optical disk drive 1120 (e.g., CD-ROM For reading the disk 1122 or for reading from or writing to other high-capacity optical media such as DVD). The hard disk drive 1114, magnetic disk drive 1116, and optical disk drive 1120 are each connected to the system bus 1108 by a hard disk drive interface 1124, a magnetic disk drive interface 1126, and an optical drive interface 1128. ) Can be connected. The interface 1124 for implementing an external drive includes at least one or both of Universal Serial Bus (USB) and IEEE 1394 interface technologies.

These drives and their associated computer readable media provide non-volatile storage of data, data structures, computer executable instructions, and the like. In the case of computer 1102, drives and media correspond to storing any data in a suitable digital format. Although the description of the computer-readable medium above refers to a removable optical medium such as a HDD, a removable magnetic disk, and a CD or DVD, those skilled in the art may use a zip drive, a magnetic cassette, a flash memory card, a cartridge, etc. It will be appreciated that other types of computer-readable media, such as the ones, may also be used in the exemplary operating environment and that any such media may contain computer-executable instructions for performing the methods of the present disclosure.

A number of program modules, including the operating system 1130, one or more application programs 1132, other program modules 1134, and program data 1136, may be stored in the drive and RAM 1112. All or part of the operating system, applications, modules and/or data may also be cached in RAM 1112. It will be appreciated that the present disclosure may be implemented on several commercially available operating systems or combinations of operating systems.

A user may input commands and information into the computer 1102 through one or more wired/wireless input devices, for example, a pointing device such as a keyboard 1138 and a mouse 1140. Other input devices (not shown) may include a microphone, IR remote control, joystick, game pad, stylus pen, touch screen, and the like. These and other input devices are often connected to the processing unit 1104 through the input device interface 1142, which is connected to the system bus 1108, but the parallel port, IEEE 1394 serial port, game port, USB port, IR interface, It can be connected by other interfaces such as etc.

A monitor 1144 or other type of display device is also connected to the system bus 1108 through an interface such as a video adapter 1146. In addition to the monitor 1144, the computer generally includes other peripheral output devices (not shown) such as speakers, printers, and the like.

Computer 1102 may operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) 1148, via wired and/or wireless communication. The remote computer(s) 1148 may be a workstation, server computer, router, personal computer, portable computer, microprocessor-based entertainment device, peer device, or other common network node, and is generally referred to as computer 1102. Although including many or all of the described components, only memory storage device 1150 is shown for simplicity. The logical connections shown include wired/wireless connections to a local area network (LAN) 1152 and/or to a larger network, eg, a wide area network (WAN) 1154. Such LAN and WAN networking environments are common in offices and companies, and facilitate enterprise-wide computer networks such as intranets, all of which can be connected to worldwide computer networks, for example the Internet.

When used in a LAN networking environment, the computer 1102 is connected to the local network 1152 via a wired and/or wireless communication network interface or adapter 1156. Adapter 1156 may facilitate wired or wireless communication to LAN 1152, which also includes a wireless access point installed therein to communicate with wireless adapter 1156. When used in a WAN networking environment, the computer 1102 may include a modem 1158, connected to a communication server on the WAN 1154, or other Have means. The modem 1158, which may be an internal or external and a wired or wireless device, is connected to the system bus 1108 through a serial port interface 1142. In a networked environment, program modules described for computer 1102 or portions thereof may be stored in remote memory/storage device 1150. It will be appreciated that the network connections shown are exemplary and other means of establishing communication links between computers may be used.

Computer 1102 is associated with any wireless device or entity deployed and operated in wireless communication, e.g., a printer, scanner, desktop and/or portable computer, portable data assistant (PDA), communication satellite, wireless detectable tag. It operates to communicate with any device or place, and a phone. This includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, the communication may be a predefined structure as in a conventional network or may simply be ad hoc communication between at least two devices.

Wi-Fi (Wireless Fidelity) allows you to connect to the Internet, etc. without wires. Wi-Fi is a wireless technology such as a cell phone that allows such devices, for example computers, to transmit and receive data indoors and outdoors, that is, anywhere within the coverage area of a base station. Wi-Fi networks use a wireless technology called IEEE 802.11 (a, b, g, etc.) to provide a secure, reliable and high-speed wireless connection. Wi-Fi can be used to connect computers to each other, to the Internet, and to a wired network (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in unlicensed 2.4 and 5 GHz radio bands, for example, at a data rate of 11 Mbps (802.11a) or 54 Mbps (802.11b), or in products that include both bands (dual band). have.

A person of ordinary skill in the art of the present disclosure includes various exemplary logical blocks, modules, processors, means, circuits and algorithm steps described in connection with the embodiments disclosed herein, electronic hardware, (convenience). For the sake of clarity, it will be appreciated that it may be implemented by various forms of program or design code or a combination of both (referred to herein as "software"). To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those of ordinary skill in the art of the present disclosure may implement the described functions in various ways for each specific application, but such implementation decisions should not be interpreted as being outside the scope of the present disclosure.

The various embodiments presented herein may be implemented as a method, apparatus, or article of manufacture using standard programming and/or engineering techniques. The term “article of manufacture” includes a computer program, carrier, or media accessible from any computer-readable device. For example, computer-readable media include magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., CD, DVD, etc.), smart cards, and flash memory. Devices (eg, EEPROM, card, stick, key drive, etc.). In addition, the various storage media presented herein include one or more devices and/or other machine-readable media for storing information. The term “machine-readable medium” includes, but is not limited to, wireless channels and various other media capable of storing, holding, and/or transmitting instruction(s) and/or data.

It is to be understood that the specific order or layer structure of steps in the presented processes is an example of exemplary approaches. Based on design priorities, it is to be understood that a particular order or layer structure of steps in processes may be rearranged within the scope of the present disclosure. The appended method claims provide elements of the various steps in a sample order, but are not meant to be limited to the specific order or layer structure presented.

A description of the presented embodiments is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be apparent to those of ordinary skill in the art, and general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure is not to be limited to the embodiments presented herein, but is to be construed in the widest scope consistent with the principles and novel features presented herein.

Claims (13)

A computer program stored in a computer-readable storage medium, wherein the computer program, when executed on one or more processors, causes the one or more processors to perform the following operations for providing startup word recognition technology, the operations:
Extracting one or more features corresponding to the voice data by using the voice data of the user received through the user terminal as an input of the voice signal processing module;
Calculating a matching score between the speech data and a predetermined keyword by using the at least one feature as an input of a keyword recognition model; And
Generating a control signal for causing the user terminal to call a voice-based user interface based on the matching score and determining to transmit it to the user terminal;
Containing,
A computer program stored on a computer readable storage medium.
The method of claim 1,
Performing pre-processing on the user's voice data;
Including more,
Pre-processing for the voice data,
To remove noise included in the voice data, which is a preprocessing for removing at least one of white noise with uniform frequency, random frequency noise with irregular frequency, and noise that may occur in a signal processing process from the voice data,
A computer program stored on a computer readable storage medium.
The method of claim 1,
The operation of extracting one or more features corresponding to the voice data by using the voice data of the user received through the user terminal as an input of the voice signal processing module,
Outputting a spectrogram through a fast Fourier transform (STFT) for the voice data;
Obtaining a Mel-Spectrogram through a Mel-Filter Bank for the spectrogram; And
Extracting one or more features by processing the Mel-Spectrogram with a Discrete Cosine Transform (DCT);
Containing,
A computer program stored on a computer readable storage medium.
The method of claim 1,
The one or more features,
An MFCC (Mel Frequency Cepstral Coefficient) feature obtained by feature vectorization of the speech data,
A computer program stored on a computer readable storage medium.
The method of claim 1,
The keyword recognition model,
A model for determining whether the voice data is appropriate as a starting word for calling the voice-based user interface, including a recycling pooling-based convolution network,
A computer program stored on a computer readable storage medium.
The method of claim 1,
The keyword recognition model,
Including one or more convolutional layers for performing a convolutional operation and one or more recycling pooling layers for performing a recycling operation,
A computer program stored on a computer readable storage medium.
The method of claim 6,
The recycling operation,
An operation of including each record included in the pooling window in the pooling feature map of a different channel with respect to the sampling feature map to which the convolution operation is applied,
A computer program stored on a computer readable storage medium.
The method of claim 6,
The convolution layer,
A sampling feature map set is generated through the convolution operation through each of the MFCC feature and one or more convolution filters,
The recycling pooling layer,
Generating a pooling feature map set based on the recycling operation for the sampled feature map set,
A computer program stored on a computer readable storage medium.
The method of claim 8,
The pooling feature map set,
Including a feature map of a size reduced than the size of the feature map of the sampled feature map set, including feature maps greater than or equal to the number of feature maps included in the sampling feature map set,
A computer program stored on a computer readable storage medium.
The method of claim 1,
Training the keyword recognition model through training data;
Including more, and
The training data,
Consisting of learning input data including a plurality of MFCC features and learning result data including a plurality of starting words corresponding to each of the MFCC features,
A computer program stored on a computer readable storage medium.
The method of claim 1,
The voice-based user interface,
A user interface for allowing a control operation related to a game based on the user's voice data,
The control operation related to the game,
Including at least one control operation of a play control operation and an application control operation,
A computer program stored on a computer readable storage medium.
As a computing device for providing a startup word recognition technology,
A processor including one or more cores;
A memory containing program codes executable in the processor; And
A network unit for transmitting and receiving data to and from a user terminal;
Contains, and
The processor,
Extracting one or more features corresponding to the voice data by using the voice data of the user received through the user terminal as an input of the voice signal processing module,
Calculating a matching score between the speech data and a predetermined keyword by using the at least one feature as an input of a keyword recognition model, and
Generating a control signal for causing the user terminal to call a voice-based user interface based on the matching score and determining to transmit it to the user terminal,
A computing device for providing startup word recognition technology.
As a method for providing startup word recognition technology,
Extracting one or more features corresponding to the voice data by using voice data of a user received through a user terminal as an input to a voice signal processing module;
Calculating a matching score between the speech data and a predetermined keyword by using the at least one feature as an input of a keyword recognition model; And
Generating a control signal for causing the user terminal to call a voice-based user interface based on the matching score and determining to transmit it to the user terminal;
Containing,
A method for providing startup word recognition technology.

Images