KR20160055059A - Method and apparatus for speech signal processing - Google Patents

Abstract

A speech signal processing method of a terminal comprises the following steps of: receiving a speech signal; detecting a personalized information section including personal information in the speech signal; performing data processing with respect to a speech signal in correspondence with the personalized information section by using a personalized model generated based on the personal information; and receiving a result of the data processing with respect to the speech signal in correspondence with a general information section, which is a section except the personalized information section, from a server. Therefore, the speech signal processing method can protect personal information while using a personalized model.

Description

TECHNICAL FIELD [0001] The present invention relates to a speech signal processing method,

BACKGROUND OF THE INVENTION 1. Field of the Invention [0002] The present invention relates to a method and apparatus for processing a voice signal, and more particularly, to a method and apparatus for processing a voice signal that can protect personal information while using a personalization model.

Speech recognition is a technique of receiving a user's voice and converting it into a character. This process is also called Automatic Speech Recognition (ASR). Recently, it has been widely spread as an interface technology for replacing keyboard input in a device such as a smart phone or a TV. Natural Language Understanding (NLU) is a technique for extracting the meaning of user utterance from recognition results of speech recognition. It is possible to more accurately grasp the meaning of the voice by analyzing the user's voice at a higher level rather than merely recognizing the user's voice.

This speech recognition and language understanding system can be generally divided into a client for inputting a voice signal and a speech recognition and language understanding engine (ASR / NLU engine) for performing voice recognition and speech understanding from a voice signal, In order to increase the processing speed, the two modules can be designed to be separated from each other. In this case, a device such as a smart phone or a TV having limited processing capability and data storage capability can be configured as a client, and a speech recognition and language understanding engine can be configured as a separate server having high computing capability. Lt; / RTI > A device located close to a user plays a role of receiving a voice signal, and a server having a high data processing speed plays a role of performing speech recognition and language understanding. In another configuration, there may be a configuration in which a voice recognition and language understanding engine is installed in a device other than the server, and two voice recognition and language understanding engines cooperate with each other to perform voice recognition and language understanding.

One method for enhancing the performance of the speech recognition and language understanding system is to collect data for each user and generate a user-specific model. This per-user model is called a personalized model, and this method is called personalized modeling. Since personalization model can generate customized model for a specific individual, it is general that the personalization model has a higher performance than a general model created for an unspecified majority.

However, if personalization modeling is used, the personal information of the user must be used to create a personalization model. In the case where personal information is transmitted and processed, information security problems may occur. In order to solve this problem, , The processing speed may be slowed.

The disclosed embodiment provides a voice signal processing method and apparatus that can protect personal information while using a personalization model.

Specifically, in the disclosed embodiment, a method and apparatus for processing a voice signal for dividing a personalized information section and a general information section into a terminal and a server are provided.

Also, in the disclosed embodiment, a method and apparatus for processing a voice signal processed by a server, the terminal performing the processing again using a personalization model.

In addition, the disclosed embodiments provide a method and apparatus for processing a voice signal using an ID-based personalization model.

According to another aspect of the present invention, there is provided a method of processing a speech signal of a terminal, the method comprising: receiving a speech signal; Detecting a personalized information section including personal information of the voice signal; Performing data processing on a voice signal corresponding to the personalized information section of the voice signal using a personalized model generated based on the personal information; And receiving a result of data processing of a voice signal corresponding to a general information section, which is a section other than the personalized information section, from the server.

The method may further include generating the personalized information section and the voice section information for the general information section and transmitting the generated voice section information to the server.

The voice interval information may include interval marking information marking at least one of the personalized information interval and the general information interval of the voice signal.

The method may further include receiving the personalized information section and the voice section information for the general information section from the server.

The result of the data processing on the audio signal corresponding to the general information section received from the server may be a result of the server processing the audio signal corresponding to the general information section using the general model.

According to another aspect of the present invention, there is provided a method of processing a voice signal of a server, the method comprising: receiving a voice signal; Detecting a personalized information section including personal information of the voice signal; Performing data processing on a speech signal corresponding to a general information section, which is a section of the speech signal other than the personalized information section, using a general model; And transmitting the data processing result of the voice signal corresponding to the general information section to the terminal.

The method may further include generating and transmitting the personalized information section and the voice section information for the general information section to the terminal.

The voice interval information may include interval marking information marking at least one of the personalized information interval and the general information interval of the voice signal.

The method may further include receiving the personalized information section and the voice section information for the general information section from the terminal.

According to another aspect of the present invention, there is provided a method of processing a voice signal of a terminal, the method comprising: receiving a voice signal; Receiving a result of data processing of the voice signal from a server using a general model; And data processing using the personalization model generated based on the personal information about the voice signal and the data processing result.

The step of performing data processing with respect to the voice signal using the personalization model generated based on the data processing result and the personal information may include a step of performing data processing on the voice signal corresponding to the personalized information section including the personal information, And performing the steps of:

The method may further include detecting the personalized information section of the voice signal.

The method may further include receiving the personalization information section and the voice section information for the general information section from the server.

The voice interval information may include interval marking information marking at least one of the personalized information interval and the general information interval of the voice signal.

In addition, the personalization model may be at least one of a personalized speech recognition model, a personalized natural language understanding model, and a personalized lexical model.

According to another aspect of the present invention, there is provided a method for processing a voice signal of a terminal, the method comprising the steps of: mapping an ID to personal information to generate a mapping table; Generating an ID-based personalization model using the mapping table; Transmitting the ID-based personalization model to a server; Receiving a result of data processing of the voice signal from the server using the ID-based personalization model; And restoring the personal information corresponding to the ID using the data processing result and the mapping table.

The step of generating an ID-based personalization model using the mapping table may include the step of representing an ID mapped to the personal information as an acoustic unit ID that is an ID mapped to the sound .

The acoustic unit ID may be an ID mapped to the sound in agreement with the server.

The method may further include generating the mapping table by mapping an ID to the additional information generated from the personal information.

According to another aspect of the present invention, there is provided a method of processing a voice signal of a server, comprising: receiving an ID-based personalization model from a terminal; Receiving a voice signal; Performing data processing on the voice signal using the ID-based personalization model; And transmitting the data processing result to the terminal.

The step of performing data processing on the voice signal using the ID-based personalization model may include the step of indicating an ID mapped to the personal information using the sound unit ID that is an ID mapped to the sound according to the agreement with the terminal .

The terminal according to the disclosed embodiments includes: a receiver for receiving a voice; A communication unit for communicating with the server; And a personalization information section that includes personal information of the voice signal and detects a voice signal corresponding to the personalized information section of the voice signal using a personalization model generated based on the personal information And a control unit for controlling the data processing of the voice signal corresponding to the general information section, which is a section other than the personalized information section, from the server.

In addition, the controller may generate voice section information for the personalized information section and the general information section and transmit the generated voice section information to the server.

The voice interval information may include interval marking information in which at least one of the personalized information interval and the general information interval of the voice signal is marked.

Also, the control unit may control the personalization information section and the voice section information on the general information section to be received from the server.

The result of the data processing on the audio signal corresponding to the general information section received from the server may be a result of the server processing the audio signal corresponding to the general information section using the general model.

The server according to the disclosed embodiment includes: a receiver for receiving a voice; A communication unit for performing communication with the terminal; And a controller for receiving a voice signal and detecting a personalization information section including personal information of the voice signal, and using a general model for a voice signal corresponding to a general information section, which is a section other than the personalized information section, And a control unit for controlling the data processing unit to transmit the data processing result of the voice signal corresponding to the general information section to the terminal.

Also, the controller may generate voice section information for the personalized information section and the general information section and transmit the generated voice section information to the terminal.

The voice interval information may include interval marking information in which at least one of the personalized information interval and the general information interval of the voice signal is marked.

In addition, the controller may control the personalization information section and the voice section information for the general information section to be received from the terminal.

According to another aspect of the present invention, there is provided a terminal comprising: a communication unit for performing communication with a server; And a control unit for receiving a result of data processing of the voice signal from the server using a general model and performing data processing on the voice signal using a personalization model generated based on the data processing result and the personal information, .

In addition, when the data processing is performed on the voice signal using the personalization model generated based on the data processing result and the personal information, the control unit may perform a process on the voice signal corresponding to the personalized information section including the personal information Data processing can be performed.

In addition, the control unit may control to detect the personalized information section of the voice signal.

In addition, the controller may control the personalization information section and the voice section information for the general information section to be received from the server.

The voice interval information may include interval marking information in which at least one of the personalized information interval and the general information interval of the voice signal is marked.

The personalization model may be at least one of a personalized speech recognition model, a language understanding model, and a personalized lexical model.

According to another aspect of the present invention, there is provided a terminal including: a receiver for receiving a voice signal; a communication unit for communicating with a server; And generating a personalization model based on the ID using the mapping table, transmitting the ID-based personalization model to the server, and transmitting the personalization model to the server using the mapping table, And a controller for receiving a result of the data processing using the ID-based personalization model and restoring the personal information corresponding to the ID using the data processing result and the mapping table.

In addition, when the ID-based personalization model is created using the mapping table, the controller may control the ID mapped to the personal information to be expressed as an ID mapped to the sound.

The acoustic unit ID may be an ID mapped to the sound in agreement with the server.

The control unit may control the generation of the mapping table by mapping an ID to the additional information generated from the personal information.

According to another embodiment of the present invention, there is provided a server comprising: a receiver for receiving a voice signal; A communication unit for performing communication with the terminal; And an ID-based personalization model from the terminal, receives the voice signal, processes the voice signal using the ID-based personalization model, and controls the data processing result to be transmitted to the terminal .

In addition, the control unit may control to display the ID mapped to the personal information by using the sound unit ID which is the ID mapped to the sound according to the agreement with the terminal.

1 is a block diagram illustrating an internal configuration of a UE according to an embodiment of the present invention.
2 is a block diagram showing an internal configuration of a server according to the disclosed embodiment.
3 is a block diagram showing the internal configuration of the terminal shown in FIG. 1 in more detail.
4 is a block diagram showing the internal configuration of the server shown in Fig. 2 in more detail.
5 is a flowchart illustrating a voice processing method of a terminal according to the disclosed embodiment.
6 is a flowchart showing a voice processing method of a server according to the disclosed embodiment.
7 is a view for explaining a personalization information section and a general information section.
8 is a flowchart illustrating an exemplary operation procedure of a terminal and a server according to the disclosed embodiment.
9 is a flowchart showing a voice processing method of a terminal according to another disclosed embodiment.
10 is a flowchart showing a voice processing method of a server according to another disclosed embodiment.
11 is a flowchart illustrating an exemplary operation procedure of a terminal and a server according to another disclosed embodiment.
12 is a block diagram illustrating an internal configuration of a terminal according to another disclosed embodiment.
FIG. 13 is a block diagram showing an internal configuration of a server according to another disclosed embodiment in more detail.
FIG. 14 is a block diagram showing the internal configuration of the terminal shown in FIG. 12 in more detail.
15 is a block diagram showing the internal configuration of the terminal shown in FIG. 13 in more detail.
16 is a flowchart showing a voice processing method of a terminal according to another disclosed embodiment.
17 is a flowchart showing a voice processing method of a server according to another disclosed embodiment.
18 is a diagram showing personal information.
FIG. 19 is a diagram showing personal information by phonetic symbols. FIG.
20 is a diagram showing a mapping table in which personal information is mapped to an ID.
21 is a diagram showing a mapping table in which pronunciation symbols of personal information are mapped to IDs.
FIG. 22 is a diagram showing the personal information ID by the pronunciation symbol ID. FIG.
23 is a flowchart showing an example of a specific operation procedure of a terminal and a server according to another disclosed embodiment.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the disclosed embodiments, and how to accomplish them, will become apparent with reference to the embodiments described below with reference to the accompanying drawings. It should be understood, however, that the disclosed embodiments are not limited to the embodiments disclosed herein but are to be embodied in different forms and should not be construed as limited to the specific embodiments set forth herein, It is to be understood that the present invention is not limited to the above-described embodiments, but may be modified and changed without departing from the scope of the present invention. Like reference numerals refer to like elements throughout the specification.

Throughout the specification, when a part is referred to as being "connected" to another part, it includes not only "directly connected" but also "electrically connected" with another part in between . Also, when an element is referred to as "including" an element throughout the specification, it is to be understood that the element may include other elements, Also, as used herein, the term "part " refers to a hardware component such as software, FPGA or ASIC, and" part " However, 'minus' is not limited to software or hardware. The " part " may be configured to reside on an addressable storage medium and may be configured to play back one or more processors. Thus, by way of example, and not limitation, "part (s) " refers to components such as software components, object oriented software components, class components and task components, and processes, Subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. The functions provided in the components and "parts " may be combined into a smaller number of components and" parts " or further separated into additional components and "parts ".

BRIEF DESCRIPTION OF THE DRAWINGS The above and other objects, features and advantages of the present invention will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which: FIG. However, the disclosed embodiments may be embodied in many different forms and are not limited to the embodiments described herein. In order to clearly illustrate the embodiments disclosed in the drawings, portions not related to the description are omitted.

Although the terms used in the disclosed embodiments have been chosen to take into account the functionality in the disclosed embodiments, the presently widely used generic terms have been chosen and may vary depending on the intent or circumstance of the skilled artisan, the emergence of new technology, and the like. Also, in certain cases, there may be a term selected arbitrarily by the applicant, in which case the meaning thereof will be described in detail in the description of the corresponding invention. Thus, the terms used in the disclosed embodiments should not be construed as merely descriptive terms, but rather should be defined on the basis of the meaning of the terms and throughout the description of the disclosed embodiments.

In this specification, a personalized information section refers to a section including personal information that can directly or indirectly identify each individual of the voice signals. For example, a section including a name stored in the telephone directory of the user terminal, a search history of the user, user location information, and the like in the voice signal may correspond to the personalized information section.

In this specification, a general information section is a section including general information, not personal information, of a speech signal, and means a section excluding a personalized information section. For example, a section including a sentence having a general meaning such as 'Call me' or 'Did you eat?' May correspond to a general information section.

In this specification, a personalized model is a speech processing model that reflects individual characteristics, and is a customized speech processing model for a specific individual.

In this specification, a general model is a general speech processing model, and is a speech processing model for speech processing of an unspecified person rather than a specific individual.

1 is a block diagram illustrating an internal configuration of a UE according to an embodiment of the present invention.

Referring to FIG. 1, a terminal 100 according to an embodiment includes a receiving unit 110, a communication unit 130, and a controller 150.

The receiving unit 110 plays a role of receiving a voice signal. The receiving unit 110 may include various components such as a microphone unit, a USB interface unit, and a DVD interface unit. For example, when the receiving unit 110 includes a microphone unit, the terminal 100 can directly receive a user voice signal through the microphone unit. Also, when the receiving unit 110 includes the USB interface unit, the terminal 100 may receive the voice signal file from the USB. Furthermore, when receiving a voice signal from an external device through the communication unit 130, the communication unit 130 may serve as the receiving unit 110.

The communication unit 130 performs communication with the external device. The communication unit 130 may be connected to the network by wire or wireless, and may perform communication with an external device. According to the disclosed embodiment, the communication unit 130 communicates with the server and can transmit and receive data. For example, the communication unit 130 may include a short-range communication module, a mobile communication module, a wireless Internet module, and a wired Internet module. Further, the communication unit 130 may include one or more components.

The control unit 150 controls the operation of the entire terminal 100 and can process the voice signal by controlling the receiving unit 110 and the communication unit 130. The controller 150 stores signals or data input from the outside of the terminal 100, a RAM used as a storage area corresponding to various jobs performed in the electronic device, a ROM (ROM) storing a control program for controlling the peripheral devices, And a processor. The processor may be implemented as a SoC (System On Chip) incorporating a core (not shown) and a GPU (not shown). A processor may also include a plurality of processors.

The control unit 150 according to the disclosed embodiment receives a voice signal through the receiving unit 110, detects a personalized information section including personal information in the received voice signal, And controls the communication unit 130 to receive the data processing result of the voice signal corresponding to the general information section, which is a section other than the personalized information section, from the server. At this time, the data processing result of the voice signal corresponding to the general information section received from the server may be a result of processing the voice signal corresponding to the general information section by using the general model of the server. Since the server has a high computation capability, it is possible to process data in a fast manner over a general information section.

In addition, the control unit 150 may generate the personalized information section and the voice section information for the general information section through the communication section 130 and may transmit the personalized information section and the general information section to the server, It may be controlled to receive the section information. Here, the voice interval information may include interval marking information that marks at least one interval of the personalized information interval and the general information interval in the voice signal.

Furthermore, the control unit 150 can control the output of the voice processing result to the user.

According to the disclosed embodiment, the terminal 100 performs data processing on a voice signal corresponding to a personalized information section using a personalization model generated based on personal information, and transmits the personalized information model corresponding to the general information section, which is a section other than the personalized information section And a result of data processing by the server is received for the voice signal. Since the server performs data processing only for the general information section, the personalization model is not used in the data processing process. Therefore, no type of information transmission including personal information occurs between the server and the terminal 100, so that no personal information is stored in the server. As a result, according to the disclosed embodiment, a voice processing system having higher performance and higher processing speed can be implemented while originally protecting personal information.

The control unit 150 according to another embodiment of the present invention receives a voice signal through the receiving unit 110 and receives a result of data processing using the general model for the voice signal from the server through the communication unit 130, And controls the data processing using the personalization model generated based on the data processing result and the personal information. In this case, since the server 200 has a high computation capability, it is possible to process data in a short period of time. In addition, the personalization model may be at least one of a personalized speech recognition model, a language understanding model, and a personalized lexical model.

In addition, when the data processing is performed using the personalization model generated based on the data processing result and the personal information with respect to the voice signal, the control unit 150 controls the data processing unit 150 to process the voice signal corresponding to the personalized information section including the personal information, Processing can be performed. At this time, the controller 150 may control to detect the personalization information section of the voice signal, or may control the personalization information section and the voice section information about the general information section from the server through the communication section 130. Here, the voice interval information may include interval marking information that marks at least one interval of the personalized information interval and the general information interval in the voice signal.

According to the disclosed embodiment, the terminal 100 receives a result of data processing of a speech signal using a general model from a server, and generates a personalization model based on the data processing result and personal information with respect to the received speech signal And performs data processing once again. That is, since voice processing using personalization modeling is performed only in the terminal 100, no information transmission including personal information occurs between the server and the terminal 100, and thus no personal information is stored in the server. As a result, according to the disclosed embodiment, a voice processing system having higher performance and higher processing speed can be implemented while originally protecting personal information.

2 is a block diagram showing an internal configuration of a server according to the disclosed embodiment.

Referring to FIG. 2, the server 200 according to the disclosed embodiment includes a receiving unit 210, a communication unit 230, and a control unit 250.

The receiving unit 210 plays a role of receiving a voice signal. The receiving unit 210 may include a component that can receive a voice signal in various forms, such as a USB interface unit and a DVD interface unit. For example, when the receiving unit 210 includes a USB interface, the server 200 can receive a voice signal file from the USB. Further, when receiving a voice signal from an external device through the communication unit 230, the communication unit 230 may also serve as the receiving unit 210.

The communication unit 230 performs communication with the external device. The communication unit 230 may be connected to the network by wire or wirelessly to perform communication with an external device. According to the disclosed embodiment, the communication unit 230 can communicate with the terminal 100 and transmit / receive data. According to the disclosed embodiment, the communication unit 130 communicates with the server and can transmit and receive data. For example, the communication unit 130 may include a short-range communication module, a mobile communication module, a wireless Internet module, and a wired Internet module. Further, the communication unit 130 may include one or more components.

The control unit 250 controls the operation of the entire server 200 and can process the voice signal by controlling the receiving unit 210 and the communication unit 230. The controller 250 stores signals or data input from the outside of the server 200, a RAM used as a storage area corresponding to various jobs performed in the electronic device, a ROM (ROM) storing a control program for controlling peripheral devices, And a processor. The processor may be implemented as a SoC (System On Chip) incorporating a core (not shown) and a GPU (not shown). A processor may also include a plurality of processors.

The control unit 250 according to the disclosed embodiment receives a voice signal through the receiving unit 210, detects a personalized information section including personal information in the voice signal, and detects a personal information section including a general information section And controls the communication unit 230 to transmit the result of the data processing to the voice signal corresponding to the general information section to the terminal 100. [

The control unit 250 may generate the personalized information section and the voice section information for the general information section through the communication section 230 and transmit the personalized information section and the general information section to the terminal 100 or may transmit the personalized information section and the general information section, And receives voice section information. Here, the voice interval information may include interval marking information that marks at least one interval of the personalized information interval and the general information interval in the voice signal.

According to the disclosed embodiment, the server 200 does not use the personalization model 171 for data processing because it performs data processing only on voice signals corresponding to the general information section. Therefore, no type of information transmission including personal information occurs between the server and the terminal 100, so that no personal information is stored in the server. As a result, according to the disclosed embodiment, a voice processing system having higher performance and higher processing speed can be implemented while originally protecting personal information.

The control unit 250 according to another embodiment of the present invention receives the voice signal through the receiving unit 210 and transmits the voice signal corresponding to the general information section, which is a section other than the personalization information section, And transmits the data processing result to the terminal through the communication unit 230. The controller 250 can generate voice section information on the personalized voice section of the voice signal and transmit the generated voice section information to the terminal 100. Here, the voice interval information may include interval marking information that marks at least one interval of the personalized information interval and the general information interval in the voice signal.

According to the disclosed embodiment, the server 200 does not use the personalization model 171 in the data processing process because it performs data processing only on the general model for the speech signal received. Therefore, no type of information transmission including personal information occurs between the server and the terminal 100, so that no personal information is stored in the server. As a result, according to the disclosed embodiment, a voice processing system having higher performance and higher processing speed can be implemented while originally protecting personal information.

3 is a block diagram showing the internal configuration of the terminal shown in FIG. 1 in more detail. In FIG. 3, the description of the components that are the same as those in FIG. 1 will be omitted.

The control unit 150 may include a voice processing engine 151. According to the disclosed embodiment, the speech processing engine 151 may include a speech recognition engine (ASR engine) and a language understanding engine (NLU engine), and performs data processing on the received speech signal to perform speech recognition and language understanding . At this time, the speech recognition engine and the language understanding engine can process the speech signal using the speech recognition model and the language understanding model, respectively.

The speech recognition model may include an acoustic model and a language model. The acoustic model is a model for speech signals, and is generated by statistical methods from a large amount of collected voice data. The language model is a grammatical model of user utterance, and it is common to obtain statistical learning from a large amount of text data that is also collected. The language understanding model is a semantic model that represents the meaning of user utterance. It is obtained by statistical learning from a large amount of text data or by creating a semantic understanding rule in consideration of a usage scenario.

The terminal 100 may further include a storage unit 170. The storage unit 170 stores programs and data necessary for the operation of the terminal 100. The storage unit 170 may be a volatile storage medium or a nonvolatile storage medium and may be a combination of both storage media. The volatile storage medium may include a semiconductor memory such as a RAM, a DRAM, and a SRAM. The nonvolatile storage medium may include a hard disk and a flash NAND memory.

According to the disclosed embodiment, the personal information 172 may be stored in the storage unit 170. The personal information 172 is information that can directly or indirectly identify each individual, and the type of data stored according to the type of the terminal can be changed. For example, in the case of a mobile device, it may include a contact, a music list, a short message content, a reception history, a call history, and a web search history. In the case of a TV, a personal playlist may be included.

In addition, the personalization model 171 may be stored in the storage unit 170. The personalization model 171 is a speech processing model that reflects personal characteristics generated using personal information. The storage unit 170 may store a personalized speech recognition model and / or a personalized language understanding model. When such a personalized speech recognition model and / or a personalized language understanding model is used, a speech processing system having higher performance can be implemented.

4 is a block diagram showing the internal configuration of the server shown in Fig. 2 in more detail. In Fig. 4, the description of the components that are the same as those in Fig. 2 will be omitted.

The control unit 250 may include a speech processing engine 251. According to the disclosed embodiment, the speech processing engine 151 may include a speech recognition engine (ASR engine) and a language understanding engine (NLU engine), and performs data processing on the received speech signal to perform speech recognition and language understanding . At this time, the speech recognition engine and the language understanding engine can process the speech signal using the speech recognition model and the language understanding model, respectively.

The server 200 may further include a storage unit 270. The storage unit 270 stores programs and data necessary for the operation of the server 200. The storage unit 270 may be a volatile storage medium or a nonvolatile storage medium, or may be a combination of both storage media. The volatile storage medium may include a semiconductor memory such as a RAM, a DRAM, and a SRAM. The nonvolatile storage medium may include a hard disk and a flash NAND memory. The general model 271 may be stored in the storage unit 270 according to the disclosed embodiment. The general model 271 is a general speech processing model, and is a speech processing model for speech processing of an unspecified person who is not a specific individual. The generic model 271, which consists of a large capacity, combines with the high computing power of the server to provide high voice processing performance for various language expressions (vocabulary) of the user. Therefore, according to the disclosed embodiment, since no type of personal information is stored in the storage unit 270, it is possible to provide high voice processing performance while originally protecting personal information.

Operations of the terminal 100 and the server 200 will be described in more detail below.

5 is a flowchart illustrating a method of processing a voice signal of a terminal according to the disclosed embodiment.

First, in step 510, the terminal 100 receives a voice signal. The terminal 100 may receive voice signals through various components. Receiving a voice signal through a microphone unit may be the most general form, but it may also receive a voice signal through a USB interface unit or a DVD interface unit. Further, it is also possible to receive a voice signal through communication with an external device. According to the disclosed embodiment, the terminal 100 can transmit the voice signal thus received to the server 200. [

Then, in step 520, the terminal 100 detects a personalization information section including personal information in the voice signal. The personalization information section refers to a section including personal information that can directly or indirectly identify each individual of the voice signals. For example, the terminal 100 refers to the personal information 172 stored in the storage unit 170 of FIG. 3 and includes the name stored in the phone book of the user terminal, the search history of the user, the user location information, Can be detected as a personalized information section. Will be described with reference to FIG.

7 is a view for explaining a personalization information section and a general information section.

Referring to FIG. 7, the sentence 'Let's call Kim Gil-dong on the 10th floor of Hong Gil-dong' can be divided into nine sections. That is, it is divided into Hong Gil Dong 701, Seo 702, 10 703, Layer 704, Kim Gil Dong 705, Se 706, Phone 708, . Classification criteria for these segments may be applied differently depending on the situation.

Herein, assuming that Hong Kil-Dong 701 is a word referring to a user and Kim Gil-Dong 705 is a name stored in a telephone directory of a user terminal, a voice signal section corresponding to Hong Gil-Dong 701 and Kim Gil- (710). In other words, Hong Gil Dong 701 and Kim Gil Dong 705 correspond to personal information, and sections 701 and 705 including such personal information correspond to the personalization information section 710. In step 520, the terminal 100 detects a personalized information section 710 including personal information among the voice signals.

The terminal 100 can detect the personalized information section through various methods. For example, a word not included in the dictionary may be determined to be personal information, and a voice section including the word may be determined to be a personalized information section. However, this method is merely an example, and various personalized information section detection methods can be used.

Also, the terminal 100 may generate personalized information section and voice section information for the general information section using the personalized information section detected as described above, and transmit the personalized information section and the voice section information to the server 200. Here, the voice interval information may include interval marking information that marks at least one interval of the personalized information interval and the general information interval in the voice signal. That is, the terminal 100 marks the personalized information section and / or the general information section of the voice signal to be a corresponding section, and generates and transmits the voice section information including the section marking information. The voice section information transmitted to the server 200 may be used by the server 200 to confirm and / or process the general information section of the voice signal. The server 200 can check the remaining sections excluding the section marked with the general information section or the section marked with the personalized information section and process data corresponding to the corresponding section.

Returning to the description of FIG. 5, in step 530, the terminal 100 performs data processing on the voice signal corresponding to the personalization information section using the personalization model 171 generated based on the personal information. The personalization model 171 is a speech processing model reflecting personal characteristics, and is a customized speech processing model for a specific individual. When the personalization model 171 is used to perform data processing on a voice signal, it is possible to perform more accurate voice processing than the general model 271 created for an unspecified number of persons. In addition, the terminal 100 performs data processing directly on the voice signal corresponding to the personalization section using the personalization model 171, so that the personalization model 171 including the personal information needs to be transmitted to an external device such as a server So that the personal information can be originally protected.

In step 540, the terminal 100 receives a result of the data processing of the voice signal corresponding to the general information section, which is a section other than the personalization information section, from the server 200. [ The general information section is a section that includes general information, not personal information, of a voice signal, and the remaining section excluding the personalized information section. For example, a section including a sentence having a general meaning such as 'Call me' or 'Did you eat?' May correspond to a general information section. Will be described with reference to FIG.

As shown above, in FIG. 7, the sentence 'I would like to call Mr. Kim Gil-dong on the 10th floor of Hong Kil-dong' can be divided into 9 sections.

Herein, the words 702, 10 703, 704, 706, 707, 708, 709 are words that are generic words, , 703, 704, 706, 707, 708 and 709 correspond to the general information section 720.

The terminal 100 does not directly process the voice signal corresponding to the general information section 720 including the general information among the voice signals in step 540, 200). Therefore, it is possible to perform fast data processing on the audio signal corresponding to the general information section 720.

The result of data processing of the audio signal corresponding to the general information section received from the server 200 may be a result of the server 200 processing the audio signal corresponding to the general information section using the general model 271 . The general model 271 can provide high speech processing performance for various language expressions (vocabulary) of the user in combination with the high computing power of the server 200. In addition, since the server 200 has a high computation capability, it is possible to perform fast data processing on a general information section.

The terminal 100 may receive the personalization information section from the server 200 and the voice section information on the general information section. Here, the voice interval information may include interval marking information that marks at least one interval of the personalized information interval and the general information interval in the voice signal. The terminal 100 may detect the personalization information section itself and generate the voice section information. Since the terminal 100 and the server 200 can perform data processing on the personalized information section and / or the general information section which are respectively detected by the terminal 100 and the server 200, the personalized information section detected by the terminal 100 and the personalized information section detected by the server 200, If the detected personalization information sections are different from each other, a period of time for data processing may occur. Accordingly, the terminal 100 and the server 200 may share the personalization information section and the voice section information for the general information section in order to perform data processing for all sections without fail.

Furthermore, the terminal 100 can output the data processing result to the user.

As a result, according to the disclosed embodiment, the personal information 172 and the personalization model 171 are stored in the terminal 100 without being transmitted to the server 200, and are stored in the terminal 100 by the personal information 172 or the personalization model 171), it is possible to implement a voice processing system having higher performance and processing speed while originally protecting personal information.

6 is a flowchart illustrating a method of processing a voice signal of a server according to an embodiment of the present invention.

First, in step 610, the server 200 receives a voice signal. The server 200 can receive voice signals through various components. Receiving the voice signal from the terminal 100 may be the most general form, but it may also receive the voice signal through the USB interface unit or the DVD interface unit.

Thereafter, in step 620, the server 200 detects a personalization information section including personal information among the received voice signals. The server 200 may detect the personalized information section by analyzing the received voice signal or may detect the personalized information section by receiving and parsing the voice section information from the terminal 100. [ The server 200 can detect the personalized information section through various methods. For example, a word not included in the dictionary may be determined to be personal information, and a voice section including the word may be determined to be a personalized information section. However, this method is merely an example, and various personalized information section detection methods can be used.

The personalization information section and the general section are described above with reference to FIG. 7, and will not be described in duplicate.

In addition, the server 200 may generate the personalization information section and the voice section information for the general information section using the personalized information section detected as described above, and transmit the personalized information section and the voice section information to the terminal 100. Here, the voice interval information may include interval marking information that marks at least one interval of the personalized information interval and the general information interval in the voice signal. That is, the server 200 may perform marking of the personalized information section and / or the general information section of the voice signal to indicate that the corresponding section is generated, and may generate and transmit the voice section information including the section marking information. The voice section information transmitted to the terminal 100 in this way can be used by the terminal 100 to identify and / or process the personalized information section of the voice signal. The terminal 100 can check the remaining sections excluding the section marked with the general information section or the section marked with the personalized information section and process the voice signal corresponding thereto.

Next, in step 630, the server 200 processes the voice signal corresponding to the general information section, which is a section other than the personalization information section, of the voice signal using the general model 271. The generic model 271 can combine with the high computing power of the server to provide high voice processing performance for the user's various language expressions (vocabulary). Further, since the server 200 has a high computing capability, it is possible to process data in a short period of time. As for the voice signal corresponding to the personalization section including the personal information, the terminal 200 having the personalization model 171 performs the data processing without processing by the server 200, so that the personal information is protected originally can do.

In addition, in step 640, the server 200 transmits to the terminal a result of the data processing on the voice signal corresponding to the general information section.

The server 200 may receive the personalization information section from the terminal 100 and the voice section information on the general information section. Here, the voice interval information may include interval marking information that marks at least one interval of the personalized information interval and the general information interval in the voice signal. The server 200 may detect the personalization information section itself and generate the voice section information. Since the terminal 100 and the server 200 can perform data processing on the personalized information section and / or the general information section which are respectively detected by the terminal 100 and the server 200, the personalized information section detected by the terminal 100 and the personalized information section detected by the server 200, If the detected personalization information sections are different from each other, a period of time for data processing may occur. Accordingly, the terminal 100 and the server 200 may share the personalization information section and the voice section information for the general information section in order to perform data processing for all sections without fail.

As a result, according to the disclosed embodiment, a voice processing system having higher performance and higher processing speed can be implemented while originally protecting personal information.

8 is a flowchart illustrating an exemplary operation procedure of a terminal and a server according to the disclosed embodiment.

First, in step 805, the terminal 100 receives a voice signal. As described above, the terminal 100 can receive voice signals through various components. Receiving a voice signal through a microphone unit may be the most general form, but it may also receive a voice signal through a USB interface unit or a DVD interface unit. Further, it is also possible to receive a voice signal through communication with an external device. Thereafter, the terminal 100 may transmit the received voice signal to the server 200 in step 810.

The terminal 100 and the server 200 detect personalized voice intervals for voice signals in steps 815 and 820, respectively. The terminal 100 and the server 200 can detect the personalized information section through various methods. For example, a word not included in the dictionary may be determined to be personal information, and a voice section including the word may be determined to be a personalized information section. However, this method is merely an example, and various personalized information section detection methods can be used.

The terminal 100 proceeds to step 825 to process the voice data based on the personalization model 171 for the personalized voice interval detected in step 815 and proceeds to step 830 for the general information interval other than the personalized voice interval, It is possible to pass the processing and generate the voice interval information. The server 200 proceeds to step 835 for the personalized voice interval detected in step 820 and passes the data processing to generate voice interval information and proceeds to step 830 for a general information interval other than the personalized voice interval And processes the voice data based on the general model 271.

Through this process, the terminal 100 performs data processing directly on the voice signal corresponding to the personalization section using the personalization model 171, and the server 200 performs data processing corresponding to the general information section Data processing is performed on the voice signal using the general model 271, so that high voice processing performance can be realized while originally protecting personal information.

After that, the terminal 100 and the server 200 share the voice interval information and the data processing result in steps 845 and 850. [ That is, the server 200 transmits the voice data processing result and the voice interval information to the AT 100 in step 845, and the AT 100 transmits the voice interval information to the server 200. 8, steps 845 and 850 are performed. However, either the terminal 100 or the server 200 may transmit data in the sharing process.

As described above, the terminal 100 and the server 200 may respectively detect the personalized information section and generate the voice section information. Since the terminal 100 and the server 200 can perform data processing on the personalized information section and / or the general information section which are respectively detected by the terminal 100 and the server 200, the personalized information section detected by the terminal 100 and the personalized information section detected by the server 200, If the detected personalization information sections are different from each other, a period of time for data processing may occur. Accordingly, the terminal 100 and the server 200 may share the personalization information section and the voice section information for the general information section in order to perform data processing for all sections without fail.

In step 855, the terminal 100 determines whether data processing has been completed for all voice signal intervals. If the data processing is completed, the terminal 100 ends the operation. If the data processing is not completed in step 855, the terminal 100 proceeds to step 865. In step 865, the terminal 100 transmits the data Can be processed. In step 860, the server 200 determines whether data processing is completed for all voice signal intervals. If the data processing is completed, the server 200 ends the operation. If the data processing is not completed in step 860, the server 200 proceeds to step 870. In step 870, the server 200 determines whether or not data processing is performed on the voice section that has not been processed based on the voice section information received from the terminal 100, using the personalization model 171 Can be processed.

8, the voice signal processing processes of steps 815 to 870 may be performed simultaneously in parallel in the terminal 100 and the server 200, or alternatively in the terminal 100 and the server 200, It is also possible to perform voice processing.

When the terminal 100 and the server 200 alternately switch and perform voice processing, the terminal 100 performs data processing based on the personalization model 171 on the voice signal corresponding to the personalized information section, When the audio signal corresponding to the information section comes out and the audio section information is transmitted to the server 200, the server 200 extracts the general model 271 from the audio signal following the audio signal processed by the terminal 100 It is also possible to perform the voice processing. Then, when a voice signal corresponding to the personalized information section again appears, the terminal 100 transmits the data processing result up to the present and the voice section information to the terminal 100, The voice processing may be performed using the general model 271 from the next voice signal.

Conversely, when the server 200 starts voice processing, the server 200 performs voice processing on the voice signal corresponding to the general section using the general model 271, and outputs voice signals corresponding to the personalized information section The terminal 100 transmits the general model 271 from the audio signal following the audio signal processed by the server 200 to the terminal 100. In this case, It is also possible to perform the voice processing.

According to the disclosed embodiments, it is possible to implement a voice processing system with higher performance and higher processing speed while at the same time protecting private information.

9 is a flowchart showing a voice processing method of a terminal according to another disclosed embodiment.

First, in step 910, the terminal 100 receives a voice signal. The terminal 100 may receive voice signals through various components. Receiving a voice signal through a microphone unit may be the most general form, but it may also receive a voice signal through a USB interface unit or a DVD interface unit. Further, it is also possible to receive a voice signal through communication with an external device. According to the disclosed embodiment, the terminal 100 can transmit the voice signal thus received to the server 200. [

Thereafter, in step 920, the terminal 100 receives a result of data processing using the general model 271 with respect to the voice signal from the server 200. [ That is, the terminal 100 receives a result of performing data processing on a voice signal independently of the terminal 100 from the server 200. [ The general model 271 can provide high speech processing performance for various language expressions (vocabulary) of the user in combination with the high computing power of the server 200. Further, since the server 200 has a high computing capability, it is possible to process data in a short period of time.

Next, in step 930, the terminal 100 processes the voice signal using the personalization model 171 generated based on the personal information and the data processing result received from the server 200. According to the disclosed embodiment, the terminal 100 can perform data processing on all or part of the voice signal again using the personalization model 171 and the data processing result from the server 200 with respect to the voice signal. As described above, the server 200 performs data processing using the general model 271, and performs data processing once again using the personalization model 171 for higher voice processing performance.

In this case, the terminal 100 can perform data processing on a voice signal corresponding to a personalized information section including personal information. That is, the server 200 performs data processing using the generic model 271 with respect to the entire voice signal without discriminating between the general information section and the personalized information section by using the high computing power. Thereafter, the personalization information section that can increase the processing performance by using the personal information 172 is once again subjected to the data processing using the personalization model in the terminal 100. [ For this, the terminal 100 can detect the personalization information section of the voice signal. The terminal 100 can detect the personalized information section through various methods. For example, a word not included in the dictionary may be determined to be personal information, and a voice section including the word may be determined to be a personalized information section. However, this method is merely an example, and various personalized information section detection methods can be used.

Alternatively, the terminal 100 may receive the personalization information section from the server 200 and the voice section information on the general information section. By receiving the voice interval information generated by the server 200 having a high computing capability, the data processing burden on the terminal 100 can be reduced and the overall voice processing speed can be increased. Here, the voice interval information may include interval marking information that marks at least one interval of the personalized information interval and the general information interval in the voice signal.

The personalized information section and the general information section are described above with reference to FIG. 7, but will not be described in duplicate.

Here, the personalization model 171 may be at least one of a personalized speech recognition model, a personalized natural language understanding model, and a personalized lexical model.

When the terminal 100 performs data processing on the voice signal using the personalized speech recognition model, the terminal 100 can receive the data processing result of the phoneme, pseudo-morpheme, or word unit from the server 200, It can receive and process multi-pass processing results such as N-best hypothesis, lattice, and confusion network for data processing.

When the terminal 100 performs data processing on the voice signal using the language understanding model, the terminal 100 receives the data processing result of phonemes, pseudo-morphemes, or words from the server 200 and the reliability of the sentence or word unit Information such as a score (confidence score) can be received and used for data processing. In addition, a multi-pass processing result may be received from the server 200 and used for data processing.

When the terminal 100 performs data processing on a voice signal using a personalized lexical model, the terminal 100 receives a result of processing data of a phoneme, pseudo-morpheme, or word from the server 200, Information such as sentence or word-based confidence score can be received and used for data processing. In addition, the terminal 100 may perform data processing using the data processing result received from the server 200 and the personal word list. In this case, May be used.

Even if any model is used, personal information can be processed more accurately by utilizing personal information without being transmitted to the server 200.

According to the disclosed embodiment, since voice processing using personalization modeling is performed only in the terminal 100, no information transmission including personal information occurs between the server 200 and the terminal 100, ) Does not store any personal information. Therefore, a voice processing system having higher performance and higher processing speed can be implemented while protecting personal information originally.

10 is a flowchart showing a voice processing method of a server according to another disclosed embodiment.

First, in step 1010, the server 200 receives a voice signal. The server 200 can receive voice signals through various components. Receiving the voice signal from the terminal 100 may be the most general form, but it may also receive the voice signal through the USB interface unit or the DVD interface unit.

Thereafter, in step 1020, the server 200 performs data processing on the received voice signal using the general model 271. That is, the server 200 performs data processing on the voice signal independently of the terminal 100 using the general model 271. The general model 271 can provide high speech processing performance for various language expressions (vocabulary) of the user in combination with the high computing power of the server 200. Further, since the server 200 has high computation capability, it is possible to process data quickly.

Next, in step 1030, the server 200 transmits the data processing result to the terminal 100. In this case, the server 200 may generate the voice section information for the personalized information section and the general information section, and may transmit the voice section information to the terminal 100 together. It is possible to reduce the data processing burden of the terminal 100 and to speed up the overall voice processing speed by generating the voice section information in the server 200 having high computing capability and transmitting the generated voice section information to the terminal 100. [ Here, the voice interval information may include interval marking information that marks at least one interval of the personalized information interval and the general information interval in the voice signal.

As a result, according to the disclosed embodiment, a voice processing system having higher performance and higher processing speed can be implemented while originally protecting personal information.

11 is a flowchart illustrating an exemplary operation procedure of a terminal and a server according to another disclosed embodiment.

First, in step 1110, the terminal 100 receives a voice signal. As described above, the terminal 100 can receive voice signals through various components. Receiving a voice signal through a microphone unit may be the most general form, but it may also receive a voice signal through a USB interface unit or a DVD interface unit. Further, it is also possible to receive a voice signal through communication with an external device. Thereafter, in step 1120, the terminal 100 can transmit the received voice signal to the server 200. [

In step 1130, the server 200 processes the received voice signal using the general model 271. The general model 271 can provide high speech processing performance for various language expressions (vocabulary) of the user in combination with the high computing power of the server 200. Further, since the server 200 has high computation capability, it is possible to process data quickly.

At this time, the server 200 can generate the personalization information section and the voice section information for the general information section. It is possible to reduce the data processing burden of the terminal 100 by generating the voice interval information in the server 200 having a high computation capability, and thereby to speed up the overall voice processing speed.

Thereafter, in step 1140, the server 200 may transmit the data processing result and the voice interval information to the terminal 100. [ In step 1150, the terminal 100 may process the voice signal using the personalization model 171 generated based on the personal information and the received data processing result. That is, the terminal 100 can perform data processing on all or part of the voice signal once again. The data processing is once again performed using the personalization model 171 for higher voice processing performance.

In this case, the terminal 100 may detect the personalized information section and perform data processing on the voice signal corresponding to the personalized information section. Also, the terminal 100 can process data on a voice signal using at least one of a personalized speech recognition model, a language understanding model, and a personalized lexical model.

According to the disclosed embodiment, since voice processing using personalization modeling is performed only in the terminal 100, no information transmission including personal information occurs between the server 200 and the terminal 100, ) Does not store any personal information. Therefore, a voice processing system having higher performance and higher processing speed can be implemented while protecting personal information originally.

12 is a block diagram illustrating an internal configuration of a terminal according to another disclosed embodiment.

Referring to FIG. 12, a terminal 1200 according to another disclosed embodiment includes a communication unit 1210 and a control unit 1230.

The communication unit 1210 performs communication with the external device. The communication unit 1210 may be connected to a network by wire or wirelessly to perform communication with an external device. According to the disclosed embodiment, the communication unit 1210 communicates with the server and can transmit and receive data. According to the disclosed embodiment, the communication unit 130 communicates with the server and can transmit and receive data. For example, the communication unit 130 may include a short-range communication module, a mobile communication module, a wireless Internet module, and a wired Internet module. Further, the communication unit 130 may include one or more components.

The control unit 1230 controls the operation of the entire terminal 1200 and can process the audio signal by controlling the communication unit 1210. The control unit 1230 may store signals or data input from the outside of the terminal 1200, a RAM used as a storage area corresponding to various jobs performed by the electronic device, a ROM (ROM) storing a control program for controlling the peripheral devices, And a processor. The processor may be implemented as a SoC (System On Chip) incorporating a core (not shown) and a GPU (not shown). A processor may also include a plurality of processors.

The control unit 1230 according to the disclosed embodiment generates a mapping table by associating the ID with the personal information, generates an ID-based personalization model using the mapping table, and transmits the ID-based personalization model to the server through the communication unit 1210 Receives a result of data processing using an ID-based personalization model for a voice signal from the server, and controls to restore personal information corresponding to the ID using the data processing result and the mapping table. At this time, the controller 1230 may generate the mapping table by associating the ID with the additional information generated from the personal information. In this case, the controller 1230 receives the result of the data processing using the ID-based personalization model with respect to the additional information from the server, and controls to restore the additional information corresponding to the ID using the data processing result and the mapping table have.

When generating the ID-based personalization model using the mapping table, the control unit 1230 may display the ID mapped to the personal information as the ID mapped to the sound unit. Here, the acoustic unit ID may be an ID mapped to the acoustic unit in agreement with the server. The acoustic unit ID indicates a specific part of the speech recognition model corresponding to the pronunciation symbol, and may not be mapped to the pronunciation symbol one-to-one.

According to the disclosed embodiments, the additional information generated from the personal information and the personal information is masked by the ID directly given by the terminal to generate the personalization model, so that even if the personalization model is exposed to the outside, it is difficult to restore the personal information masked by the ID, Can be protected. In addition, a personalization model is transmitted to a server having a high computation capability and data processing is performed on a voice signal, thereby realizing a voice processing system having higher performance and processing speed.

FIG. 13 is a block diagram showing an internal configuration of a server according to another disclosed embodiment in more detail.

13, the server 1300 according to the disclosed embodiment includes a receiving unit 1310, a communication unit 1330, and a control unit 1350.

The receiving unit 1310 plays a role of receiving a voice signal. The receiving unit 1310 may include a component capable of receiving voice signals in various forms, such as a USB interface unit and a DVD interface unit. For example, when the receiving unit 1310 includes a USB interface, the server 1300 can receive a voice signal file from the USB. In addition, when receiving a voice signal from an external device through the communication unit 1330, the communication unit 1330 may serve as the receiving unit 1310. [

The communication unit 1330 performs communication with the external device. The communication unit 1330 may be connected to the network by wire or wirelessly to perform communication with an external device. According to the disclosed embodiment, the communication unit 1330 can communicate with the terminal 1200 and transmit / receive data. According to the disclosed embodiment, the communication unit 130 communicates with the server and can transmit and receive data. For example, the communication unit 130 may include a short-range communication module, a mobile communication module, a wireless Internet module, and a wired Internet module. Further, the communication unit 130 may include one or more components.

The control unit 1350 controls the operation of the entire server 1300 and can process the audio signal by controlling the receiving unit 1310 and the communication unit 1330. The controller 1350 stores signals or data input from the outside of the server 1300, a RAM used as a storage area corresponding to various jobs performed in the electronic device, a ROM (ROM) storing a control program for controlling the peripheral devices, And a processor. The processor may be implemented as a SoC (System On Chip) incorporating a core (not shown) and a GPU (not shown). A processor may also include a plurality of processors.

The control unit 1350 according to the disclosed embodiment receives the ID-based personalization model from the terminal 1200 through the communication unit 1330, receives the voice signal through the receiving unit 1310, and performs ID-based personalization And controls the communication unit 1330 to transmit the data processing result to the terminal 1200.

In addition, the control unit 1350 may use the ID-based personalization model for the voice signal to map the voice signal to the personal information using the sound unit ID, which is an ID mapped to the sound unit, It is possible to control to display the ID.

According to the disclosed embodiment, the server 1300 receives an ID-based personalization model from the terminal 1200 and processes the data using the received ID-based personalization model, so that even if the personalization model is exposed to the outside, It is difficult to restore the personal information 1272, and thus the personal information 1272 can be protected. In addition, a server 1300 having a high computing capability performs data processing on a voice signal, thereby realizing a voice processing system having higher performance and processing speed.

FIG. 14 is a block diagram showing an internal configuration of a terminal according to another disclosed embodiment in more detail. In FIG. 14, the description of the elements that are the same as those in FIG. 12 will be omitted.

The control unit 1230 may include a personalization model generation unit 1231. According to the disclosed embodiment, the personalization model generation unit 1231 can generate a personalization model based on the personal information 1272 included in the storage unit 1270. [ The personalization model generation unit 1231 generates the mapping table 1273 by associating the ID with the additional information generated from the personal information 1272 or the personal information 1272 and generates the mapping table 1273 based on the ID based The personalization model of the user can be created.

The terminal 1200 may further include a receiving unit 1250. The receiver 1250 plays a role of receiving a voice signal. The receiving unit 1250 may include various components such as a microphone unit, a USB interface unit, and a DVD interface unit. For example, when the receiving unit 1250 includes a microphone unit, the terminal 1200 can directly receive a user voice signal through the microphone unit. Also, when the receiving unit 1250 includes a USB interface unit, the terminal 1200 may receive a voice signal file from the USB. Further, when receiving a voice signal from an external device through the communication unit 130, the communication unit 1210 may serve as the receiving unit 1250.

The terminal 1200 may further include a storage unit 1270. The storage unit 1270 stores programs and data necessary for the operation of the terminal 1200. The storage unit 1270 may be a volatile storage medium or a nonvolatile storage medium and may be a combination of both storage media. The volatile storage medium may include a semiconductor memory such as a RAM, a DRAM, and a SRAM. The nonvolatile storage medium may include a hard disk and a flash NAND memory. According to the disclosed embodiment, the personal information 1272 and the mapping table 1273 may be stored in the storage unit 170.

The personal information 1272 is information that can directly or indirectly identify each individual. The type of data stored according to the type of the terminal can be changed. For example, in the case of a mobile device, it may include a contact, a music list, a short message content, a reception history, a call history, and a web search history. In the case of a TV, a personal playlist may be included.

The mapping table 1273 includes an ID corresponding to the additional information generated from the personal information 1272 or the personal information 1272. [ The mapping table 1273 is used by the personalization model generation unit 1231 to generate an ID-based personalization model. The mapping table 1273 is also used for restoring the personal information 1272 corresponding to the ID or the additional information.

15 is a block diagram showing in more detail the internal configuration of the server shown in Fig.

The control unit 1350 may include a voice processing engine 1351. According to the disclosed embodiment, the speech processing engine 1351 may include a speech recognition engine and a language understanding engine and performs data processing on the received speech signal to perform speech recognition and language understanding. At this time, the speech recognition engine and the language understanding engine can process the speech signal using the speech recognition model and the language understanding model, respectively.

The server 1300 may further include a storage unit 1370. The storage unit 1370 stores programs and data necessary for the operation of the server 1300. The storage unit 1370 may be a volatile storage medium or a nonvolatile storage medium, or may be a combination of both storage media. The volatile storage medium may include a semiconductor memory such as a RAM, a DRAM, and a SRAM. The nonvolatile storage medium may include a hard disk and a flash NAND memory.

According to the disclosed embodiment, the personalization model 1372 and the general model 1373 may be stored in the storage unit 1370. [ The personalization model 1372 is an ID-based personalization model received from the terminal 1200, and even if the personalization model is exposed to the outside, it is difficult to restore the personal information 1272 masked by the ID, so that the personal information 1272 can be protected . The general model 271 is a general speech processing model, and is a speech processing model for speech processing of an unspecified person who is not a specific individual. The generic model 271, which is composed of a large capacity, can combine with the high computing power of the server to provide high voice processing performance for various language expressions (vocabulary) of the user.

Operations of the terminal 1200 and the server 1300 will be described in more detail below.

16 is a flowchart showing a voice processing method of a terminal according to another disclosed embodiment.

First, in step 1610, the terminal 1200 generates a mapping table 1273 by associating an ID with the personal information 1272. In this case, the mapping table 1273 may be generated by associating the ID with the additional information generated from the personal information 1272 together with the personal information 1272. [ Here, the additional information may include a phonetic symbol, a pronunciation string, and the like. According to the disclosed embodiment, the terminal 1200 can generate a pronunciation dictionary using the word list of the personal information 1272, and map the ID to the pronunciation symbol and the word using the pronunciation dictionary. In this case, the terminal 1200 can assign an arbitrary ID. This will be described in more detail with reference to FIGS. 18 to 20. FIG.

18 is a diagram showing personal information.

The personal information 1272 is information that can directly or indirectly identify each individual. Examples of the personal information 1272 include the contents of a contact, a music list, and a short message, a transmission history, a web search history, and a play list. Referring to FIG. 18, various types of personal information are shown. It is noted that personal information 1272 such as a name stored in a contact item, a music list in a music playlist or an artist, a search result, and the like is stored. According to the disclosed embodiment, the terminal 1200 can generate a mapping table by mapping an ID to this personal information 1272. [ This will be described with reference to FIG.

19 is a diagram showing a mapping table in which personal information is mapped to an ID.

Referring to FIG. 19, the terminal 1200 stores the words 0, 1, 2, 3, 4, 5, 6, 7, 0x07 to generate a word mapping table 1910. [ Furthermore, the terminal 1200 may generate the mapping table 1273 by associating the ID with the additional information generated from the personal information 1272 as well as the personal information 1272. [ Such additional information may include a pronunciation symbol, a pronunciation column, and the like. This will be described with reference to FIG.

20 is a diagram showing individual information by phonetic symbols.

According to the disclosed embodiment, the terminal 1200 can generate a phonetic dictionary 1910 using the word list of the personal information 1272. Referring to FIG. 20, the name of the personal information 1272, which is the name stored in the contact, indicates the pronunciation symbol with 'HH OW NX K IY T OW NX'. Also, the Gangnam style included in the music list can be displayed as 'K AA NX N A M ST AI L'. The terminal 1200 can generate the pronunciation dictionary 1910 by displaying the personal information 1272 as a pronunciation symbol in this manner. At this time, the terminal 1200 can use various types of pronunciation symbols. The pronunciation may be displayed using the alphabet as described above, or, in the case of English words, the pronunciation dictionary 2010 may be generated using the English pronunciation symbol. Referring to FIG. 20, TOXIC is represented by 't?: K s? k ', Galaxy is' g? l? personal information 1272 can be displayed in English pronunciation symbols as indicated by k s i '. The terminal 1200 can generate the mapping table 1273 by associating the ID with the additional information generated from the personal information 1272 such as the pronunciation symbol as well as the personal information 1272. [

Returning to the description of FIG. 16, the terminal 1200 generates an ID-based personalization model 1372 using the mapping table 1273 in step 1620. The ID-based personalization model 1372 is generated using the mapping table 1273 generated in step 1610, and the personal information 1272 and the additional information are masked by the ID. According to the disclosed embodiment, the terminal 1200 can generate the personalization model, thereby indicating the ID mapped to the private information 1272 and the additional information by the acoustic unit ID. Will be described with reference to Figs. 21 and 22. Fig.

21 is a diagram showing a mapping table in which acoustic units are mapped to IDs.

The acoustic unit ID indicates a specific part of the speech recognition model corresponding to the pronunciation symbol. Referring to FIG. 21, the terminal 100 converts the pronunciation symbols HH, OW, NX, K, IY, and L included in each word into IDs 0x101, 0x102, 0x103, 0x104, 0x105, 0x106, ..., So as to generate a sound unit mapping table 2110. [ According to the disclosed embodiment, when the terminal 1200 maps an ID to a sound, it can map a specific ID to a specific sound according to agreement with the server 1300. [ That is, the ID 0x101 mapped to the acoustic unit HH by the terminal 1200 may be an ID agreed with the server 1300 in advance. Accordingly, the server 1300 can correspond to a specific voice, that is, a specific sound, with a specific ID agreed with the terminal 1200, when processing the voice signal. The terminal 1200 and the server 1300 can specify an ID mapped to a specific sound from either the terminal 1200 or the server 1300 to notify the other party, exchange opinions, map an ID to a pronunciation symbol, You may. In FIG. 20, the pronunciation symbol and the sound unit ID are mapped one to one for convenience, but the sound unit ID may not be mapped to the pronunciation symbol one to one. For example, a sound unit ID may be assigned to the HH OW by considering the sound combining the pronunciation symbols HH and OW as one sound unit.

22 is a diagram showing the personal information ID by the sound unit ID.

The terminal 1200 can map the word ID 2220 to the personal information 2210 and use the pronunciation symbol and acoustic model of the personal information 2210 to assign the acoustic unit ID 2230 to the word ID 2220 Can be mapped. The acoustic unit ID indicates a specific part of the speech recognition model corresponding to the pronunciation symbol, and may not be mapped to the pronunciation symbol one-to-one. Here, it is assumed that the acoustic unit ID and the pronunciation symbol are mapped on a one-to-one basis for convenience.

Referring to FIG. 22, the terminal 1200 arbitrarily maps the word 'Hong Kil Dong' to the personal information 1272 'ID 0x01. The word 'Hong Gil Dong' can be represented by the pronunciation symbol 'HH OW NX K IY T OW NX', and each pronunciation symbol is represented by the sound unit IDs 0x101, 0x102, 0x103, 0x104, 0x105, 0x106, ... Lt; / RTI > Therefore, the ID of 0x01 corresponding to 'Hong Kil-Dong' corresponds to the sound unit IDs 0x101, 0x102, 0x103, 0x104, 0x105, 0x106, ... .

According to the disclosed embodiment, the terminal 1200 arbitrarily maps an ID to a word, and the corresponding word ID can be represented by an acoustic unit ID agreed with the server. Accordingly, the personal information 1272 can protect the personal information 1272 even if the personalization model is exposed to the outside by masking with the ID, and the server 1300 can perform data processing on the voice signal using the agreed acoustic unit ID can do.

In step 1630, the terminal 1200 transmits an ID-based personalization model 1372 to the server. As described above, the ID-based personalization model 1372 may be generated based on the word ID 2220 and acoustic unit ID 2230 shown in FIG. Accordingly, when the server 1300 receives the voice signal to be recognized, the server 1300 processes the voice signal and outputs a word ID 2220 corresponding to the voice unit ID 2230 of the voice signal as a result. At this time, the mapping table 1273 is stored only in the terminal 1200 without being transmitted to the server 1300, thereby protecting the personal information 1272 even if the personalization model is exposed to the outside.

After that, the terminal 1200 receives the result of the data processing using the ID-based personalization model 1372 with respect to the voice signal from the server 1300 in step 1640. For example, the result of data processing using the ID-based personalization model 1372 at the server may include a word ID 2220 as shown in FIG.

Next, in step 1650, the terminal 1200 restores the personal information 1272 or the additional information corresponding to the ID using the data processing result received from the server 1300 and the mapping table 1273. That is, when the terminal 1200 receives the word ID 2220 as shown in FIG. 22 as a data processing result from the server 1300, the terminal 1200 uses the stored word mapping table 1273 to obtain the word ID 2220 corresponding to the word ID 2220 Personal information can be restored. Referring to FIG. 20, the terminal 1200 can restore the ID 0x01 to 'Hong Kil Dong'. According to the disclosed embodiment, the terminal 1200 can complete the data processing by restoring the personal information 1272 masked by the ID using the mapping table 1273. [ The terminal 1200 generates a personalization model 1372 to enable the voice processing system to have high performance and the actual data processing is performed in the server 1300 having high computing ability, so that the voice signal can be processed quickly.

 Furthermore, the terminal 1200 can output the data processing result to the user.

As a result, according to the disclosed embodiment, the personal information 1272 is transformed into a form that can not grasp the contents of a word or a sentence and is transmitted to the server 1300, so that even if the personalization model is exposed to the outside, The personal information 1272 can be protected. In addition, a personalization model is transmitted to a server having a high computation capability and data processing is performed on a voice signal, thereby realizing a voice processing system having higher performance and processing speed.

17 is a flowchart showing a voice processing method of a server according to another disclosed embodiment.

First, in step 1710, the server 1300 receives an ID-based personalization model 1273 from the terminal 1200. The ID-based personalization model 1273 has a function of protecting the personal information 1272 because it is difficult to restore the personal information 1272 masked by the ID even if the personalization model is exposed to the outside because the portion of the personal information 1272 is masked by the ID can do.

Thereafter, the server 1300 receives the voice signal in step 1720. [ The server 200 can receive voice signals through various components. Receiving the voice signal from the terminal 100 may be the most general form, but it may also receive the voice signal through the USB interface unit or the DVD interface unit.

Next, the server 1300 processes the voice signal using the ID-based personalization model 1273 in step 1730. In this case, the server 1300 can control to display the personal information ID using the ID mapped to the sound unit according to the pre-agreement with the terminal 1200. Will be described with reference to Figs. 21 to 22. Fig.

Referring to FIG. 21, an ID is mapped to a sound unit according to an agreement with the terminal 1200 by the server 1300. FIG. The server 1300 recognizes the audio signal of 'Hong Gil Dong' as a set of continuous sounds. Therefore, a speech signal of 'Hong Kil-Dong' is assigned to the terminal 1200 in a manner of assigning 0x101, which is an ID agreed in advance with the terminal 1200, and 0x102, for the sound 'OW' 0x101, 0x102, 0x103, 0x104, 0x105, 0x106, ... .

The server 1300 can find and correspond to the word ID corresponding to this set of pronunciation symbol IDs in the personalization model 1273 received from the terminal 1200. [ Referring to FIG. 22, an ID of 0x01 is mapped to the word 'Hong Kil Dong'. Accordingly, the server 1300 transmits a voice signal of 'Hong Kil-Dong' as a set of pronunciation symbol IDs 0x101, 0x102, 0x103, 0x104, 0x105, 0x106, ... Quot; 0x01 " corresponding to " 0x01 ". The server 1300 can have high performance by receiving the personalization model 1372 from the terminal 1200 and performing the data processing and the server 1300 has high computation ability and can perform fast processing of the voice signal have.

In addition, the server 1300 transmits the data processing result to the terminal 1200 in step 1740.

 As a result, according to the disclosed embodiment, the personal information 1272 is transformed into a form that can not grasp the contents of a word or a sentence and is transmitted to the server 1300, so that even if the personalization model is exposed to the outside, The personal information 1272 can be protected. In addition, a server 1300 having a high computation capability can perform a data processing on a voice signal using a personalization model, thereby realizing a voice processing system having higher performance and processing speed.

23 is a flowchart showing an example of a specific operation procedure of a terminal and a server according to another disclosed embodiment.

First, in step 2310, the terminal 1200 generates a mapping table 1273 by associating an ID with the personal information 1272. The terminal 1200 can generate a pronunciation dictionary 1910 using the word list of the personal information 1272 and map an ID to the pronunciation symbol and the word using the pronunciation dictionary 1910. [ In this case, the terminal 1200 can assign an arbitrary ID.

Next, in step 2320, the terminal 1200 generates an ID-based personalization model 1372 using the mapping table 1273, and transmits the ID-based personalization model 1372 to the server 1300 in step 2330. The server 1300 stores the ID-based personalization model 1372 received in step 2340 in the storage unit 1370. The ID-based personalization model 1372 can be generated based on the word ID 2220 and the acoustic unit ID 2230 shown in Fig.

Thereafter, the terminal 1200 receives the voice signal in step 2350 and transmits the voice signal to the server 1300 in step 2360. As described above, the terminal 100 can receive voice signals through various components. Receiving a voice signal through a microphone unit may be the most general form, but it may also receive a voice signal through a USB interface unit or a DVD interface unit. Further, it is also possible to receive a voice signal through communication with an external device.

The server 1300 processes the voice signal received in step 2370 using an ID-based personalization model, and transmits the data processing result to the terminal 1200 in step 2380. In this case, the server 1300 can display the ID mapped to the private information 1272 or the additional information, using the ID mapped to the sound unit, in agreement with the terminal 1200.

Then, in step 2390, the terminal 1200 restores the personal information 1272 or the additional information corresponding to the ID using the data processing result and the mapping table 1273.

According to the disclosed embodiment, the server 1300 receives an ID-based personalization model from the terminal 1200 and processes the data using the received ID-based personalization model, so that even if the personalization model is exposed to the outside, It is difficult to restore the personal information 1272, and thus the personal information 1272 can be protected. In addition, a server 1300 having a high computing capability performs data processing on a voice signal, thereby realizing a voice processing system having higher performance and processing speed.

The above-described embodiments may be embodied in a general-purpose digital computer that can be embodied as a program that can be executed by a computer and operates the program using a computer-readable recording medium.

The computer readable recording medium may be a magnetic storage medium such as a ROM, a floppy disk, a hard disk, etc., an optical reading medium such as a CD-ROM or a DVD and a carrier wave such as the Internet Lt; / RTI > transmission).

While the embodiments have been described with reference to the above description and accompanying drawings, those skilled in the art will appreciate that the disclosed embodiments may be practiced in other specific forms without departing from the spirit or essential characteristics thereof. It will be possible. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive.

100: terminal
110:
130:
150:
200: Server
210:
230:
250:

Claims (42)

Receiving a speech signal;
Detecting a personalized information section including personal information of the voice signal;
Performing data processing on a voice signal corresponding to the personalized information section of the voice signal using a personalized model generated based on the personal information; And
And receiving a result of data processing of a voice signal corresponding to a general information section, which is a section other than the personalized information section, from the server.
The method according to claim 1,
And generating and transmitting the personalized information section and the voice section information for the general information section to the server.
3. The method of claim 2,
The voice section information includes:
And marking at least one of the personalized information section and the general information section of the voice signal.
The method according to claim 1,
Further comprising the step of receiving the personalized information section and the voice interval information for the general information section from the server.
The method according to claim 1,
And a result of data processing of the audio signal corresponding to the general information section received from the server,
Wherein the server is a result of processing a voice signal corresponding to the general information section using a general model.
Receiving a voice signal;
Detecting a personalized information section including personal information of the voice signal;
Performing data processing on a speech signal corresponding to a general information section, which is a section of the speech signal other than the personalized information section, using a general model; And
And transmitting the data processing result of the voice signal corresponding to the general information section to the terminal.
The method according to claim 6,
Generating the personalized information section and the voice section information for the general information section and transmitting the generated voice section information to the terminal.
8. The method of claim 7,
The voice section information includes:
And marking at least one of the personalized information section and the general information section of the voice signal.
The method according to claim 6,
Further comprising the step of receiving the personalized information section and the voice section information for the general information section from the terminal.
Receiving a voice signal;
Receiving a result of data processing of the voice signal from a server using a general model; And
A personalization model generated based on personal information on the voice signal, and data processing using the personalization model and the data processing result.
11. The method of claim 10,
Wherein the personalization model generated based on the personal information with respect to the voice signal and the data processing using the data processing result include:
And performing data processing on a voice signal corresponding to a personalized information section including the personal information.
12. The method of claim 11,
Further comprising detecting the personalized information section of the voice signal.
11. The method of claim 10,
Further comprising the step of receiving voice section information on the personalized information section and the general information section from the server.
14. The method of claim 13,
The voice section information includes:
And marking at least one of the personalized information section and the general information section of the voice signal.
11. The method of claim 10,
The personalization model includes:
Wherein the at least one model is at least one of a personalized speech recognition model, a personalized natural language understanding model, and a personalized lexical model.
Generating a mapping table by mapping IDs to private information;
Generating an ID-based personalization model using the mapping table;
Transmitting the ID-based personalization model to a server;
Receiving a result of data processing of the voice signal from the server using the ID-based personalization model; And
And restoring the personal information corresponding to the ID using the data processing result and the mapping table.
17. The method of claim 16,
Wherein the step of generating an ID-based personalization model using the mapping table comprises:
And displaying the ID mapped to the personal information as an acoustic unit ID that is an ID mapped to an acoustic signal.
18. The method of claim 17,
The acoustic unit ID includes:
And an ID mapped to the sound in agreement with the server.
18. The method of claim 17,
And mapping the ID to the additional information generated from the private information to generate the mapping table.
Receiving an ID-based personalization model from a terminal;
Receiving a voice signal;
Performing data processing on the voice signal using the ID-based personalization model; And
And transmitting the data processing result to the terminal.
21. The method of claim 20,
Wherein the step of data processing the voice signal using the ID-based personalization model comprises:
And displaying an ID mapped to the personal information using an acoustic unit ID that is an ID mapped to the acoustic unit in agreement with the terminal.
A receiving unit for receiving a voice;
A communication unit for communicating with the server; And
A personalization information section that includes personal information of the voice signal and detects a voice signal corresponding to the personalized information section of the voice signal using a personalization model generated based on the personal information, And a control unit for controlling the data processing unit to receive a result of data processing of a voice signal corresponding to a general information section, which is a section other than the personalized information section, from the server.
23. The method of claim 22,
Wherein,
And generates and transmits voice section information for the personalized information section and the general information section to the server.
24. The method of claim 23,
The voice section information includes:
And marking at least one of the personalized information section and the general information section of the voice signal.
23. The method of claim 22,
Wherein,
And controls to receive the personalized information section and the voice section information for the general information section from the server.
23. The method of claim 22,
And a result of data processing of the audio signal corresponding to the general information section received from the server,
Wherein the server is a result of processing a voice signal corresponding to the general information section using a general model.
A receiving unit for receiving a voice;
A communication unit for performing communication with the terminal; And
The method includes receiving a voice signal, detecting a personalized information section including personal information of the voice signal, and transmitting the voice signal corresponding to the general information section, which is a section other than the personalized information section, And a controller for controlling the terminal to transmit a data processing result of the voice signal corresponding to the general information section to the terminal.
28. The method of claim 27,
Wherein,
And generates voice section information on the personalized information section and the general information section and transmits the generated voice section information to the terminal.
29. The method of claim 28,
The voice section information includes:
And marking at least one of the personalized information section and the general information section of the voice signal.
28. The method of claim 27,
Wherein,
And controls to receive the personalized information section and the voice section information for the general information section from the terminal.
A communication unit for communicating with the server; And
A control unit for receiving a result of data processing of the voice signal from the server using a general model and for controlling data processing of the voice signal using the personalization model generated based on the data processing result and the personal information And the terminal.
32. The method of claim 31,
Wherein,
When data processing is performed on the voice signal using the personalization model generated based on the data processing result and the personal information,
And controls to perform data processing on a voice signal corresponding to a personalized information section including the personal information.
33. The method of claim 32,
Wherein,
And controls to detect the personalized information section of the voice signal.
32. The method of claim 31,
Wherein,
And controls to receive from the server the personalized information section and the voice section information for the general information section.
35. The method of claim 34,
The voice section information includes:
And marking at least one of the personalized information section and the general information section of the voice signal.
32. The method of claim 31,
The personalization model includes:
A personalized speech recognition model, a language understanding model, and a personalized lexical model.
Receiving unit for receiving a voice signal:
A communication unit for communicating with the server; And
Based personalization model by using the mapping table, transmitting the ID-based personalization model to the server, and transmitting the ID from the server to the ID Based on a result of the data processing using the personalization model based on the personalization model, and restoring the personal information corresponding to the ID using the data processing result and the mapping table.
39. The method of claim 37,
Wherein,
When creating an ID-based personalization model using the mapping table,
And controls the ID mapped to the personal information to be represented by an acoustic unit ID which is an ID mapped to the sound.
39. The method of claim 38,
The acoustic unit ID includes:
And an ID mapped to the sound in agreement with the server.
39. The method of claim 38,
Wherein,
And generates the mapping table by mapping an ID to the additional information generated from the personal information.
A receiving unit for receiving a voice signal;
A communication unit for performing communication with the terminal; And
Based personalization model, receives an audio-based personalization model, receives a voice signal, processes the voice signal using the ID-based personalization model, and transmits the data processing result to the terminal The server.
42. The method of claim 41,
Wherein,
Based on the ID-based personalization model,
And controls to display an ID mapped to the personal information using an acoustic unit ID that is an ID mapped to the acoustic unit in agreement with the terminal.

Images