KR102389995B1 - Method for generating spontaneous speech, and computer program recorded on record-medium for executing method therefor - Google Patents

Abstract

Provided by the present invention is a method for generating a naturally-uttered voice which can be used for speech-recognition learning. The method comprises the steps of: identifying, by a voice data generating device, voice data from which all noise has been removed from a database provided in advance; identifying, by the voice data generating device, a speaker who recorded the voice data based on metadata matched to the voice data; identifying, by the voice data generating device, man-made noise corresponding to the identified speaker, and synthesizing the man-made noise into the voice data; and identifying, by the voice data generating device, background sound data corresponding to a pre-set environmental condition, and synthesizing the identified background sound data into the voice data synthesized with the artificial noise. In this case, the artificial noise is uttered according to the speech method, exclamation, or cheering words of the speaker of the voice data and thus can be a phoneme that has no verbal meaning. Accordingly, through the generation of voice data, the performance of the voice recognition system can be improved.

Description

Method for generating spontaneous speech, and computer program recorded on record-medium for executing method therefor

The present invention relates to the collection of data for artificial intelligence (AI) machine learning. More particularly, it relates to a method for generating spontaneous speech that can be used for learning speech recognition, and a computer program recorded on a recording medium for executing the method.

Artificial intelligence (AI) refers to a technology that artificially implements some or all of human learning ability, reasoning ability, and perception ability using computer programs. In relation to artificial intelligence (AI), machine learning refers to learning to optimize parameters with given data using a model composed of multiple parameters. Such machine learning is classified into supervised learning, unsupervised learning, and reinforcement learning according to the type of data for learning.

In general, the design of data for artificial intelligence (AI) machine learning proceeds in the stages of data structure design, data collection, data purification, data processing, data expansion, and data verification.

To describe each step in more detail, the design of the data structure is made through the definition of an ontology, a definition of a classification system, and the like. Data collection is performed by collecting data through direct shooting, web crawling, or association/professional organizations. Data purification is performed by removing duplicate data from the collected data and de-identifying personal information. Data processing is performed by performing annotations or inputting metadata. Data expansion is performed by performing ontology mapping, and supplementing or extending the ontology as necessary. And, the verification of the data is performed by verifying the validity according to the set target quality using various verification tools.

Meanwhile, speech recognition refers to a technology for recognizing a linguistic meaning from a voice data signal. As such, speech recognition is classified into word recognition, continuous speech recognition, and speaker recognition according to a recognition target.

In general, word speech recognition and continuous speech recognition among the classification of speech recognition may be utilized as a user interface for calling a computing system through a voice, instructing a command, or inputting information. In addition, speaker recognition may be used for security purposes such as determining the authenticity of the speaker and identifying the speaker from the crowd.

To describe the speech recognition process in more detail, speech recognition is performed by performing pattern recognition by comparing an acoustic model including a reference pattern with voice data input from the outside. The acoustic model used for such speech recognition can be largely divided into a template-based method and a statistical method based on probability distribution. And, among acoustic models, dynamic time warping (DTW) is widely used as a template-based method, and a hidden Markov model (HMM) is widely used as a statistical method based on probability distribution.

On the other hand, human voices have different wave forms according to countries, languages, regions, genders, or ages, even if they are words or sentences having the same meaning. Therefore, in order to construct a reference pattern for performing voice recognition for each country, language, region, gender, and age, it is necessary to secure a lot of voice data.

Republic of Korea Patent Publication No. 10-2014-0079988, 'Mobile terminal and voice recognition processing method using the same', (published on June 30, 2014)

It is an object of the present invention to provide a method for generating spontaneous speech that can be used for learning speech recognition.

Another object of the present invention is to provide a computer program recorded on a recording medium for executing a method capable of generating spontaneous speech.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

In order to achieve the technical problem as described above, the present invention proposes a method capable of generating a spontaneous speech that can be used for learning speech recognition. The method includes: identifying, by a voice data generating apparatus, voice data from which all noise has been removed from a database provided in advance; identifying, by the voice data generating apparatus, a speaker who recorded the voice data based on metadata matched to the voice data; identifying, by the voice data generating device, a man-made noise corresponding to the identified speaker, and synthesizing the identified artificial noise into the voice data; and identifying, by the voice data generating apparatus, background sound data corresponding to a preset environmental condition, and synthesizing the identified background sound data with the artificial noise synthesized voice data. In this case, the artificial noise may be a phoneme that is uttered according to the utterance method, interjection, or chuimsae of the speaker of the voice data, but does not have a word meaning.

Specifically, in the step of synthesizing the background sound data, a noise signal and an echo signal satisfying the environmental conditions are identified from a database provided in advance, and then the artificial noise is synthesized. The noise signal and the echo signal may be sequentially synthesized with the obtained voice data.

In the synthesizing of the background sound data, after identifying an inaudible signal that satisfies the environmental condition from the database, the inaudible signal can be synthesized with the voice data in which the noise signal and the echo signal are synthesized. there is.

In the synthesizing of the background sound data, the volume of each channel of the voice signal included in the voice data in which the inaudible signal is synthesized may be adjusted so that the direction of the speaker is changed in the process of reproducing the voice data.

On the other hand, the step of synthesizing the artificial noise includes generating a transcript by STT (Speech To Text) of the voice data, and parsing the transcript to obtain a plurality of tokens constituting the transcript. It is possible to identify and perform semantic analysis on each of the identified tokens.

In the synthesizing of the artificial noise, as a result of performing the semantic analysis, when one or more tokens predefined as expressing emotions exist, one emotional state that the speaker may have based on the one or more tokens can be estimated.

In the synthesizing of the artificial noise, one or more phonemes corresponding to the estimated emotional state and capable of generating artificial noise may be identified, and the identified one or more phonemes may be synthesized into the voice data.

Alternatively, the step of synthesizing the artificial noise may include identifying the speaker's country, language, region, gender and age from the metadata of the speaker, and corresponding to the identified speaker's country, language, region, gender and age. and one or more phonemes capable of generating artificial noise may be identified, and the identified one or more phonemes may be synthesized into the voice data.

In addition, the step of synthesizing the artificial noise may include identifying one or more phonemes capable of generating artificial noise related to another speaker that is different from the speaker in at least one of country, language, region, gender, and age, and the identified one or more Phonemes may be synthesized into the voice data.

In order to achieve the technical problem as described above, the present invention proposes a computer program recorded on a recording medium for executing a spontaneous speech synthesis method. The computer program includes a memory; transceiver; and a processor for processing instructions resident in the memory. The computer program may further include: identifying, by the processor, voice data from which all noise has been removed from a database provided in advance; identifying, by the processor, a speaker who recorded the voice data based on metadata matched to the voice data; identifying, by the processor, an artificial noise corresponding to the identified speaker, and synthesizing the identified artificial noise into the voice data; and a computer program recorded on a recording medium for executing, by the processor, identifying background sound data corresponding to a preset environmental condition, and synthesizing the identified background sound data with the artificial noise synthesized voice data. this can be

Specific details of other embodiments are included in the detailed description and drawings.

According to embodiments of the present invention, it is possible to generate various types of voice data required to construct reference patterns to be included in an acoustic model of voice recognition. In particular, the voice recognition system can improve the performance of the voice recognition system by generating voice data that can also learn the speaker's utterance, exclamation, or chuimsae.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

1 is a block diagram of a voice data collection system according to an embodiment of the present invention.
2 is a logical configuration diagram of an apparatus for recording voice data according to an embodiment of the present invention.
3 is a logical configuration diagram of an apparatus for generating voice data according to an embodiment of the present invention.
4 is a detailed configuration diagram of a data processing unit according to an embodiment of the present invention.
5 is a hardware configuration diagram of an apparatus for generating voice data according to an embodiment of the present invention.
6 and 7 are exemplary views for explaining a process of decomposing voice data according to an embodiment of the present invention.
8 and 9 are exemplary views for explaining a process of generating voice data according to an embodiment of the present invention.
10 is an exemplary diagram for explaining a process of learning a calling word according to an embodiment of the present invention.
11 is an exemplary diagram for explaining a process of recording voice data according to an embodiment of the present invention.
12 is an exemplary diagram for explaining a process of generating metadata using a speaker's face according to an embodiment of the present invention.
13 is a flowchart illustrating a voice data recording method according to an embodiment of the present invention.
14 is a flowchart illustrating a method for generating voice data according to an embodiment of the present invention.
15 is a flowchart for explaining a voice data processing step according to an embodiment of the present invention.
16 is a flowchart for explaining a voice data processing step according to another embodiment of the present invention.

It should be noted that the technical terms used herein are used only to describe specific embodiments, and are not intended to limit the present invention. In addition, the technical terms used in this specification should be interpreted as meanings generally understood by those of ordinary skill in the art to which the present invention belongs, unless otherwise specifically defined in this specification, and excessively inclusive It should not be construed in the meaning of a human being or in an excessively reduced meaning. In addition, when the technical terms used in the present specification are incorrect technical terms that do not accurately express the spirit of the present invention, they should be understood by being replaced with technical terms that can be correctly understood by those skilled in the art. In addition, the general terms used in the present invention should be interpreted according to the definition in the dictionary or according to the context before and after, and should not be interpreted in an excessively reduced meaning.

Also, as used herein, the singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as "consisting of" or "having" should not be construed as necessarily including all of the various components or various steps described in the specification, and some of the components or some steps are included. It should be construed that it may not, or may further include additional components or steps.

Also, terms including ordinal numbers such as first, second, etc. used herein may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but another component may exist in between. On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that no other element is present in the middle.

Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. In addition, in the description of the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, it should be noted that the accompanying drawings are only for easy understanding of the spirit of the present invention, and should not be construed as limiting the spirit of the present invention by the accompanying drawings. The spirit of the present invention should be interpreted as extending to all changes, equivalents or substitutes other than the accompanying drawings.

Human voices have different wave forms according to countries, languages, regions, genders, or ages, even if they are words or sentences having the same meaning. Therefore, in order to establish a reference pattern for performing voice recognition for each country, language, region, gender, and age, a lot of voice data must be secured.

In order to respond to this need, the present invention intends to propose various means for collecting, processing and generating numerous voice data.

1 is a block diagram of a voice data collection system according to an embodiment of the present invention.

1, the voice data collection system according to an embodiment of the present invention includes a variety of voice data recording devices 100a, 100b, ..., 100n; 100, a voice data generating device 200, and voice recognition learning. It may be configured to include device 300 .

As such, since the components of the voice data collection system according to an embodiment are merely functionally distinct elements, two or more components are implemented by being integrated with each other in an actual physical environment, or one component is an actual physical environment. may be implemented separately from each other.

Each component will be described. The voice data recording apparatus 100 may collect voice data in various environments. As such, all or a part of the voice data recording apparatus 100 may be a device capable of collecting voice data through a clouding service. In addition, various voice data collected by the voice data recording apparatus 100 may be utilized to construct reference patterns that may be included in an acoustic model of voice recognition.

Characteristically, the voice data recording device 100 controls so as to record voice data only when the voice data recording device 100 is located in a place that satisfies the environmental conditions indicated by the voice data generating device 200 . can be

Also, the voice data recording apparatus 100 may time-series images including the user's face through the camera while receiving (ie, recording) voice data from the user. In this way, the image captured by the voice data recording apparatus 100 may be used to estimate the user's emotional state.

For convenience of description below, a user of the voice data recording apparatus 100 may be referred to as a speaker while the voice data recording apparatus 100 records voice data.

As described above, the voice data recording apparatus 100 does not correspond to a dedicated device for only recording voice data, but UE (User Equipment) defined by the 3rd Generation Partnership Project (3GPP), or IEEE (Institute of Electrical and It may correspond to a general-purpose device such as MS (Mobile Station) specified by Electronics Engineers.

That is, the voice data recording apparatus 100 may be any device as long as it is capable of transmitting and receiving data to and from the voice data generating apparatus 200 and performing an operation based on the transmitted/received data. For example, the voice data recording apparatus 100 includes a smart phone, a laptop, a tablet, a phablet, a portable multimedia player (PMP), and a personal portable terminal. It may be any one of a portable computing device such as (Personal Digital Assistants, PDA) or an E-book reader, but is not limited thereto.

A detailed configuration and operation of the voice data recording apparatus 100 will be described later with reference to FIG. 2 .

With the following configuration, the voice data generating apparatus 200 may generate various reference patterns that can be used in an acoustic model of voice recognition based on the voice data collected by the plurality of voice data recording apparatuses 100 .

Characteristically, the voice data generating apparatus 200 may disassemble and manage the voice data collected by the plurality of voice data recording apparatuses 100 into detailed components, and meet the requirements of the voice recognition learning apparatus 300 . Correspondingly, various types of voice data may be generated. In addition, the voice data generating apparatus 200 may set a criterion for distinguishing between a simple conversation voice and a wake-word, and may also generate various types of voice data for learning a wake-word based on this. can Furthermore, the voice data generating apparatus 200 may generate metadata for recognizing the emotional state of the speaker.

As such, the voice data generating device 200 may transmit and receive data to and from the voice data recording device 100 and the voice recognition learning device 300, and any device capable of performing an operation based on the transmitted/received data is acceptable. can be For example, the voice data generating apparatus 200 may be any one of a fixed computing device such as a desktop, a workstation, or a server, but is not limited thereto.

A detailed configuration and operation of the voice data generating apparatus 200 will be described later with reference to FIGS. 3 to 5 .

With the following configuration, the voice recognition learning apparatus 300 builds an acoustic model based on the voice data (ie, a reference pattern) provided from the voice data generating device 200 , and performs voice recognition using the constructed acoustic model. can

Specifically, the voice recognition learning apparatus 300 may transmit a requirement for building an acoustic model to the voice data generating apparatus 200 . The voice recognition learning apparatus 300 may receive a plurality of voice data from the voice data generating apparatus 200 , and may build an acoustic model by setting the received voice data as a reference pattern. In addition, the voice recognition learning apparatus 300 may perform voice recognition using the built acoustic model.

As such, the voice recognition learning apparatus 300 may transmit and receive data to and from the voice data generating apparatus 200, and any device capable of performing an operation using the transmitted/received data may be permitted. For example, the voice recognition learning apparatus 300 may be any one of a desktop, a workstation, or a stationary computing device such as a server, but is not limited thereto.

As described above, various types of voice data recording apparatus 100, voice data generating apparatus 200, and voice recognition learning apparatus 300, as described above, are among the secure line, public wired communication network, or mobile communication network directly connecting the devices. One or more combined networks may be used to transmit/receive data.

For example, public wired networks include Ethernet, x Digital Subscriber Line (xDSL), Hybrid Fiber Coax (HFC), and Fiber To The Home (FTTH). However, it is not limited thereto. In addition, the mobile communication network includes Code Division Multiple Access (CDMA), Wideband CDMA, WCDMA, High Speed Packet Access (HSPA), Long Term Evolution, LTE) and 5th generation mobile communication may be included, but are not limited thereto.

Hereinafter, the configuration of the voice data recording apparatus 100 having the above-described characteristics will be described in more detail.

2 is a logical configuration diagram of an apparatus for recording voice data according to an embodiment of the present invention.

As shown in FIG. 2 , the apparatus 100 for recording voice data according to an embodiment of the present invention includes a terminal communication unit 105 , a terminal input/output unit 110 , a recording environment identification unit 115 , and a voice recording control unit 120 . ), the metadata granting unit 125 and the collection data providing unit 130 may be included.

As such, the components of the voice data recording apparatus 100 merely represent functionally distinct elements, so that two or more components are integrated with each other in an actual physical environment, or one component is implemented with each other in an actual physical environment. It may be implemented separately.

Each of the components will be described. The terminal communication unit 105 may transmit/receive data to and from the voice data generating apparatus 200 .

Specifically, the terminal communication unit 105 may receive the environmental condition from the voice data generating apparatus 200 . Here, the environmental conditions are conditions that the voice data generating apparatus 200 must satisfy in a situation where voice data is recorded. For example, environmental conditions include a place where the voice data generating device 200 should be located in a situation where voice data is recorded, a time period for recording voice data, and the congestion level of a place where the voice data generating device 200 is located, etc. may be included, but is not limited thereto.

The terminal communication unit 105 may transmit the recorded voice data to the voice data generating apparatus 200 . In addition, the terminal communication unit 105 may transmit an image captured in the process of recording the audio data to the audio data generating apparatus 200 .

With the following configuration, the terminal input/output unit 110 may receive a signal from the user of the voice data recording apparatus 100 through a user interface (UI), or may output a calculation result to the outside.

Specifically, the terminal input/output unit 110 may activate or deactivate a user interface (UI) capable of controlling the start and end of recording according to the instruction of the recording environment identification unit 115 or the voice recording control unit 120 . there is. For example, a user interface (UI) capable of controlling the start and end of recording may be an interface input/output through a touch screen of the voice data recording apparatus 100 .

The terminal input/output unit 110 may receive a command instructing the start of recording or a command instructing the end of the recording from the user through a user interface (UI) capable of controlling the start and end of the recording.

The terminal input/output unit 110 may output a script preset by the voice data generating apparatus 200 through the display. In this case, the script may be a script to be read by the user of the voice data recording apparatus 100 during the voice data recording process.

The terminal input/output unit 110 may collect location information at which the voice data recording apparatus 100 is located through a global positioning system (GPS). The terminal input/output unit 110 may receive voice data through a microphone. The terminal input unit 110 may time-series images through a camera.

In addition, the terminal input/output unit 110 may reproduce voice data through a speaker according to the user's control.

With the following configuration, the recording environment identification unit 115 may identify information about an environment in which the voice data recording apparatus 100 is located in order to record voice data.

Specifically, the recording environment identification unit 115 is a microphone of the terminal input/output unit 110 before the user interface (UI) capable of controlling the start and end of recording is activated through the microphone of the terminal input/output unit 110 . can receive background sound data through

In this case, the background sound data may be voice data in which the user's (ie, the speaker)'s voice is not included. Such background sound data may include acoustic echo and background noise.

The recording environment identification unit 115 may identify information about the recording environment based on the input background sound data. In this case, the information about the recording environment is information about the environment in which the voice data recording apparatus 100 can record voice data. For example, the information about the recording environment may include, but is not limited to, the magnitude and frequency of acoustic echo, and the magnitude and frequency of background noise.

Also, the recording environment identification unit 115 may collect location information where the voice data recording apparatus 100 is located through the GPS of the terminal input unit 110 . In addition, the recording environment identification unit 115 may further include the collected location information in the information about the recording environment.

With the following configuration, the voice recording controller 120 may control recording of voice data.

Specifically, the voice recording control unit 120 may determine whether the information about the recording environment identified by the recording environment identification unit 115 satisfies the environmental condition preset by the voice data generating apparatus 200 . .

In this case, the environmental conditions are conditions that the voice data recording apparatus 100 must satisfy in a situation where voice data is recorded. For example, the environmental conditions include a place where the voice data recording apparatus 100 should be located in a situation where voice data is recorded, a time period in which voice data recording can be performed, and the degree of congestion of the place where the voice data recording apparatus 100 is located, etc. may be included, but is not limited thereto.

That is, the voice recording control unit 120 determines whether the size and frequency of the acoustic echo and the size and frequency and location information of the background noise included in the information about the recording environment satisfy the location, time zone, and degree of congestion included in the environmental condition, etc. can be judged

For example, if it corresponds to an indoor required by an environmental condition, the voice recording controller 120 controls the voice data recording apparatus 100 based on the size of the acoustic reverberation included in the background sound data. It can be determined whether the position required by the

When the identified information on the recording environment satisfies the environmental condition, the voice recording control unit 120 may activate a user interface (UI) capable of controlling the start and end of the recording of the terminal input/output unit 110 .

In addition, when a command for instructing the start of recording is input from the user through a user interface (UI) capable of controlling the start and end of recording, the voice recording control unit 120 receives voice data through the microphone of the terminal input/output unit 110 . can be input.

In this case, two or more microphones of the voice data recording apparatus 100 may be configured, and the voice recording control unit 120 includes two or more voices simultaneously recorded through two or more microphones constituting the voice data recording apparatus 100 . It may contain signals.

Meanwhile, after a user interface (UI) capable of controlling the start and end of recording is activated, the voice recording controller 120 may re-collect the positions of the voice data recording apparatus 100 at preset intervals. When the size and frequency of the re-collected location information and the voice data do not satisfy the environmental conditions, the voice recording control unit 120 deactivates a user interface (UI) capable of controlling the start and end of the previously activated recording, or , or the input of the voice data being input may be stopped.

Also, the voice recording control unit 120 may output a script set in advance by matching the environmental conditions by the voice data generating apparatus 200 through the terminal input/output unit 110 . In this case, the script may be a script to be read by the user of the voice data recording apparatus 100 during the voice data recording process.

After the script is output through the display, the voice recording controller 120 may verify whether the result of STT (Speech To Text) on the input voice data and the script correspond to each other. In this case, the voice recording control unit 120 may remove a noise signal included in the voice data based on the background sound data received by the recording environment identification unit 115 . In addition, the voice recording controller 120 may perform STT on voice data from which the noise signal has been removed.

The voice recording control unit 120 may stop inputting voice data when, as a result of preparing for whether the STT result and the script correspond to each other, the number of syllables greater than or equal to a preset number are different from each other.

In addition, when a command for instructing the end of recording is input from the user through a user interface (UI) capable of controlling the start and end of recording, the voice recording control unit 120 receives voice data through the microphone of the terminal input/output unit 110 . input can be terminated.

On the other hand, the voice recording control unit 120 receives the user's voice data through the microphone of the terminal input/output unit 110 (that is, after the command instructing the start of recording is input, the command instructing the end of the recording is inputted) up to), it is also possible to time-series images including the user's face through the camera at preset intervals.

With the following configuration, the metadata granting unit 125 may provide metadata for the recorded voice data. In this case, the metadata provided by the voice data recording apparatus 100 does not have completeness, and may be used as basic data of the metadata to be provided by the voice data generating apparatus 200 .

Specifically, the metadata granting unit 125 uses the information on the recording environment identified by the recording environment identification unit 115 to generate metadata for voice data input through the voice recording control unit 120 . can create

In this case, when a command for instructing the end of recording is input from the user through a user interface (UI) capable of controlling the start and end of recording, the metadata granting unit 125 receives the information from the user through the terminal input/output unit 110 . You can receive information about your emotional state. Also, the metadata granting unit 125 may generate metadata by including information about the input emotional state.

Also, the metadata imparting unit 125 may further include an image captured in the process of receiving the user's voice data through the voice recording control unit 120 in the metadata.

In addition, the metadata imparting unit 125 may match the generated metadata with voice data and store it in a database.

With the following configuration, the collected data providing unit 130 may provide the collected voice data and metadata to the voice data generating apparatus 200 through the terminal communication unit 105 .

Specifically, the collected data providing unit 130 may immediately transmit the collected voice data and metadata to the voice data generating apparatus 200, but is not limited thereto, and the collected voice data and metadata are accumulated and stored in a database. Then, the accumulated and stored voice data and metadata may be transmitted to the voice data generating apparatus 200 every preset period.

Hereinafter, the configuration of the voice data generating apparatus 200 having the above-described characteristics will be described in more detail.

3 is a logical configuration diagram of an apparatus for generating voice data according to an embodiment of the present invention.

As shown in FIG. 3 , the apparatus 200 for generating voice data according to an embodiment of the present invention includes a server communication unit 205 , a server input/output unit 210 , a data structure design unit 215 , and a data collection and refinement unit ( 220 ), a data processing unit 225 , and a voice data providing unit 230 .

As such, the components of the voice data generating apparatus 200 merely represent functionally distinct elements, so that two or more components are integrated with each other in an actual physical environment, or one component is implemented with each other in an actual physical environment. It may be implemented separately.

When each component is described, the server communication unit 205 may transmit/receive data to and from one or more of the voice data recording apparatus 100 and the voice recognition learning apparatus 300 .

Specifically, a requirement for building an acoustic model may be received from the voice recognition learning apparatus 300 .

The server communication unit 205 may transmit environmental conditions and scripts to a plurality of voice data recording apparatuses 100 . The server communication unit 205 may receive voice data and metadata collected from a plurality of voice data recording apparatuses 100 .

In addition, the server communication unit 205 may transmit the processed voice data that can be used to construct reference patterns of the acoustic model to the voice recognition learning apparatus 300 .

With the following configuration, the server input/output unit 210 may receive a signal from the manager of the voice data generating apparatus 200 through a user interface (UI) or may output an operation result to the outside.

Specifically, the server input/output unit 210 may receive a control signal for designing a data structure of voice data from a manager. The server input/output unit 210 may receive a control signal for designing a data structure of metadata from a manager.

The server input/output unit 210 may receive, from a manager, an environmental condition that a plurality of voice data recording apparatuses 100 must satisfy in a process of recording voice data.

In addition, the server input/output unit 210 may receive a script readable by the speakers of the plurality of voice data recording apparatuses 100 from the manager.

With the following configuration, the data structure design unit 215 may design a data structure of voice data and metadata that may be a reference pattern of an acoustic model.

Specifically, the data structure design unit 215 may be a reference pattern of the acoustic model based on a control signal input from the manager through the server input/output unit 210 or a requirement received from the voice recognition learning apparatus 300 . The data structure of voice data and metadata can be designed.

The data structure design unit 215 may define an ontology for an acoustic model of speech recognition, a classification system of speech data and metadata, and the like, based on a control signal input from the manager or a received requirement.

With the following configuration, the data collection and refinement unit 220 may collect and refine voice data and metadata based on the data structure designed by the data structure design unit 215 .

Specifically, the data collection and refinement unit 220 may transmit environmental conditions and scripts to the plurality of voice data recording apparatuses 100 through the server communication unit 205 . The data collection and refinement unit 220 may receive voice data and metadata from a plurality of voice data recording apparatuses 100 through the server communication unit 205 .

However, the present invention is not limited thereto, and the data collection and refinement unit 220 performs web crawling through the server communication unit 205 to directly collect voice data, or from an external device holding voice data. You can also download voice data.

The data collection and refinement unit 220 may remove duplicate or extremely similar voice data from among the collected voice data. In addition, the data collection and refinement unit 220 may de-identify personal information included in the collected voice data.

With the following configuration, the data processing unit 225 is based on the voice data and metadata collected and refined by the data collection and refinement unit 220, voice data and meta data that can be utilized to build reference patterns of the acoustic model. data can be processed.

Characteristically, the data processing unit 225 may disassemble and manage a plurality of voice data into detailed components, and may generate various types of voice data in response to the requirements of the voice recognition learning apparatus 300 . In addition, the data processing unit 225 may set a standard for distinguishing between a simple conversation voice and a calling word, and may also generate various types of voice data for learning a calling word based on this. Furthermore, the data processing unit 225 may also generate metadata for recognizing the emotional state of the speaker.

A detailed description of the data processing unit 225 having such a characteristic will be described later with reference to FIG. 4 .

With the following configuration, the voice data providing unit 230 may provide the voice data and metadata processed by the data processing unit 225 to the voice recognition learning apparatus 300 . In this case, the voice data and metadata processed by the data processing unit 225 may be voice data that can be utilized to construct reference patterns of the acoustic model.

Hereinafter, the data processing unit 225 having the above-described characteristics will be described in more detail.

4 is a detailed configuration diagram of a data processing unit according to an embodiment of the present invention.

As shown in FIG. 4 , the data processing unit 225 of the voice data generating apparatus 200 includes a voice data processing module 225-1, a call word processing module 225-2, and a metadata processing module 225- 3) may be included.

As such, the components of the data processing unit 225 merely represent functionally distinct elements, so that two or more components are integrated with each other in the actual physical environment, or one component is separated from each other in the actual physical environment. and can be implemented.

Each of the detailed components will be described. The voice data processing module 225 - 1 may decompose and accumulate the collected voice data, and may generate various voice data based on the decomposed and stored voice data.

First, a feature in which the voice data processing module 225-1 decomposes and accumulates and stores voice data will be described.

The voice data processing module 225 - 1 may analyze a waveform of the collected voice data to separate background sound data from the voice data.

Here, the collected voice data may be voice data collected through the voice data recording apparatus 100 in order to construct reference patterns to be included in the acoustic model of voice recognition.

Also, the background sound data may be voice data that does not include a user's (ie, a speaker) voice. Such background sound data may include acoustic echo and background noise.

More specifically, the voice data processing module 225 - 1 applies a frequency domain adaptive filter (FDAF) to the collected sound data so as to be used between the speaker and the microphone of the voice data recording apparatus 100 related to the voice data. A transfer function can be identified.

In addition, the voice data processing module 225 - 1 may generate an echo signal from the voice data using the identified transfer function, and then subtract the echo signal from the voice data to remove the echo signal included in the voice data.

That is, the voice data processing module 225 - 1 may separate the echo signal from the voice data using a transfer function.

The voice data processing module 225 - 1 performs the voice data recording apparatus 100 and the speaker (ie, the voice data recording apparatus) based on a Time Difference of Arrival (TDoA) of two or more voice signals included in the voice data. You can estimate the directionality between (100) users.

The voice data processing module 225 - 1 may generate a noise signal from the voice data based on the estimated directionality.

More specifically, the voice data processing module 225-1 maintains a preset constant value in which the gain of applying a Minimum Variance Distortionless Response (MVDR) filter to two or more voice signals is maintained. In , it is possible to generate a blocking matrix (BM) designed to minimize the final output power of two or more voice signals.

The voice data processing module 225-1 may generate a noise signal in which voice components in only a target direction are removed from the voice data by passing the voice data through the blocking matrix BM. In this case, the target direction is a relative direction in which the speaker is located with respect to the voice data recording apparatus 100 (ie, the direction between the voice data recording apparatus 100 and the speaker).

In addition, the voice data processing module 225 - 1 may remove the noise signal included in the voice data by subtracting the noise signal from the voice data from which the echo signal has been removed.

That is, the voice data processing module 225 - 1 may separate the noise signal from the voice data based on the directionality between the voice data recording apparatus 100 and the speaker.

In this case, the voice data processing module 225-1 determines that the sum of the weighted cross correlations for the arrival time difference (TDoA) of two or more voice signals is the maximum until the reproduction of the voice data is finished. It is possible to control the destination direction for the voice data so as to remain in the state.

On the other hand, when the voice data processing module 225-1 includes an image including the speaker's face in the metadata of the voice data, the voice data processing module 225-1 controls the mouth of the speaker in the video. Based on whether the speaker is open or closed, a noise section and a voice section can be distinguished from each other in the voice data of the corresponding speaker. In addition, the voice data processing module 225 - 1 may remove the noise signal included in the voice data by subtracting the signal of the noise section from the signal of the differentiated voice section.

The voice data processing module 225 - 1 may identify man-made noise from voice data from which background sound data is separated.

In this case, the artificial noise may be a phoneme that is uttered by the speaker according to the speaker's utterance method, exclamation or chuimsae of the voice data, but does not have a word meaning.

For example, in the phrase "Um, that's an apple", "um" is a phoneme that is uttered according to the speaker's utterance method, but does not have a special word meaning, and may correspond to artificial noise.

More specifically, the voice data processing module 225 - 1 may STT the voice data from which the background sound data is separated to generate a transcript. The voice data processing module 225 - 1 may parse the generated transcript to identify a plurality of tokens constituting the transcript. The voice data processing module 225 - 1 may perform semantic analysis on each identified token. Specific means for performing semantic analysis on the parsed token are not described as they are the same as widely known.

As a result of semantic analysis, the speech data processing module 225 - 1 may identify the unclassified corresponding token as artificial noise when there is a token whose word meaning is not classified as a phoneme defined in advance.

In addition, the voice data processing module 225-1 identifies the country, language, region, gender and age of the speaker from the metadata of the speaker of the voice data, and identifies the country, language, region, gender and age of the speaker. As a basis, we may verify that the artificial noise identified from the unclassified token is correct.

In addition, the voice data processing module 225-1 may separate only the identified artificial noise from the voice data.

Meanwhile, the voice data processing module 225-1 is configured to match the voice data from which the echo signal, the noise signal, the artificial noise, and the artificial noise have been removed by the metadata processing module 225-3 to match the metadata for the voice data. When is generated, the generated voice data and metadata may be accumulated and stored in a database.

Next, a feature in which the voice data processing module 225-1 generates new voice data by synthesizing voice signals will be described.

The voice data processing module 225-1 may identify the country, language, region, gender, and age of a speaker that satisfies a preset environmental condition in relation to voice data to be synthesized. In this case, the speaker may not be the user of the voice data recording apparatus 100 but a virtual person who can be identified as uttering the voice data.

The voice data processing module 225-1 identifies a plurality of voice signals corresponding to a preset script in relation to voice data to be synthesized from the database based on the country, language, region, gender and age of the identified speaker. can do.

More specifically, the voice data processing module 225-1 may parse the script to identify a plurality of tokens constituting the script. The voice data processing module 225 - 1 may perform semantic analysis on each identified token.

As a result of performing semantic analysis on each token, the voice data processing module 225-1 generates a voice signal corresponding to each token when there is a token whose word meaning is not classified into a predefined phoneme. In the process of identification, the corresponding token can identify a voice signal classified as artificial noise.

On the other hand, the voice data processing module 225-1 performs semantic analysis on each token, and as a result of performing semantic analysis, when one or more tokens predefined as expressing an emotion exist, based on the one or more tokens It is possible to estimate one emotional state that the speaker may have.

The voice data processing module 225-1 may identify one or more phonemes that correspond to the estimated emotional state and can generate artificial noise from the database, and randomly add the identified one or more phonemes to the script.

The speech data processing module 225 - 1 may generate speech data in units of word segments by combining a plurality of speech signals identified as corresponding to each token.

In this case, the voice data processing module 225-1 identifies one or more phonemes that can generate artificial noise corresponding to the speaker's country, language, region, gender, and age, and generates word-based voice data. A voice signal corresponding to one or more phonemes identified as capable of generating artificial noise may be added.

In addition, the voice data processing module 225-1 may identify background sound data that satisfies a preset environmental condition in relation to the voice data to be synthesized, and synthesize the identified background sound data into voice data in units of words. there is.

More specifically, the voice data processing module 225 - 1 may identify a noise signal and an echo signal satisfying an environmental condition from the database, and then sequentially synthesize the noise signal and the echo signal into the word unit voice data.

Also, the voice data processing module 225-1 may identify an inaudible signal that satisfies the environmental condition from the database, and then additionally synthesize the inaudible signal into the voice data in which the noise signal and the echo signal are synthesized. there is.

Meanwhile, the voice data processing module 225-1 may adjust the volume for each channel of the voice signal included in the voice data synthesized up to the inaudible signal so that the direction of the speaker is changed during the reproduction of the voice data.

Next, a feature in which the voice data processing module 225-1 generates new voice data based on the previously decomposed and stored voice data will be described.

The voice data processing module 225 - 1 may identify voice data from which all noise has been removed from the database. In this case, the voice data from which all noises have been removed may be voice data from which artificial noises, echo signals, and noise signals have been removed from the voice data recorded by the voice data recording apparatus 100 .

The voice data processing module 225 - 1 may identify the speaker who recorded the voice data based on metadata matched to the identified voice data.

The voice data processing module 225 - 1 may identify artificial noise corresponding to the identified speaker, and synthesize the identified artificial noise into voice data.

More specifically, the voice data processing module 225 - 1 may STT the voice data to generate a transcript. The voice data processing module 225-1 may parse the generated transcript to identify a plurality of tokens constituting the transcript. The voice data processing module 225 - 1 may perform semantic analysis on each identified token.

As a result of performing semantic analysis on each token, the voice data processing module 225-1 performs semantic analysis on each token. As a result, when one or more tokens predefined as expressing emotions exist, the speaker can have based on the one or more tokens. One emotional state can be estimated.

The voice data processing module 225-1 may identify one or more phonemes that correspond to the estimated emotional state and may generate artificial noise from the database, and synthesize the identified one or more phonemes into voice data.

Alternatively, the voice data processing module 225-1 identifies the country, language, region, gender and age of the speaker from the metadata about the speaker, and responds to the identified country, language, region, gender and age of the speaker. One or more phonemes capable of generating artificial noise may be identified, and the identified one or more phonemes may be synthesized into voice data.

Alternatively, the voice data processing module 225-1 identifies one or more phonemes that may generate artificial noise related to another speaker that is different from the speaker in at least one of country, language, region, gender, and age, and One or more phonemes may be synthesized into voice data.

In addition, the voice data processing module 225-1 identifies background sound data that satisfies a preset environmental condition in relation to voice data to be generated, and synthesizes the identified background sound data into voice data synthesized up to artificial noise. can do.

More specifically, the voice data processing module 225 - 1 may identify a noise signal and an echo signal satisfying an environmental condition from the database, and then sequentially synthesize the noise signal and the echo signal into the word unit voice data.

Also, the voice data processing module 225 - 1 may identify an inaudible signal that satisfies the environmental condition from the database, and then additionally synthesize the inaudible signal into the voice data in which the noise signal and the echo signal are synthesized.

Meanwhile, the voice data processing module 225-1 may adjust the volume for each channel of the voice signal included in the voice data synthesized up to the inaudible signal so that the direction of the speaker is changed during the reproduction of the voice data.

With the following configuration, the calling word processing module 225 - 2 may set a standard for discriminating between a simple conversation voice and a calling word, and may generate various types of voice data for learning the calling word.

Specifically, the calling word processing module 225 - 2 may identify voice data classified as a calling word among voice data stored in the database. In this case, the calling word may be a word preset to activate an artificial intelligence (AI) system that is in a standby status.

To this end, the calling word processing module 225 - 2 may determine whether the voice data is classified as a calling word based on the metadata for the voice data.

Next, the calling word processing module 225 - 2 is configured to include, among voice data of the same speaker as the speaker of the voice data classified as the calling word, the voice data classified as the calling word and the syllable are the same but not classified as the calling word Data can be identified from the database.

If, from the database, the voice data classified as the calling word and the syllable are the same, but the voice data that is not classified as the calling word cannot be identified, the calling word processing module 225-2 may combine the syllable and the calling word with a combination of two or more phonemes. To compose the same syllable, by combining one or more phonemes that are different from the syllable of the calling word but not classified as the calling word, respectively, the voice data classified as the calling word and the syllable are the same and not classified as the calling word. It is also possible to generate non-voice data.

Next, the calling word processing module 225-2 compares the waveform of the voice data classified as the calling word with the waveform of the voice data not classified as the calling word with each other, and the voice data classified as the calling word and the voice data not classified as the calling word The difference between the values can be calculated.

In this case, the difference value between the voice data classified as the calling word and the voice data not classified as the calling word may consist of a difference between the length of the waveform, the amplitude of the waveform, and the frequency of the waveform, but is not limited thereto.

Next, the call word processing module 225 - 2 may add the calculated difference value to the previously generated metadata about the speaker and store the difference value in the database. In particular, the call word processing module 225 - 2 may match the calculated difference value with the speaker's attribute and accumulate and store the same in the database. In this case, the speaker's attributes may include the speaker's country, language, region, gender, and age.

Meanwhile, the call word processing module 225 - 2 may perform correlation analysis on the difference value accumulated and stored in the database and the speaker's attribute to evaluate the correlation between the difference value and the speaker's attribute. The call word processing module 225-2 performs factor analysis and regression on the evaluated relevance to identify the factor that has the greatest influence on the difference value among the factors included in the speaker's attribute. can

In addition, the call word processing module 225 - 2 may set a factor that has the greatest influence on the difference value as a criterion for distinguishing between a call word and a non-call word for the speaker. In addition, the call word processing module 225 - 2 may store the set discrimination criterion by adding it to metadata about the speaker.

Next, the calling word processing module 225 - 2 may identify one of voice data composed of syllables having a size that can be registered as a calling word from among voice data uttered by the same speaker and not classified as a calling word. In this case, the registerable size of the call word may be preset by the voice data generating apparatus 200 or the voice recognition learning apparatus 300 .

The calling word processing module 225 - 2 may generate new voice data that can be classified as a calling word by applying the speaker difference value to the voice data identified as being composed of syllables of the calling word registerable size.

The call word processing module 225-2 identifies one echo signal based on the metadata of the voice data identified as being composed of syllables of the call word registerable size, and then converts the identified echo signal into a new voice to which the difference value is applied. data can be synthesized.

Then, the calling word processing module 225-2 randomly selects a plurality of noise signals included in the database, and then adds each randomly selected noise signal to new voice data synthesized up to an echo signal to recognize the calling word. It is possible to construct a plurality of reference patterns for

With the following configuration, the metadata processing module 225-3 may provide metadata to the voice data processed by the voice data processing module 225-1 and the call word processing module 225-2.

Specifically, the metadata processing module 225-3 is configured to match the voice data from which the echo signal, the noise signal, the artificial noise, and the artificial noise separated by the voice data processing module 225-1 have been removed to match each other. You can create metadata for

In addition, the metadata processing module 225 - 3 may match the generated metadata with the voice data and store it in the database.

On the other hand, when the metadata processing module 225-3 receives an image including the speaker's face from the voice data recording apparatus 100 through the server input/output unit 205, the change in the speaker's face included in the image is the standard. , it is possible to estimate the speaker's emotional state.

More specifically, the metadata processing module 225-3 may track movements of the speaker's eyes, nose, and mouth from an image including the speaker's face. The metadata processing module 225 - 3 may query artificial intelligence (AI) based on the tracked movements of the eyes, nose, and mouth of the talker to estimate the speaker's emotional state. In this case, the artificial intelligence (AI) may be an artificial intelligence machine learned based on the facial images of the existing multiple speakers.

The metadata processing module 225 - 3 may STT the voice data to generate a transcript. The metadata processing module 225 - 3 may parse the generated transcript to identify a plurality of tokens constituting the transcript. The metadata processing module 225 - 3 may perform semantic analysis on each identified token.

As a result of performing semantic analysis on each token, the metadata processing module 225-3 performs artificial intelligence (AI) based on the one or more tokens when one or more tokens predefined as expressing emotions exist. ), it is possible to verify whether the estimated emotional state is accurate.

In addition, the metadata processing module 225-3 performs semantic analysis on each token. As a result, when there is a token whose word meaning is not classified as a phoneme defined in advance, the unclassified corresponding token is artificially analyzed. can be identified as noise. In addition, the metadata processing module 225 - 3 may verify whether the emotional state estimated by the artificial intelligence (AI) is accurate, based on the identified attribute of the artificial noise.

Meanwhile, the metadata processing module 225 - 3 may evaluate the speaker's concentration level in a process in which voice data is input (ie, a recording process) based on the tracked eye movement of the talker. In addition, the metadata processing module 225 - 3 may generate metadata by further including the evaluated speaker's concentration level.

In addition, the metadata processing module 225-3 may generate metadata for the voice data by including information on the emotional state estimated by artificial intelligence (AI). In this case, the metadata processing module 225-3 extracts the pause image corresponding to the estimated emotional state from the image taken in time series by the audio data recording apparatus 100, and further adds the extracted pause image. You can also include it to create metadata.

Hereinafter, hardware for implementing the logical components of the apparatus 200 for generating voice data as described above will be described in more detail.

5 is a hardware configuration diagram of an apparatus for generating voice data according to an embodiment of the present invention.

As shown in FIG. 5 , the voice data generating apparatus 200 includes a processor 250 , a memory 255 , a transceiver 260 , an input/output device 265 , and a data bus. (Bus, 270) and storage (Storage, 275) may be configured to include.

The processor 250 may implement the operation and function of the voice data generating apparatus 200 based on a command according to the software 280a in which the method according to the embodiments of the present invention is implemented residing in the memory 255 . . In the memory 255, the software 280a in which the method according to the embodiments of the present invention is implemented may be loaded. The transceiver 260 may transmit/receive data to and from the voice data recording apparatus 100 and the voice recognition learning apparatus 300 . The input/output device 265 may receive data necessary for the operation of the voice data generating device 200 , and may output processed voice data and metadata. The data bus 270 is connected to the processor 250 , the memory 255 , the transceiver 260 , the input/output device 265 , and the storage 275 . can play a role.

The storage 275 stores an application programming interface (API), a library file, a resource file, etc. necessary for the execution of the software 280a in which the method according to the embodiments of the present invention is implemented. can be saved The storage 275 may store the software 280b in which the method according to the embodiments of the present invention is implemented. In addition, the storage 275 may include a database in which information necessary for performing the method according to the embodiments of the present invention, voice data, and metadata are stored.

According to an embodiment of the present invention, the software (280a, 280b) for implementing the method of decomposing speech data resident in the memory 255 or stored in the storage 275 is included in the processor 250 to be included in the acoustic model of speech recognition. Separating the background sound data from the voice data by analyzing the waveform of the collected voice data to build reference patterns, the processor 250 identifying artificial noise from the voice data from which the background sound data is separated, and the processor ( 250) may be a computer program recorded on a recording medium to execute the step of separating only the identified artificial noise from the voice data.

According to another embodiment of the present invention, the software (280a, 280b) for implementing the voice data synthesis method resident in the memory 255 or stored in the storage 275 satisfies the environmental conditions set in advance by the processor 250 identifying the country, language, region, gender and age of the speaker, the processor 250 identifies a plurality of voice signals corresponding to a preset script based on the identified country, language, region, gender and age step, the processor 250 combining the plurality of voice signals to generate word-based voice data, and the processor 250 identifying background sound data that satisfies an environmental condition, and using the identified background sound data It may be a computer program recorded on a recording medium in order to execute the step of synthesizing the speech data in word unit.

According to another embodiment of the present invention, the software (280a, 280b) for implementing the calling word learning method resident in the memory 255 or stored in the storage 275 is stored in a database in which the processor 250 is provided in advance. Recognizing, by the processor 250, voice data classified as a calling word from among voice data, the processor 250 classifies as a calling word even though the voice data classified as a calling word has the same syllable as the calling word among voice data of the same speaker as the speaker of the voice data classified as the calling word The step of identifying the not-noted voice data, the processor 250 comparing the waveform of the voice data classified as the calling word with the waveform of the voice data not classified as the calling word with each other to compare the voice data classified as the calling word and the voice data not classified as the calling word The computer program may be a computer program recorded on a recording medium to execute the steps of calculating the difference value between them, and the processor 250 adding the difference value to metadata about the speaker and storing the difference value in the database.

According to another embodiment of the present invention, the software (280a, 280b) for implementing the spontaneous speech generation method resident in the memory 255 or stored in the storage 275 is the processor 250 from a database provided in advance. Identifying voice data from which all noise has been removed, the processor 250 identifying the speaker who recorded the voice data based on metadata matched to the voice data, the processor 250 artificially corresponding to the identified speaker identifying noise and synthesizing the identified artificial noise into voice data, and the processor 250 identifies background sound data corresponding to a preset environmental condition, and background sound data identified in the artificial noise synthesized voice data It may be a computer program recorded on a recording medium to execute the step of synthesizing the .

According to another embodiment of the present invention, the software (280a, 280b) for implementing the metadata grant method resident in the memory 255 or stored in the storage 275 is the processor 250 via the transceiver 260. Receive voice data and images from the voice data recording device 100, but the images are of the talker captured in time series through a camera while the voice data recording device 100 receives the talker’s voice data through a microphone The step of the image including the face, the processor 250 estimating the emotional state of the speaker based on the change of the face included in the image, and the processor 250 including information about the estimated emotional state It may be a computer program recorded on a recording medium to execute the step of generating metadata regarding the input voice data and storing the generated metadata by matching with the voice data.

More specifically, the processor 250 may include an application-specific integrated circuit (ASIC), another chipset, a logic circuit, and/or a data processing device. The memory 255 may include read-only memory (ROM), random access memory (RAM), flash memory, memory cards, storage media, and/or other storage devices. The transceiver 260 may include a baseband circuit for processing wired and wireless signals. The input/output device 265 includes an input device such as a keyboard, a mouse, and/or a joystick, and a liquid crystal display (LCD), an organic light emitting diode (OLED) and/or an input device such as a joystick. Alternatively, an image output device such as an active matrix OLED (AMOLED) may include a printing device such as a printer or a plotter.

When the embodiment included in this specification is implemented in software, the above-described method may be implemented as a module (process, function, etc.) that performs the above-described function. Modules reside in memory 255 and may be executed by processor 250 . The memory 255 may be internal or external to the processor 250 , and may be coupled to the processor 250 by various well-known means.

Each component shown in FIG. 5 may be implemented by various means, for example, hardware, firmware, software, or a combination thereof. In the case of implementation by hardware, an embodiment of the present invention provides one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs ( Field Programmable Gate Arrays), a processor, a controller, a microcontroller, a microprocessor, etc. may be implemented.

In addition, in the case of implementation by firmware or software, an embodiment of the present invention is implemented in the form of a module, procedure, function, etc. that performs the functions or operations described above, and is stored in a recording medium readable through various computer means. can be recorded. Here, the recording medium may include a program command, a data file, a data structure, etc. alone or in combination. The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. For example, the recording medium includes a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a compact disk read only memory (CD-ROM), a digital video disk (DVD), and a floppy disk. Magneto-Optical Media, such as a disk, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions may include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes such as those generated by a compiler. Such hardware devices may be configured to operate as one or more software to perform the operations of the present invention, and vice versa.

Hereinafter, the features of the voice data collection system according to various embodiments of the present invention as described above will be described in detail with reference to the drawings.

6 and 7 are exemplary views for explaining a process of decomposing voice data according to an embodiment of the present invention.

6 and 7 , the voice data generating apparatus 200 may decompose and store the voice data 10 collected from the plurality of voice data recording apparatuses 100 .

Specifically, the voice data generating apparatus 200 may analyze the waveform of the voice data 10 collected by the voice data recording apparatus 100 to separate the background sound data 20 from the voice data 10 . . As such, the background sound data 20 may include acoustic echo and background noise.

First, the voice data generating apparatus 200 applies a frequency domain filter (FDAF) to the collected sound data 10 to obtain a transfer function between the speaker and the microphone of the voice data recording apparatus 100 related to the voice data 10 . can be identified.

The voice data generating apparatus 200 generates an echo signal from the voice data 10 using the identified transfer function, and then subtracts the echo signal from the voice data 10 to generate the echo signal included in the voice data 10 . can be removed

The voice data generating apparatus 200 is a voice data recording apparatus 100 and the speaker (ie, the voice data recording apparatus 100) based on the arrival time difference (TDoA) of two or more voice signals included in the voice data 10 . direction between users) can be estimated.

The voice data generating apparatus 200 may minimize the final output power of two or more voice signals while maintaining the gain of applying the minimum variance non-distortion response (MVDR) filter to two or more voice signals as a preset constant value. It is possible to create a blocking matrix (BM) designed to be

The voice data generating apparatus 200 may generate a noise signal from which a voice component in a target direction is removed from the voice data by passing the voice data 10 from which the echo signal is removed through the blocking matrix BM.

Also, the voice data generating apparatus 200 may subtract the noise signal from the voice data from which the echo signal has been removed to remove the noise signal included in the voice data.

Next, the voice data generating apparatus 200 may identify the artificial noise 32 from the voice data 30 from which the background sound data 20 is removed. In this case, the artificial noise 32 may be a phoneme that is uttered by the speaker according to the speaker's utterance method, exclamation or chuimsae of the voice data, but does not have a word meaning.

First, the voice data generating apparatus 200 may generate a transcript by performing STT on the voice data 30 from which the background sound data 20 is separated. The voice data generating apparatus 200 may parse the generated transcript to identify a plurality of tokens constituting the transcript. The voice data generating apparatus 200 may perform semantic analysis on each identified token.

As a result of performing semantic analysis, the voice data generating apparatus 200 may identify the unclassified corresponding token as artificial noise 32 when there is a token whose word meaning is not classified as a phoneme defined in advance. .

In addition, the voice data generating apparatus 200 identifies the country, language, region, gender and age of the speaker from the metadata of the speaker of the voice data 10 , and the country, language, region, gender and age of the identified speaker Based on , it may be verified that the artificial noise 32 identified from the unclassified token is correct.

The voice data generating apparatus 200 may separate only the identified artificial noise 32 from the voice data 30 and generate the voice data 34 from which all noises are removed.

In addition, the voice data generating apparatus 200 generates metadata such that the separated echo signal, the noise signal, the artificial noise 32, and the voice data 34 from which all noises have been removed match each other, and the generated metadata and voice Data can be stored in a database.

8 and 9 are exemplary views for explaining a process of generating voice data according to an embodiment of the present invention.

Referring to FIG. 8 , the apparatus 200 for generating voice data according to an embodiment of the present invention may generate new voice data by synthesizing voice signals.

Specifically, the voice data generating apparatus 200 may identify a plurality of voice signals 40 corresponding to a preset script in relation to voice data to be synthesized from the database.

To this end, the voice data generating apparatus 200 may parse the script to identify a plurality of tokens constituting the script. In addition, the voice data generating apparatus 200 may generate the voice data 30 by combining a plurality of voice signals 40 identified as corresponding to each token.

In this case, as a result of performing semantic analysis on each token, the voice data generating apparatus 200 determines that if one or more tokens predefined as expressing emotions exist, the speaker will have One possible emotional state can be estimated. In addition, the voice data generating apparatus 200 may identify one or more phonemes that correspond to the estimated emotional state and can generate artificial noise from the database, and randomly add the identified one or more phonemes to the script.

Then, the voice data generating apparatus 200 identifies the background sound data 20 satisfying a preset environmental condition in relation to the voice data 10 to be synthesized, and is generated by combining the voice signals 40 . The background sound data 20 may be synthesized with the voice data 30 .

Referring to FIG. 9 , the apparatus 200 for generating voice data according to another embodiment of the present invention may generate new voice data based on previously decomposed and stored voice data.

Specifically, the voice data generating apparatus 200 may identify the voice data 34 from which all noise is removed from the database. In this case, the voice data 34 from which all noise has been removed may be voice data from which the artificial noise 32 , the echo signal, and the noise signal have been removed from the voice data recorded by the voice data recording apparatus 100 .

The voice data generating apparatus 200 may identify the speaker who recorded the voice data based on metadata matched to the identified voice data 34 . The voice data generating apparatus 200 may identify an artificial noise 32 corresponding to the identified speaker and synthesize the identified artificial noise 32 into the voice data 34 .

In this case, the voice data generating apparatus 200 may STT the voice data 34 to generate a transcript. The voice data generating apparatus 200 may parse the generated transcript to identify a plurality of tokens constituting the transcript. The voice data generating apparatus 200 may perform semantic analysis on each identified token.

As a result of performing semantic analysis on each token, the voice data generating apparatus 200 performs a semantic analysis on each token. As a result, when one or more tokens predefined as expressing emotions exist, one or more tokens that the speaker can have based on the one or more tokens. can estimate the emotional state of In addition, the voice data generating apparatus 200 identifies one or more phonemes that correspond to the estimated emotional state and can generate artificial noise 32 from the database, and synthesizes the identified one or more phonemes into the voice data 34 . can

Then, the voice data generating apparatus 200 identifies the background sound data 20 satisfying a preset environmental condition in relation to the voice data 10 to be generated, and generates it from the previously decomposed and stored voice data 34 . The background sound data 20 may be synthesized with the voice data 30 .

10 is an exemplary diagram for explaining a process of learning a calling word according to an embodiment of the present invention.

As shown in FIG. 10 , the apparatus 200 for generating voice data according to an embodiment of the present invention sets a standard for distinguishing between a simple conversation voice and a calling word, and various types of voices for learning the calling word. data can be created.

Specifically, the voice data generating apparatus 200 may identify voice data 30_w classified as a calling word from among voice data stored in the database. In this case, the call word may be a word preset to activate the artificial intelligence (AI) system in the standby state.

The voice data generating apparatus 200 includes voice data that has the same syllable as the voice data 30_w classified as a calling word, but is not classified as a calling word, among voice data of the same speaker as the speaker of the voice data 30w classified as a calling word. 30_g) can be identified from the database.

If, from the database, the voice data 30_w classified as the calling word and the syllable are the same, but the voice data 30_g that is not classified as the calling word cannot be identified, the voice data generating apparatus 200 calls a combination of two or more phonemes. The voice data 30_w classified as the calling word and the syllable by combining one or more phonemes that are different from the syllable of the calling word but composing the voice data that are not classified as the calling word, respectively, to form the same syllable as the syllable of the calling word It is also possible to generate the same voice data 30_g that is not classified as a calling word.

The voice data generating apparatus 200 compares the waveform of the voice data 30_w classified as the calling word with the waveform of the voice data 30_g that is not classified as the calling word, and classifies the voice data 30_w classified as the calling word and the calling word. It is possible to calculate a difference value between the unresolved voice data 30_g.

In this case, the difference value between the voice data classified as the calling word (30_w) and the voice data not classified as the calling word (30_g) is the length of the waveform (T_w - T_g), the amplitude of the waveform (A_w - A_g), and the frequency of the waveform ( It may be composed of a difference between F_w - F_g), but is not limited thereto.

In addition, the voice data generating apparatus 200 may add the calculated difference value to the previously generated metadata about the speaker and store it in the database.

Meanwhile, the voice data generating apparatus 200 may identify one of voice data composed of syllables having a size that can be registered as a calling word from among voice data uttered by the same speaker and not classified as a calling word. In this case, the registerable size of the call word may be preset by the voice data generating apparatus 200 or the voice recognition learning apparatus 300 .

The voice data generating apparatus 200 may generate new voice data that can be classified as a calling word by applying a speaker difference value to voice data identified as being composed of syllables having a size that can be registered as a calling word.

The voice data generating apparatus 200 identifies one echo signal on the basis of the metadata of the voice data identified as being composed of syllables with a size that can be registered as a call word, and then applies the identified echo signal to the new voice data to which the difference value is applied. can be synthesized.

In addition, the voice data generating apparatus 200 randomly selects a plurality of noise signals included in the database, and then adds each randomly selected noise signal to new voice data synthesized up to an echo signal to recognize a call word. A plurality of reference patterns may be constructed.

11 is an exemplary diagram for explaining a process of recording voice data according to an embodiment of the present invention.

As shown in FIG. 11 , the voice data recording apparatus 100 records voice data only when the voice data recording apparatus 100 is located in a place that satisfies the environmental conditions indicated by the voice data generating apparatus 200 . can be controlled to perform.

Specifically, the voice data recording apparatus 100 may receive background sound data through a microphone before the user interface 150 capable of controlling the start and end of recording is activated. The voice data recording apparatus 100 may collect location information where the voice data recording apparatus 100 is located through GPS.

The voice data recording apparatus 100 may identify information about the recording environment based on the input background sound data and location information. In this case, the information about the recording environment is information about the environment in which the voice data recording apparatus 100 can record voice data.

The voice data recording apparatus 100 may determine whether the information about the recording environment satisfies the environmental condition 50 preset by the voice data generating apparatus 200 . In this case, the environmental conditions 50 are conditions that the voice data recording apparatus 100 must satisfy in a situation of recording voice data.

When the information about the recording environment satisfies the environmental condition, the voice data recording apparatus 100 may activate the user interface 150 capable of controlling the start and end of the recording.

The voice data recording apparatus 100 may receive voice data through a microphone when a command for instructing the start of recording is input from the user through the user interface 150 capable of controlling the start and end of the recording.

On the other hand, the voice data recording apparatus 100 may re-collect the location of the voice data recording apparatus 100 every preset period after a user interface (UI) capable of controlling the start and end of recording is activated. . The voice data recording apparatus 100 deactivates the user interface 150 capable of controlling the start and end of the previously activated recording when the re-collected location information, the size and frequency of the voice data do not satisfy the environmental conditions. Alternatively, the input of the voice data being input may be stopped.

The voice data recording apparatus 100 may end the input of voice data through the microphone when a command for instructing the end of the recording is input from the user through the user interface 150 capable of controlling the start and end of the recording.

The voice data recording apparatus 100 may generate metadata for voice data by using information about the recording environment. In addition, the voice data recording apparatus 100 may match the generated metadata with the voice data, store it in a database, and then transmit it to the voice data generating apparatus 200 .

12 is an exemplary diagram for explaining a process of generating metadata using a speaker's face according to an embodiment of the present invention.

11 , the voice data generating apparatus 200 may generate metadata for recognizing the emotional state of the speaker.

Specifically, the voice data recording apparatus 100 receives the user's voice data through the microphone (that is, after a command instructing to start recording is input until a command instructing end of recording is input), in advance An image including the user's face 60 may be captured in time series through the camera at every cycle set in .

When an image including the speaker's face 60 is received from the voice data recording apparatus 100, the voice data generating apparatus 200 determines the speaker's emotional state based on the change in the speaker's face 60 included in the image. can be estimated

To this end, the voice data generating apparatus 200 may track movements of the talker's eyes, nose, and mouth from the image including the talker's face 60 . The voice data generating apparatus 200 may query an artificial intelligence (AI) based on the tracked movements of the eyes, nose, and mouth of the talker to estimate the speaker's emotional state. In this case, the artificial intelligence (AI) may be an artificial intelligence machine learned based on the facial images of the existing multiple speakers.

In addition, the voice data generating apparatus 200 may generate metadata for the voice data 34 by including information about the emotional state estimated by artificial intelligence (AI).

Meanwhile, the voice data generating apparatus 200 may evaluate the speaker's concentration level in a process in which voice data is input (ie, a recording process) based on the tracked eye movement of the talker. Also, the voice data generating apparatus 200 may generate metadata by further including the evaluated speaker's concentration level.

Hereinafter, the operation of the components of the voice data collection system according to various embodiments of the present invention as described above will be described in detail with reference to the drawings.

13 is a flowchart illustrating a voice data recording method according to an embodiment of the present invention.

Referring to FIG. 13 , the voice data recording apparatus 100 may receive background sound data through a microphone before the user interface 150 capable of controlling the start and end of recording is activated ( S110 ). Also, the voice data recording apparatus 100 may collect location information where the voice data recording apparatus 100 is located through GPS.

The voice data recording apparatus 100 may identify information about the recording environment based on the input background sound data and location information ( S120 ). In this case, the information about the recording environment is information about the environment in which the voice data recording apparatus 100 can record voice data.

The voice data recording apparatus 100 may determine whether the information on the recording environment satisfies the environmental condition preset by the voice data generating apparatus 200 ( S130 ). In this case, the environmental conditions are conditions that the voice data recording apparatus 100 must satisfy in a situation where voice data is recorded.

As a result of the determination, when the information on the recording environment does not satisfy the environmental condition, the voice data recording apparatus 100 receives the background sound data until the information on the recording environment satisfies the environmental condition ( S110 ), and the recording environment The step of identifying information about ( S120 ) may be repeatedly performed.

As a result of the determination, when the information on the recording environment satisfies the environmental condition, the voice data recording apparatus 100 may activate a user interface for controlling the start and end of the recording ( S140 ).

The voice data recording apparatus 100 may receive voice data through a microphone when a command for instructing a recording start is input from a user through a user interface capable of controlling the start and end of the recording (S150).

On the other hand, the voice data recording apparatus 100 may re-collect the location of the voice data recording apparatus 100 every preset period after a user interface (UI) capable of controlling the start and end of recording is activated. . In addition, the voice data recording apparatus 100 deactivates a user interface capable of controlling the start and end of the previously activated recording when the size and frequency of the re-collected location information, voice data, and frequency do not satisfy environmental conditions, or , or the input of the voice data being input may be stopped.

The voice data recording apparatus 100 terminates the input of voice data through a microphone when a command for instructing the end of recording is input from the user through a user interface capable of controlling the start and end of recording, and then information on the recording environment can be used to generate metadata for voice data (S160).

Then, the voice data recording apparatus 100 may match the generated metadata with the voice data, store it in a database, and then transmit it to the voice data generating apparatus 200 ( S170 ).

14 is a flowchart illustrating a method for generating voice data according to an embodiment of the present invention.

Referring to FIG. 14 , the voice data generating apparatus 200 may design a data structure of voice data and metadata that may be a reference pattern of an acoustic model ( S210 ).

Specifically, the voice data generating apparatus 200 is a control signal input from the manager, or based on the requirements received from the voice recognition learning apparatus 300, voice data and metadata that can be a reference pattern of the acoustic model. You can design data structures.

The voice data generating apparatus 200 may collect and refine voice data and metadata based on the designed data structure (S220 and 230).

Specifically, the voice data generating apparatus 200 may receive voice data and metadata from a plurality of voice data recording apparatuses 100 . In addition, the voice data generating apparatus 200 may remove duplicate or extremely similar voice data from among the collected voice data and de-identify personal information included in the collected voice data.

The voice data generating apparatus 200 may process the collected and refined voice data and metadata into voice data and metadata that can be used to construct reference patterns of an acoustic model ( S240 ). A detailed description of the processing step of voice data will be described later with reference to FIGS. 15 and 16 .

The voice data generating apparatus 200 may provide the processed voice data and metadata to the voice recognition learning apparatus 300 ( S250 ). In this case, the voice data and metadata may be voice data that can be utilized to construct reference patterns of the acoustic model.

15 is a flowchart for explaining a voice data processing step according to an embodiment of the present invention.

Referring to FIG. 15 , the voice data generating apparatus 200 may analyze the waveform of the voice data collected by the voice data recording apparatus 100 to identify and separate an echo signal from the voice data ( S241 ).

Specifically, the voice data generating apparatus 200 may identify a transfer function between the speaker and the microphone of the voice data recording apparatus 100 related to the voice data by applying a frequency domain filter (FDAF) to the collected sound data. Then, the voice data generating apparatus 200 generates an echo signal from the voice data using the identified transfer function, and then subtracts the echo signal from the voice data 10 to remove the echo signal included in the voice data 10 . can do.

The voice data generating apparatus 200 may identify and separate the noise signal from the voice data from which the echo signal is separated and removed ( S242 ).

Specifically, the voice data generating apparatus 200 includes the voice data recording apparatus 100 and the talker (ie, the voice data recording apparatus ( 100) of users) can be estimated. The voice data generating apparatus 200 may minimize the final output power of two or more voice signals while maintaining the gain of applying a minimum variance non-distortion response (MVDR) filter to two or more voice signals as a preset constant value. It is possible to create a blocking matrix (BM) designed to be The voice data generating apparatus 200 may generate a noise signal from which a voice component in a target direction is removed from the voice data by passing the voice data from which the echo signal is removed through the blocking matrix BM. In addition, the voice data generating apparatus 200 may subtract the noise signal from the voice data from which the echo signal is removed to remove the noise signal included in the voice data.

The voice data generating apparatus 200 may identify and separate artificial noise from the voice data from which the noise signal is separated and removed ( S243 ).

Specifically, the voice data generating apparatus 200 may generate a transcript by performing STT on voice data from which background sound data is separated. The voice data generating apparatus 200 may parse the generated transcript to identify a plurality of tokens constituting the transcript. The voice data generating apparatus 200 may perform semantic analysis on each identified token. As a result of performing semantic analysis, the voice data generating apparatus 200 may identify the unclassified corresponding token as artificial noise when there is a token whose word meaning is not classified as a phoneme defined in advance. In addition, the voice data generating apparatus 200 may generate voice data from which all noises are removed by separating only the identified artificial noise from the voice data.

The voice data generating apparatus 200 may generate metadata such that the separated echo signal, the noise signal, the artificial noise 32, and the voice data 34 from which all noises have been removed match each other ( S244 ).

Then, the voice data generating apparatus 200 may match the generated metadata and the voice data with each other and store them in the database (S245).

16 is a flowchart for explaining a voice data processing step according to another embodiment of the present invention.

Referring to FIG. 16 , the voice data generating apparatus 200 may generate new voice data by synthesizing voice signals or may generate new voice data based on previously decomposed and stored voice data ( S246 ).

Specifically, the voice data generating apparatus 200 may identify a plurality of voice signals corresponding to a preset script in relation to voice data to be synthesized from the database. In addition, the voice data generating apparatus 200 may generate the voice data 30 by combining the plurality of identified voice signals.

Alternatively, the voice data generating apparatus 200 may identify voice data from which all noise is removed from the database.

The voice data generating apparatus 200 may synthesize artificial noise into the voice data (S247).

Specifically, the voice data generating apparatus 200 identifies a plurality of tokens constituting the script by parsing a script preset in relation to voice data to be synthesized, and one token predefined as expressing emotion If there is an abnormality, one emotional state that the speaker may have is estimated based on the one or more tokens, and phonemes that correspond to the estimated emotional state and generate artificial noise may be randomly added.

Alternatively, the voice data generating apparatus 200 may identify a speaker who recorded voice data based on metadata matched to the identified voice data, and may synthesize artificial noise corresponding to the identified speaker into the voice data.

The voice data generating apparatus 200 identifies a noise signal that satisfies a preset environmental condition with respect to the voice data 10 to be synthesized or generated, and additionally synthesizes the identified noise signal in the voice data synthesized with artificial noise. It can be done (S248).

In addition, the voice data generating apparatus 200 identifies an echo signal that satisfies a preset environmental condition with respect to the voice data 10 to be synthesized or generated, and generates the echo signal identified in the voice data synthesized with the noise signal. It can be additionally synthesized (S249).

As described above, although preferred embodiments of the present invention have been disclosed in the present specification and drawings, it is in the technical field to which the present invention pertains that other modifications based on the technical idea of the present invention are possible in addition to the embodiments disclosed herein. It is obvious to those with ordinary knowledge. In addition, although specific terms have been used in the present specification and drawings, these are only used in a general sense to easily explain the technical contents of the present invention and help the understanding of the present invention, and are not intended to limit the scope of the present invention. Accordingly, the above detailed description should not be construed as restrictive in all respects but as exemplary. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of the present invention.

Voice data recording device: 100 Voice data generating device: 200
Speech recognition learning device: 300
Terminal communication unit: 105 Terminal input/output unit: 110
Recording environment identification unit: 115 Voice recording control unit: 120
Metadata granting unit: 125 Collecting data providing unit: 130
Server communication unit: 205 Server input/output unit: 210
Data structure design department: 215 Data collection and refinement department: 220
Data processing unit: 225 Voice data providing unit: 230
Voice data processing module: 225-1 Call word processing module: 225-2
Metadata processing module: 225-3

Claims (10)

In a process in which the voice data recording apparatus receives voice data of a speaker through a microphone, time-series recording of an image including the speaker's face through a camera;
estimating, by the voice data generating apparatus, the emotional state of the speaker based on the change in the face included in the image;
The apparatus for generating voice data includes information on the estimated emotional state to generate metadata about the input voice data, and matches the voice data with the generated metadata to a database. saving to;
identifying, by the voice data generating device, voice data from which all noise has been removed from the database;
identifying, by the voice data generating device, a speaker who recorded the voice data based on metadata matched to the voice data;
identifying, by the voice data generating device, a man-made noise corresponding to the identified speaker, and synthesizing the identified artificial noise into the voice data; and
identifying, by the voice data generating apparatus, background sound data corresponding to a preset environmental condition, and synthesizing the identified background sound data with the artificial noise synthesized voice data;
The step of estimating the emotional state
From the image containing the face of the talker, the movements of the eyes, nose, and mouth of the talker are tracked, and machine-learning based on the facial images of the multiple talkers is applied to artificial intelligence (AI). It is characterized in that the emotional state of the speaker is estimated by querying based on the movements of the eyes, nose and mouth,
The step of estimating the emotional state
The voice data is STT (Speech To Text) to generate a transcript, and the transcript is parsed to identify a plurality of tokens constituting the transcript, and to each of the identified tokens It is characterized in that semantic analysis is performed on
The step of estimating the emotional state
As a result of performing the semantic analysis, if there is a token whose word meaning is not classified as a phoneme defined in advance, the unclassified token is identified as artificial noise,
Each of the artificial noises in the step of synthesizing the background sound data and the step of estimating the emotional state is
The spontaneous speech generating method, characterized in that the phoneme is uttered according to the speaker's utterance method, interjection, or chuimsae of the voice data, but does not have a word meaning. The method of claim 1, wherein synthesizing the background sound data comprises:
After identifying a noise signal and an echo signal that satisfy the environmental condition from a database provided in advance, the noise signal and the echo signal are sequentially arranged in the voice data synthesized with the artificial noise. A method for generating spontaneous speech, characterized in that it is synthesized as The method of claim 2, wherein the synthesizing the background sound data comprises:
After identifying an inaudible signal that satisfies the environmental condition from the database, the inaudible signal is synthesized with voice data in which the noise signal and the echo signal are synthesized. . The method of claim 3, wherein the synthesizing the background sound data comprises:
The method of claim 1, wherein the volume of each channel of the voice signal included in the voice data synthesized with the inaudible signal is adjusted so that the direction of the speaker is changed in the process of reproducing the voice data. The method of claim 1 , wherein synthesizing the artificial noise comprises:
The voice data is STT (Speech To Text) to generate a transcript, and the transcript is parsed to identify a plurality of tokens constituting the transcript, and to each of the identified tokens A method for generating spontaneous speech, characterized in that the semantic analysis is performed. The method of claim 5, wherein the synthesizing of the artificial noise comprises:
As a result of performing the semantic analysis, when one or more tokens predefined as expressing emotions exist, one emotional state that the speaker may have is estimated based on the one or more tokens, A method for generating spontaneous speech. 7. The method of claim 6, wherein synthesizing the artificial noise comprises:
and identifying one or more phonemes corresponding to the estimated emotional state and capable of generating artificial noise, and synthesizing the identified one or more phonemes into the voice data. The method of claim 1 , wherein synthesizing the artificial noise comprises:
The country, language, region, gender and age of the speaker are identified from the metadata about the speaker, and a phoneme that corresponds to the country, language, region, gender and age of the identified speaker and can generate artificial noise is one An abnormality is identified, and the identified one or more phonemes are synthesized into the voice data. The method of claim 1 , wherein synthesizing the artificial noise comprises:
Identifying one or more phonemes capable of generating artificial noise related to another speaker different from the speaker in at least one of country, language, region, gender and age, and synthesizing the identified one or more phonemes into the voice data A method for generating spontaneous speech. memory;
transceiver; and
Combined with a computing device configured to include a processor (processor) for processing instructions resident in the memory,
The processor receives audio data and an image from the audio data recording device through the transceiver, wherein the video is captured in time series through a camera while the audio data recording device receives the voice data of the speaker through a microphone an image including the face of one of the talkers;
estimating, by the processor, the emotional state of the speaker based on the change in the face included in the image;
The processor generates metadata about the input voice data by including information about the estimated emotional state, and stores the generated metadata in a database by matching the voice data step;
identifying, by the processor, voice data from which all noise has been removed from the database;
identifying, by the processor, a speaker who recorded the voice data based on metadata matched to the voice data;
identifying, by the processor, an artificial noise corresponding to the identified speaker, and synthesizing the identified artificial noise into the voice data; and
executing, by the processor, identifying background sound data corresponding to a preset environmental condition, and synthesizing the identified background sound data with the artificial noise synthesized voice data;
The step of estimating the emotional state
From the image containing the face of the talker, the movements of the eyes, nose, and mouth of the talker are tracked, and machine-learning based on the facial images of the multiple talkers is applied to artificial intelligence (AI). It is characterized in that the emotional state of the speaker is estimated by querying based on the movements of the eyes, nose and mouth,
The step of estimating the emotional state
The voice data is STT (Speech To Text) to generate a transcript, and the transcript is parsed to identify a plurality of tokens constituting the transcript, and to each of the identified tokens Characterized in performing semantic analysis for
The step of estimating the emotional state
As a result of performing the semantic analysis, if there is a token whose word meaning is not classified as a phoneme defined in advance, the unclassified token is identified as artificial noise,
Each of the artificial noises in the step of synthesizing the background sound data and the step of estimating the emotional state is
A computer program recorded on a recording medium, characterized in that it is a phoneme that is uttered according to the utterance method, interjection, or chuimsae of the speaker of the voice data, but does not have a word meaning.

Images