KR101203188B1 - Method and system of synthesizing emotional speech based on personal prosody model and recording medium - Google Patents

Abstract

A method and system are provided for synthesizing emotional speech based on a personal rhyme model. Emotional speech synthesis method according to the present invention, the step of extracting the emotional rhyme structure characteristics to extract the characteristics of the individual emotional rhyme structure by analyzing the individual speech, storing the extracted individual emotional rhyme structure in the emotional speech database, input text and target Receiving an emotion, retrieving an individual emotional rhyme structure corresponding to a speaker to synthesize speech from the emotional speech database, and converting the input text into an emotionless speech, and converting the emotionless speech And an emotional speech synthesis step of generating and outputting an emotional speech corresponding to the talker by modifying based on an individual emotional rhyme structure corresponding to the speech target emotion. Since the present invention uses an individual emotional rhyme structure when synthesizing emotional speech, from a series of repeated recognition tests, the average recognition rate reaches up to 95.5%.

Description

Method and system of synthesizing emotional speech based on personal prosody model and recording medium}

The present invention relates to speech synthesis techniques, and in particular, to synthesize an emotional language with the emotions of a talker, using the results of modeling the emotional prosody structure of basic emotions in consideration of the individuality and relative differences of the talkers. And a speech synthesis system.

Speech is the most basic and widely used form of communication for expressing ideas during human-human interaction. It is also studied as a user-friendly interface between humans and machines. For example, assistive robots use voice for daily activities such as weather, TV programs and medication administration schedules to provide information. The latest technology in speech synthesis synthesizes artificial speech with a very high recognition rate. In particular, the output of a text-to-speech (TTS) system using a unit-selection algorithm may be considered to be almost the same as a real human voice. The rhyme structure includes duration, loudness, and fundamental frequency changes, with sound quality representing the details of the source. Although the synthesis results show different rhyme structure and voice quality for each unit selection speech database, the general purpose TTS system provides speech results of about the same quality for a given sentence. If the same voice database and sentences are selected, they will result in the same synthesis results regardless of contextual information such as the emotional state of the talker. However, the quality of synthesized speech and the naturalness of intonation remain important overcoming challenges. With the demand for improved sound quality and naturalness, there is a need for speech synthesis techniques to provide the necessary information in a natural and effective manner. To achieve this goal, first, various types of emotional representations are generally converted into corresponding datasets, and the results are used for modeling each type of emotional speech. This type of massive dataset analysis technology has quantitatively and qualitatively improved the performance of the information it provides.

In other words, along with the need to improve the sound quality of general-purpose speech synthesis systems, various studies have been conducted to produce emotional colloquial expressions in a natural and effective manner, some of which are incorporated by reference in the specification. et al., 2001; Jurafsky and Martin, 2000; Tatham and Morton, 2005, Cowie et al., 2001; Gobl and Chasaide, 2003; Johnstone and Scherer, 1999; Tatham and Morton, 2004). However, the average recognition rate of these techniques is only 27.1% for neutral texts synthesized using emotional effects. In this study, we used a database corresponding to the target emotion, and used parameters related to voice quality and rhyme as the selection criteria specific to emotion. In addition, in recent years, research has been actively conducted on how to create more natural speech synthesis results by expressing intonation information according to emotion changes in addition to the above research. Among the accent information, the change in pitch is known to have the greatest influence on the expression of emotional state. However, this change of pitch is influenced by the speaker's speech characteristics and habits, and because of the problem that the pitch of pitch is not shown in the general speech, it is a sentence unit rather than a study on the change of pitch of detailed sentence elements. The research on the treatment of is mainly conducted.

However, this solution is not suitable for interaction based on personal experience, such as emotional speech synthesis. As a result of extensive research, it has been found that their own way of expressing emotions by individual talkers is based on their personal experiences and that large datasets have many differences from these individual experiences. In other words, this approach does not work well with interactions based on personal experiences, such as emotional speech synthesis. Experimentally, individual speakers have various ways of expressing emotions based on personal experiences. In addition, through extensive dataset management, these personalized and relative differences are easy to overlook.

That is, the conventional emotion speech synthesis technique has the following limitations.

First, since emotions are very delicate and personal mental states, it is not possible to categorize them uniformly. However, the prior art adopts the concept of "basic" emotions and alters or mixes these basic emotions to represent different emotions. However, there is no consensus on what constitutes a basic emotion, and such an approach is not appropriate, as different studies introduce different numbers of basic emotions.

The second problem of the prior art is that there are various emotional speech synthesis techniques for modifying the rhyme, sound quality and information of the original text (see for example Baenziger and Scherer, 2005; Cowie et al., 2001; Mozziconacci). , 2002; Oudeyer, 2003; Scherer, 1986; Schroeder, 2001; Tatham and Morton, 2004; Murray and Arnott, 2008) .However, according to the prior art, there is a clear definition of sound quality and textual information to distinguish each emotion. It is very difficult to set guidelines, and the emotional rhythm structure of pitch and speech rate is more for individual talkers (ie personal information) than intensity and pause length. This personal information allows us to model the relative differences in each emotional rhyme structure (ie the personal rhyme model), which is a massive analysis of conventional datasets. It was impossible in technology.

Therefore, embodying the emotional prosody structure of basic emotions, such as anger, fear, happiness and sadness, taking into account the individual and relative differences of the talkers, and synthesizing the emotional voice using the implemented emotional rhyme structure. Technology is urgently needed.

In addition, based on the personal rhyme model, there is an urgent need for a Korean emotional speech synthesis system that can add emotional information to speech expressions.

E. Abadjieva, I.R. Murray, and J. L. Arnott, "Applying analysis of human emotional speech to enhance synthetic speech," Third European Conference on Speech Communication and Technology, 1993. J. Allen, M.S. Hunnicutt, D.H. Klatt, R. C. Armstrong, and D.B. Pisoni, "From text to speech: the MITalk system," Cambridge Studies In Speech Science And Communication, 1987, p. 216. T. B ㅴ nziger and K.R. Scherer, "The role of intonation in emotional expressions," Speech Communication, vol. 46, 2005, pp. 252-267. M.E. Beckman and J. Hirschberg, The ToBI annotation conventions, Ohio State University, 1994. A. Black, P. Taylor, and R. Caley, The Festival speech synthesis system, University of Edinburgh, 1999. A.F. Bobick, S.S. Intille, J.W. Davis, F. Baird, C.S. Pinhanez, L.W. Campbell, Y.A. Ivanov, A. Sch ㆌ tte, and A. Wilson, "The KidsRoom: A perceptually-based interactive and immersive story environment," Presence, vol. 8, 1999, pp. 369-393. P. Boersma and D. Weenink, "Praat, a system for doing phonetics by computer," Glot international, vol. 5, 2001, pp. 341-345. P. Boersma and D. Weenink, Praat: doing phonetics by computer, 2005, Computer program (http://www.praat.org). C. Breazeal, "Emotion and sociable humanoid robots," International Journal of Human-Computer Studies, vol. 59, 2003, pp. 119-155. M. Bulut, C. Busso, S. Yildirim, A. Kazemzadeh, C.M. Lee, S. Lee, and S. Narayanan, "Investigating the role of phoneme-level modifications in emotional speech resynthesis," Ninth European Conference on Speech Communication and Technology, 2005. F. Burkhardt, "Simulation emotionaler Sprechweise mit Sprachsyntheseverfahren [Simulation of emotional speech by means of speech synthesis]," Doctoral dissertation, Technische Universit et Berlin, Berlin, Germany, 2001. F. Burkhardt and W.F. Sendlmeier, "Verification of acoustical correlates of emotional speech using formant-synthesis," ISCA Tutorial and Research Workshop (ITRW) on Speech and Emotion, 2000. J.E. Cahn, "Generating expression in synthesized speech," Master's thesis, Massachusetts Institute of Technology, 1989. J.E. Cahn, "The generation of affect in synthesized speech," Journal of the American Voice I / O Society, vol. 8, 1990, pp. 1-1. N. Campbell and J. Venditti, "J-ToBI: An intonation labeling system for Japanese," Proceedings of the Autumn meeting of the Acoustical Society of Japan, 1995, pp. 317-318. A. Chen, R.R. Muntz, S. Yuen, I. Locher, S.I. Sung, and M.B. Srivastava, "A support infrastructure for the smart kindergarten," IEEE Pervasive Computing, vol. 1, 2002, pp. 49-57. J.W. Chung and J. C Park, "Automated Classification of Sentential Types in Korean with Morphological Analysis," Language and Information, vol. 13, 2009, pp. 59-97. R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, and J.G. Taylor, "Emotion recognition in human-computer interaction," IEEE Signal processing magazine, vol. 18, 2001, pp. 32-80. R. Donovan, "Trainable speech synthesis," Doctoral dissertation, Cambridge University, 1996. E. Douglas-Cowie, N. Campbell, R. Cowie, and P. Roach, "Emotional speech: Towards a new generation of databases," Speech Communication, vol. 40, 2003, pp. 33-60. M. Edgington, "Investigating the limitations of concatenative synthesis," Fifth European Conference on Speech Communication and Technology, 1997. C. Gobl and A. N┱- Chasaide, "The role of voice quality in communicating emotion, mood and attitude," Speech Communication, vol. 40, 2003, pp. 189-212. L. Gong, "Is happy better than sad even if they are both non-adaptive? Effects of emotional expressions of talking-head interface agents," International journal of human-computer studies, vol. 65, 2007, pp. 183-191. A. Gravano, S. Benus, H. Chavez, J. Hirschberg, and L. Wilcox, "On the role of context and prosody in the interpretation of 'okay'," Proceedings of ACL, 2007, pp. 800-807. S.J. Haberman, "The analysis of residuals in cross-classified tables," Biometrics, vol. 29, 1973, pp. 205-220. M. Heerink, B. Kröse, B. Wielinga, and V. Evers, "Enjoyment intention to use and actual use of a conversational robot by elderly people," Proceedings of the 3rd ACM / IEEE international conference on Human robot interaction, 2008, pp. 113-120. B. Heuft, T. Portele, and M. Rauth, "Emotions in time domain synthesis," Proceedings of the 4th International Conference on Spoken Language Processing, 1996. X. Huang, A. Acero, and H.W. Hon, Spoken language processing: A guide to theory, algorithm, and system development, Prentice Hall PTR Upper Saddle River, NJ, USA, 2001. E. Hudlicka, "To feel or not to feel: The role of affect in human-computer interaction," International Journal of Human-Computer Studies, vol. 59, 2003, pp. 1-32. A. Iida, N. Campbell, S. Iga, F. Higuchi, and M. Yasumura, "A speech synthesis system with emotion for assisting communication," ISCA Tutorial and Research Workshop (ITRW) on Speech and Emotion, 2000, pp. 167-172. T. Johnstone and K.R. Scherer, "The effects of emotions on voice quality," Proceedings of the XIV Int. Congress of Phonetic Sciences, 1999, pp. 2029-2032. S.A. Jun, "K-ToBI (Korean ToBI) Labeling Conventions," Speech Science, vol. 7, 2000, pp. 143-169. D. Jurafsky, J.H. Martin, and A. Kehler, Speech and language processing: An introduction to natural language processing, computational linguistics, and speech recognition, MIT Press, 2000. J.W. Kang, H.S. Hong, B.S. Kim, and M.J. Chung, "Assistive Mobile Robot Systems helping the Disabled Workers in a Factory Environment," International Journal of Assistive Robotics and Mechatronics, vol. 9, 2008, pp. 42-52. Y. Kitahara and Y. Tohkura, "Prosodic control to express emotions for man-machine speech interaction," IEICE Transactions on Fundamentals of Electronics, communications and computer sciences, vol. 75, 1992, pp. 155-163. H.-J. Lee and J.C. Park, "Lexical Disambiguation for Intonation Synthesis: A CCG Approach," Proceedings of the Korean Society for Language and Information (KSLI), 2005a, pp. 103-118. H.-J. Lee and J.C. Park, "Vowel Sound Disambiguation for Proper Intonation Synthesis," Proceedings of the 19th Pacific Asia Conference on Language, Information and Computation, 2005b, pp. 131-142. H.-J. Lee and J.C. Park, "Intonation Synthesis using Emotional Information from Spoken Fairy Tale," Proceedings of the 17th Korean Association of Speech Science (KASS), 2005c, pp. 88-97. H.-J. Lee and J.C. Park, "Customized Message Generation and Speech Synthesis in Response to Characteristic Behavioral Patterns of Children," Human-Computer Interaction. HCI Intelligent Multimodal Interaction Environments, 2007a, pp. 114-123. H.-J. Lee and J.C. Park, "Characteristics of Spoken Discourse Markers and their Application to Speech Synthesis Systems," Proceedings of the 19th Annual Conference on Human and Cognitive Language Technology, 2007b, pp. 254-260. H.-J. Lee and J.C. Park, "Analysis and Use of Intonation Features for Emotional States," Proceedings of the 20th Annual Conference on Human and Cognitive Language Technology, 2008, pp. 144-149. H.-J. Lee and J.C. Park, "Interpretation of User Evaluation for Emotional Speech Synthesis System," Proceedings of the 13th International Conference on Human-Computer Interaction. Part I: New Trends, 2009, pp. 295-303. J. Ma, L. T. Yang, B.O. Apduhan, R. Huang, L. Barolli, and M. Takizawa, "Towards a smart world and ubiquitous intelligence: a walkthrough from smart things to smart hyperspaces and UbicKids," International Journal of Pervasive Computing and Communications, vol. 1, 2005, p. 53. T. Marumoto and N. Campbell, "Control of speaking types for emotion in a speech re-sequencing system," Proc. of the Acoustic Society of Japan, Spring meeting 2000, pp. 213-214. H.-J. Min, D. Park, E. Chang, H.-J. Lee, and J.C. Park, "u-SPACE: Ubiquitous Smart Parenting and Customized Education," Proceedings of the 15th Human Computer Interaction, 2006, pp. 94-102. J.M. Montero, J.M. Gutierrez-Arriola, S. Palazuelos, E. Enriquez, S. Aguilera, and J.M. Pardo, "Emotional speech synthesis: From speech database to TTS," Fifth International Conference on Spoken Language Processing, 1998. J.M. Montero, J. Guti ㅹ rrez-Arriola, J. Col ㅱ s, E. Enr'quequez, and J.M. Pardo, "Analysis and modeling of emotional speech in Spanish," Proc. of ICPhS, 1999, pp. 957-960. K.E. Moyer and B. von Haller Gilmer, "The Concept of Attention Spans in Children," The Elementary School Journal, 1954, pp. 464-466. S. Mozziconacci, "Prosody and emotions," Speech Prosody 2002, International Conference, 2002. I.R. Murray, Simulating emotion in synthetic speech, University of Dundee, 1989. I.R. Murray and J.L. Arnott, "Toward the simulation of emotion in synthetic speech: A review of the literature on human vocal emotion," JOURNAL-ACOUSTICAL SOCIETY OF AMERICA, vol. 93, 1993, pp. 1097-1097. I.R. Murray and J.L. Arnott, "Implementation and testing of a system for producing emotion-by-rule in synthetic speech," Speech Communication, vol. 16, 1995, pp. 369-390. I.R. Murray and J.L. Arnott, "Applying an analysis of acted vocal emotions to improve the simulation of synthetic speech," Computer Speech & Language, vol. 22, 2008, pp. 107-129. I.R. Murray, M.D. Edgington, D. Campion, and J. Lynn, "Rule-based emotion synthesis using concatenated speech," ISCA Tutorial and Research Workshop (ITRW) on Speech and Emotion, 2000. A. Ortony and T.J. Turner, "What's basic about basic emotions ?," Psychological review, vol. 97, 1990, pp. 315-331. P.Y. Oudeyer, "The production and recognition of emotions in speech: features and algorithms," International Journal of Human-Computer Studies, vol. 59, 2003, pp. 157-183. S.W. Park, Y.J. Heo, S.W. Lee, and J. H. Park, "Non-Fatal Injuries among Preschool Children in Daegu and Kyungpook.," Korean Journal of Preventive Medicine and Public Health, vol. 37, 2004, pp. 274-281. J. Pierrehumbert, "The perception of fundamental frequency declination," Journal of the Acoustical Society of America, vol. 66, 1979, pp. 363-369. J. Pineau, M. Montemerlo, M. Pollack, N. Roy, and S. Thrun, "Towards robotic assistants in nursing homes: Challenges and results," Robotics and Autonomous Systems, vol. 42, 2003, pp. 271-281. J.F. Pitrelli, M.E. Beckman, and J. Hirschberg, "Evaluation of prosodic transcription labeling reliability in the ToBI framework," Third International Conference on Spoken Language Processing, 1994. E. Rank and H. Pirker, "Generating emotional speech with a concatenative synthesizer," Fifth International Conference on Spoken Language Processing, 1998. P. Roach, Introducing phonetics, Penguin Books, 1992. N. Roy, G. Baltus, D. Fox, F. Gemperle, J. Goetz, T. Hirsch, D. Margaritis, M. Montemerlo, J. Pineau, J. Schulte, and others, "Towards personal service robots for the elderly, "Workshop on Interactive Robots and Entertainment (WIRE 2000), 2000, p. 184. C. Sas and A. Dix, "Designing for reflection on experience," Proceedings of the 27th international conference extended abstracts on Human factors in computing systems, 2009, pp. 4741-4744. K.R. Scherer, "Vocal affect expression: A review and a model for future research.," Psychological Bulletin, vol. 99, 1986, pp. 143-165. M. Schröder, "Emotional Speech Synthesis: A Review," Seventh European Conference on Speech Communication and Technology, 2001. K. Silverman, M. Beckman, J. Pitrelli, M. Ostendorf, C. Wightman, P. Price, J. Pierrehumbert, and J. Hirschberg, "ToBI: A standard for labeling English prosody," Second International Conference on Spoken Language Processing, 1992, pp. 867-870. J. Stallo, "Simulating emotional speech for a talking head," Honours Thesis, Curtin University of Technology, Perth, Australia, 2000. R. Stibbard, "Vocal expressions of emotions in non-laboratory speech An investigation of the Reading / Leeds Emotion in Speech Project annotation data," 2001. O. Trk and M. Schröder, "A comparison of voice conversion methods for transforming voice quality in emotional speech synthesis," Proceedings of Interspeech, 2008, pp. 2282-2285. M. Tatham and K. Morton, Expression in speech: analysis and synthesis, Oxford University Press, 2004. M. Tatham and K. Morton, Developments in speech synthesis, Wiley & Sons Ltd., 2005. J. Vaissiere, "Language-independent prosodic features," Prosody: Models and measurements, 1983, pp. 53-66. J.J. Venduti, "Japanese ToBI labeling guidelines," Manuscript, Ohio State University, 1995. J. Vroomen, R. Collier, and S. Mozziconacci, "Duration and intonation in emotional speech," Third European Conference on Speech Communication and Technology, 1993

It is an object of the present invention to provide a personalized emotional speech synthesis technique by analyzing the emotional rhyme structure of four basic emotions such as anger, fear, joy and sadness.

In addition, the study of the emotional rhythm structure of basic emotions, taking into account the individuality and relative differences of the talkers, shows that the pitch and the speaking speed depend more on the individual talker (ie, personal information) than the intensity and pause length. Another object of the present invention is to provide a speech synthesis technique for modeling each person's emotional rhyme structure.

In addition, another object of the present invention is to provide an emotional speech synthesis system in Korea that can add emotion information to a speech expression based on a personal rhyme model.

One aspect of the present invention for achieving the above object relates to a method for synthesizing emotional speech based on a personal rhyme model. Emotional speech synthesis method according to the present invention, the step of extracting the emotional rhyme structure characteristics to extract the characteristics of the individual emotional rhyme structure by analyzing the individual speech, storing the extracted individual emotional rhyme structure in the emotional speech database, input text and target Receiving an emotion, retrieving an individual emotional rhyme structure corresponding to a speaker to synthesize speech from the emotional speech database, and converting the input text into an emotionless speech, and converting the emotionless speech And an emotional speech synthesis step of generating and outputting an emotional speech corresponding to the talker by modifying based on an individual emotional rhyme structure corresponding to the speech target emotion. In particular, the emotion rhyme structure characteristic extraction step includes a general emotion rhyme structure extraction step of extracting a general emotion rhyme structure from a dataset including voice information according to the basic emotion of the talker, and an individual emotion rhyme structure of each talker. Comparing the general emotion rhyme structure with the individual emotion rhyme structure extraction step of parameterizing the relative difference of the individual emotion rhyme structure. Furthermore, the general emotion rhyme structure extraction step includes extracting voice characteristics corresponding to anger, fear, happiness, sadness, and neutral emotion state for each talker, and the individual emotion rhyme structure extraction step includes extracting individual emotion rhyme structures. Sentence level emotion rhythm structure analysis step of analyzing the overall pitch, intensity, and speech rate of the voice according to the emotion of the sentence at the sentence level as a parameter; A step of analyzing intonation level emotion rhyme structure, which analyzes at the intonation level as a parameter the pause length between included innation phrases (IP), and intonations included in the speech according to each emotion of the talker A syllable level emotional rhythm structure analysis step of analyzing at each syllable level an IP boundary pattern as a parameter. Further, the sentence level emotional rhyme structure analysis step of calculating the interquartile mean (IQM) of the pitch value of speech by emotion, selecting the intensity of a predetermined value or more of the intensity of speech by emotion, by emotion Calculating the speech rate from the total speech length of the speech, normalizing the pitch value, intensity, and speech rate to remove disparity and sentence-inconsistency, and using the normalized result, the neutral emotional state Computing the difference between the parameters of the individual emotional rhyme for each emotion based on the individual emotional rhyme structure corresponding to the standard, and constructing the individual emotional rhyme structure using the calculation result. In particular, analyzing the intonation level emotion rhyme structure analysis includes detecting a pause region between the intonations of emotion-based speech, and calculating the total idle length by summing the total lengths of the pause regions. In addition, the syllable level emotional rhyme structure analysis step includes a pitch contour corresponding to one of the speech intonation boundary patterns: L%, H%, LH%, HL%, LHL%, HLH%, HLHL%, LHLH%, and LHLHL%. analyzing as a pitch contour. Meanwhile, the emotional speech synthesis step may include converting an input text into an emotional voice using a text-to-speech system, retrieving an emotional rhyme structure corresponding to the target emotion of the talker from the emotional speech database, and And a voice modification step of generating an emotional voice by modifying the emotionless voice using the retrieved emotional rhyme structure parameters. In addition, the speech correction step includes extracting, as parameters, a pitch contour corresponding to the target emotion from the individual emotional rhyme structure, and a syllable level correction step of correcting the pitch contour of the emotionless speech using the extracted pitch contour. . In addition, the voice correction step includes extracting a pause length corresponding to the target emotion from the individual emotion rhyme structure as a parameter, and using the extracted pause length, to modify the intonation level of the unaffected voice. . On the other hand, the voice correction step is to extract the overall pitch, the overall intensity, and the speech rate corresponding to the target emotion from the individual emotion rhyme structure as a parameter, and using the extracted total pitch, the overall intensity, and the speech rate, the emotionless speech A sentence level correction step of modifying the overall pitch, the overall intensity, and the rate of speech.

Another aspect of the present invention for achieving the above objects relates to a system for synthesizing emotional speech based on a personal rhyme model. The emotional speech synthesis system according to the present invention analyzes individual speech and extracts an emotion rhyme structure characteristic extractor to extract characteristics of an individual emotion rhyme structure, an emotion speech database storing the extracted individual emotion rhyme structure, an input text and a target emotion are received. Converts the input text into an emotionless voice, retrieves an individual emotional rhyme structure corresponding to the speaker who synthesizes the voice from the emotional voice database, and corrects the converted emotional emotion based on the individual emotional rhyme structure corresponding to the target emotion. Thereby including an emotional speech synthesizer for generating and outputting an emotional speech corresponding to the talker. In particular, the emotion rhythm structure feature extracting unit may extract the general emotion rhyme structure from the dataset including voice information according to the basic emotion of the talker, and the individual emotion rhyme structure of each talker as the general emotion rhyme structure. By comparison, it is adapted to perform the operation of parameterizing the relative difference of the individual emotional rhyme structure. Furthermore, in order to extract an individual rhyme structure, the emotion rhyme structure characteristic extractor analyzes the sentence-level emotional rhyme structure that analyzes the overall pitch, intensity, and utterance speed of the voice according to each emotion of the talker at the sentence level as parameters. Analysis of intonation level level rhyme structure analysis, which analyzes the rest length between the intonations (IPs) included in the speech according to each emotion of the narrator, at the intonation level, and the speech according to each emotion of the narrator. It is adapted to perform a syllable level emotional rhythm structure analysis operation of analyzing the intonation boundary pattern of each of the included intonations as a parameter at the syllable level. In particular, the emotional rhythm structure characteristic extracting unit selects an intensity greater than or equal to a predetermined value from an operation of calculating an interquartile mean (IQM) of pitch values of speech for emotions and an emotion-based speech intensity to analyze a sentence-level emotional rhyme structure. Operation of calculating speech rate from the total speech length of emotion-by-emotion speech, normalizing pitch value, intensity, and speech rate to remove sentence inconsistency and speaker-inconsistency, and using normalized results. Based on the individual emotional rhyme structure corresponding to the emotional state as a standard, it is adapted to calculate the difference between the parameters of the individual emotional rhyme for each emotion, and perform an operation of constructing the individual emotional rhyme structure using the calculation result. On the other hand, the emotion rhyme structure characteristic extracting unit, to analyze the intonation level level emotion rhyme structure, the operation of detecting the rest area between the intonation of the speech by emotion, and the operation of calculating the total idle length by summing the total length of the rest areas; Is adapted to perform. Furthermore, the emotional rhythm structure characteristic extracting unit analyzes the syllable-level emotional rhyme structure by using the speech accent boundary patterns of L%, H%, LH%, HL%, LHL%, HLH%, HLHL% and LHLH%. And LHLHL% are adapted to perform the analyzing as a pitch contour corresponding to one of. Also, the emotional speech synthesis unit converts the input text into an emotional voice using a TTS system, retrieves an emotional rhyme structure corresponding to the target emotion of the talker from the emotional speech database, and uses the parameters of the found emotional rhyme structure. Thereby modifying the unaffected voice so as to perform a voice correction operation to generate the emotional voice. In particular, the emotion speech synthesis unit extracts, as a parameter, a pitch contour corresponding to the target emotion from the individual emotion rhyme structure as a parameter, and corrects the pitch contour of the emotion-free speech using the extracted pitch contour to perform the voice correction operation. Is adapted to perform a syllable level correction operation. In addition, the emotion speech synthesis unit extracts, as a parameter, a pause length corresponding to the target emotion from the individual emotion rhyme structure as a parameter, and corrects the pause length of the unemotional voice by using the extracted pause length. Is adapted to perform an accent level correcting operation. Furthermore, the emotion speech synthesis unit extracts, as parameters, the overall pitch, the total intensity, and the speech rate corresponding to the target emotion from the individual emotion rhyme structure as parameters, and the extracted overall pitch, the overall intensity, in order to perform the voice correction operation. , And is adapted to perform a sentence level correction operation to modify the overall pitch, overall intensity, and speech rate of the unaffected speech using the speech rate.

According to the present invention, the emotional rhyme structure of each individual may be modeled by analyzing the emotional rhyme structure of the basic emotion, and a speech sentence reflecting the emotion may be synthesized using the modeled result.

The present invention also provides a Korean emotional speech synthesis system that can add emotional information to a speech expression based on a personal rhyme model.

1 is a flowchart conceptually illustrating an emotional speech synthesis method according to an aspect of the present invention.
2 and 3 are graphs showing an average pitch value and an average intensity value for each talker, respectively.
4 is a graph showing an average pitch value of each emotional state in consideration of personal information.
5 is a graph showing strength values of each emotion in consideration of personal information.
6 is a graph showing the speech rate for each emotion.
7 is a graph showing the normalized rest length for each emotion.
8 is a block diagram conceptually illustrating an emotional speech synthesis system according to another aspect of the present invention.
9 is a graph showing the original rhyme structure and the result of emotion synthesis.

In order to fully understand the present invention, operational advantages of the present invention, and objects achieved by the practice of the present invention, reference should be made to the accompanying drawings and the accompanying drawings which illustrate preferred embodiments of the present invention.

Hereinafter, the present invention will be described in detail with reference to the preferred embodiments of the present invention with reference to the accompanying drawings. However, the present invention can be implemented in various different forms, and is not limited to the embodiments described. In order to clearly describe the present invention, parts that are not related to the description are omitted, and the same reference numerals in the drawings denote the same members.

Throughout the specification, when a part is said to "include" a certain component, it means that it may further include other components, without excluding the other components unless otherwise stated. In addition, the terms "... unit", "... unit", "module", "block", etc. described in the specification mean a unit for processing at least one function or operation, which means hardware, software, or hardware. And software.

1 is a flowchart conceptually illustrating an emotional speech synthesis method according to an aspect of the present invention.

Emotion is a psychological state based on personal experience. Recently, the interaction between various types of people and machines is rapidly increasing, and it has direct and indirect effects on various fields based on interaction. Prior art has attempted to find universality in the process of categorizing emotions and recognizing and expressing emotions of the categories. Of course, the prior art approach has shown some successful results in the process of information acquisition, information processing, information expression, etc. in the information technology field, and has shown some positive results in interactions based on emotions.

However, through several experiments, we analyze the rhyme structure that is highly related to the emotions of joy, sadness, and anger, but that the recognition result of some emotions is not very good in the emotional speech synthesis system developed based on these analysis results. Confirmed. At first, we thought of this problem as the limitation of emotion synthesis based on the rhyme structure, but even in the recognition test conducted by combining the synthesized emotional voice and the emotional voice uttered by a real person, the actual uttered emotional voice shows poor recognition results. There was a difficulty explaining why.

Since emotions are psychological states based on individual experiences, perceptions and expressions of emotions are also based on individual experiences. Therefore, if individuality is emphasized without sufficiently reflecting individual characteristics, individual and relative forms of information disappear in the process of analysis and expression. Therefore, the emotion speech synthesis method according to the present invention sees emotions as individual and relative types of information and discusses how to express such information in the emotion speech synthesis system.

To this end, we analyze the rhyme information according to emotions of joy, sadness, anger and anger in the form of a user model, and analyze the characteristic emotion expressions of each talker on the pitch curve, the average pitch, the average loudness, and the negative This study examines the average length of speech and the average length of speech, and examines the effect of the user model on the synthesis results of the emotional speech synthesis system. In addition, when evaluating the emotional speech synthesis results according to the user model to analyze the relative characteristics of the emotional speech expressions, the subject is given a sufficient period of adaptation to recognize the relative characteristics of the synthesized emotional speech expressions and the change of the recognition results is examined. . As a result, the recognition test of the emotional speech synthesis result using the user model showed a significant improvement in the recognition rate compared with the previous results, and the recognition rate considering the relative characteristics of the emotional speech expression was found to be almost accurate.

Hereinafter, the present invention will be described in detail with reference to FIG. 1.

Referring to FIG. 1, first, an individual voice according to a plurality of basic emotions is analyzed to extract characteristics of a personal emotional prosody structure (S110). At this time, in order to extract the characteristics of the emotion rhyme structure, first, the general emotion rhyme structure is extracted from the data set including voice information according to the basic emotion of the talker. Then, the voice is analyzed for each syllable, intonation, and sentence to generate an individual emotional rhyme structure. In this case, the overall pitch, intensity, and speech rate of the voice according to each emotion of the talker are analyzed at the sentence level, and the accents included in the voice according to the emotion of the talker ( The pause length between the intonation phrases (IP) is analyzed at the accent level, and the IP boundary pattern of each of the accents included in the speech according to the narrator's emotion is determined at the syllable level. Is analyzed. Sentences, intonations, and syllable level parameters are described in detail later in the relevant sections of the specification. In addition, the individual emotional rhyme structure of each talker is compared with the general emotional rhyme structure, and the relative difference of the individual emotional rhyme structure with respect to the general emotional rhyme structure is determined as a parameter of the individual emotional rhyme structure.

Then, the determined individual emotional rhyme structure is stored in the emotional voice database (S120).

When the individual emotional rhyme structure is analyzed and stored, the input text and the target emotion are received from the outside. If the input text and the target emotion are not received, they wait until they are received (S130). In the present specification, the target emotion refers to emotion information to be added to a signal obtained by converting an input text into a voice.

When the input text and the target emotion are received, the input text is converted into an unaffected voice using the TTS system (S140). In the present specification, an emotionless speech refers to a result of collectively converting input text using the same conversion algorithm. Such an emotional voice is distinguished from an emotional voice which is a voice additionally added with emotional information.

Then, the individual emotion rhyme structure corresponding to the target emotion of the talker to synthesize the voice is searched from the emotion speech database (S150). Then, the emotional voice is generated by modifying the unaffected voice at the syllable, intonation, and sentence level using the retrieved individual emotional rhyme structure parameters (S160). In particular, in order to generate an emotional voice from an emotional voice, a pitch contour corresponding to the target emotion is extracted from the individual emotional rhyme structure. Then, the pitch contour of the unaffected voice is modified using the extracted pitch contour.

In addition, in order to correct the accent level, the pause length corresponding to the target emotion is extracted from the individual emotion rhyme structure as a parameter, and then the pause length of the unaffected voice is modified using the extracted pause length. In addition, for the sentence level correction, the total pitch, the total intensity, and the speech rate corresponding to the target emotion are extracted from the individual emotion rhyme structure as parameters. Then, the extracted overall pitch, overall intensity, and speech rate may be used to modify the overall pitch, overall intensity, and speech rate of the unaffected voice.

The syllable, intonation, and sentence-level speech correction operations performed in the emotional speech synthesis step will be described later in detail in the corresponding part of the specification.

Then, the finally obtained emotional voice is reproduced to the outside (S170).

The reason for using the individual emotional rhyme structure to synthesize the speech in the emotional speech synthesis method according to the present invention is that the recognition rate of the synthesis result through the conventional TTS system is not high. In other words, considering emotion as universal information that can be obtained from extensive dataset analysis, the results are not good. In particular, even using real human voice recording, the recognition rate is only 17.1%. Thus, it was concluded that this low recognition rate is due to the limitation of rhyme correction. Therefore, when developing a Korean emotion speech synthesis system, it is necessary to consider personalized information, not general emotion, in finding guidelines for analyzing emotion information.

Hereinafter, the present invention will be described in more detail.

Korean Emotional Speech Synthesis Using Personal Rhymes Model

During everyday conversation, we choose specific words or phrases according to their main meaning. But depending on the context and its pronunciation, the main meaning can be changed in different ways. For example, if a word is expressed in an emotional state of anger, it will convey a negative sense. On the contrary, if the word is expressed in the emotional state of happiness, it will be interpreted as positive except in the ironic case.

The emotional speech synthesis method according to the present invention regards the emotional prosody structure as a relative rhythm difference between different emotions. In order to analyze the emotional rhyme structure, the present invention uses the Korean emotional speech corpus distributed by the Speech Information Technology and Industry Promotion Center of the Republic of Korea. However, this should not be understood as limiting the invention. In the present invention, ten emotional neutral sentences are used for example, and five basic emotions such as anger, fear, happiness, sadness and neutrality are selected.

Sentence Level Prosody Structure Analysis

In order to find the sentence level rhythm structure, the present invention analyzes the overall pitch, intensity, and speech rate values for each emotional state. First, the interquartile mean (IQM) of the extracted pitch values is calculated. And an intensity of 50 dB or more is chosen, which is the value known as the minimum level of the audible range.

Analysis of variance (ANOVA) on pitch and intensity values of speech uttered in a neutral emotional state to discover the effect of each speaker's speech style (personal information) on the analysis of sentence-level rhyme structure. The test is performed.

Figure 2 shows the average pitch value for each talker, where the relationship of F = 364.924, p = 0.000 <0.05 is satisfied.

3 shows the average intensity value for each talker, where a relationship of F = 16.338, p = 0.000, <0.05.

From the ANOVA test results as shown in FIGS. 2 and 3, all null hypotheses are discarded and a strong correlation between sex information and pitch values is confirmed. Also, after extracting each value for the emotional pitch and intensity structure, a normalization process is performed to correct the sentence dependent disparities and the speaker dependent inconsistencies. In the present invention, neutral is regarded as a standard state, and the difference between each standard and the emotional rhyme structure extracted from the same sentence pronounced by the same talker is calculated.

Pitch analysis

In order to examine the attributes of personal information, the present invention performed 240 series of ANOVA tests, classifying 240 sentences according to six talkers and focusing on the difference in pitch values for different emotions. The test results are shown in FIG.

4 is a graph showing an average pitch value of each emotional state in consideration of personal information.

Referring to FIG. 4, we can see that the average pitch value of each emotion state is very sensitive for each talker, so that the generalized pitch value is not suitable for determining the representative value for each emotion. have.

Strength analysis

One-way ANOVA tests are performed on 240 sentences, thereby discovering generalized intensity values between different emotions. From the results of the ANOVA test shown in FIG. 4, the null hypothesis was explicitly discarded.

To understand the role of personal information in the emotional strength structure, 240 sentences were subdivided into 40 sentences each, taking into account six speakers. A series of ANOVA tests were performed and the results are shown in FIG. 5.

5 is a graph showing strength values of each emotion in consideration of personal information.

Referring to FIG. 5, it can be seen that FIG. 5 is well preserved in order of sadness, fear, happiness and anger, except for two special cases such as KKS and PYH. Through this, it can be seen that the talker has his / her own same intensity value for each emotion. This is the same even if the normalization process is performed in advance.

Fire rate analysis

To analyze the relative speech rate for each emotional state, speech regions are annotated for 300 speech sentences. After annotating each sentence, measure the total length of the speech zone for each emotional state of speech spoken by the same talker. The length of the voice zone is then normalized to calculate a specific emotion-to-neutral ratio. Table 1 shows the normalized speech rate for each emotion, and FIG. 6 is a graph showing the speech rate for each emotion.

As a result, it can be seen that in order to add information related to the conversation to the spoken expression, the personal speech rate should be considered.

Accent Structure Rhythm Structure Analysis

In order to analyze the emotional rhyme structure through the level of the unit smaller than the sentence level, the Korean emotional speech corpus is annotated using the K-ToBI labeling system. In Korean, it is known that nine accent boundary (IP) boundary tones are identified, such as L%, H%, LH%, HL%, LHL%, HLH%, HLHL%, LHLH% and LHLHL%. have. It is also well known that this IP boundary pattern plays an important role in modeling the change in pitch contour according to different emotional states. On the other hand, the surface tonal pattern of an accentual phrase (AP) generally has a "L + H L + Ha" or "L Ha" pattern regardless of the emotional state.

In the present invention, a method using a K-ToBI (Korean Tone and Break Indices) labeling system based on linguistic knowledge was used for rhyme studies. For K-ToBI-based modeling, two tasks are needed. First, the label suitable for the document should be estimated from the labels classified into several types of labels with the accent from the input sentence, and the F0 trajectory should be generated from the estimated label. The feature of this method is that it is possible to extract F0 trajectories from a large amount of data in a statistical manner, but there is a drawback that a large-scale corpus must be constructed using a systematic classification system and linguistic knowledge.

K-ToBI System

The ToBI and Tone Indices (ToBI) labeling system was introduced in 1992 as an English-based rhyming labeling system. Since then, ToBI labeling systems have been devised in many languages. The K-ToBI labeling system is based on English-language ToBI and Japanese-language J-ToBI, with recent models being a word tier, phonological tone tier, and break-index tier. ), And five tiers of miscellaneous tiers. Each layer consists of the time and symbol of the event, and the representation of the F0 trajectory is composed of an initial tone, an accentual tone, and a boundary tone.

The rhyme structure of Korean language consists of two rhyme units, accent sphere and accent sphere, and accent sphere consists of one or more accent spheres, and symbols such as 'H%', 'L%' and 'HL%' signify the last tone change. It is expressed as 'H-' may be at the beginning of the bullish ball, and the last part of the bullish ball is composed of 'Lha'. There is also a stop index from '0' to '3' depending on the degree of reading. '0' means choon, and '3' is the symbol that indicates where the reading is most pronounced. Of these four indices, '2' is the boundary of accent sphere, and '3' is the boundary of accent sphere.

The other two layers are the word tier, which marks the boundaries of words, and the miscellaneous tier, which displays other information, such as breaths and laughter.

Pitch Contour Analysis

To analyze the predominant IP boundary tones, annotate 300 pieces of voice using the K-ToBI labeling system. For statistical analysis of K-ToBI labeled data, Pearson's Chi-square test is performed. Table 2 shows the results of Pearson's Chi-square test on K-ToBI labeling.

As can be seen from Table 2, the null hypothesis is excluded from the results (<0.05 of p = 0.0). This result provides statistical support for the fact that each emotion has a distinct IP boundary to distinguish one emotion from the others. Then, adjusted residuals are calculated to identify the distinct pitch contour of each emotion. Table 3 shows the adjustment residuals calculated to identify the distinct IP boundary tones along with each emotion including neutral.

Statistical analysis of the pitch contour pattern shows that we have a very strong correlation between anger and HL%, fear and H%, happiness and LH%, sadness and H%, and neutral and L%. If we exclude neutral from the pitch contour analysis, the L% IP boundary pattern will be assigned to the sadness shown in Table 4. Table 4 shows the adjustment residuals calculated to identify the distinct IP boundary tones along with each emotion excluding neutrality.

To evaluate the role of personalization information, we classify 300 sentences according to six talkers and perform a series of Chi-square tests, but no significant differences are found. This result can be judged to be due to the fact that the IP boundary pattern is a symbolic representation of the pitch contour, not the relative pitch value between different emotions.

Pause Length Analysis

Although the symbolic representation of break indices is obtained from the analysis of K-ToBI labeling data, the relative pause length must also be analyzed to correct the information reflected in the dialogue.

For 300 spoken sentences, annotating the holding zone, and calculating the total length of the resting zone of each sentence spoken by the same talker. Generalized rest lengths are listed in Table 5. Table 5 shows the generalized rest lengths for each emotion, and FIG. 7 is a graph showing the results.

Except for two special cases, such as CWJ and KKS, it can be seen that the order of rest length is well preserved in the order of happiness, anger, fear, and sadness. The normalized idle length cannot be used as a value representing all talkers. For example, if you use the generalized pause length as a representative value for the emotional rhyme structure, the distinct characteristics of each talker will be lost. Therefore, speech must be synthesized using personal information.

Personal typical based emotional rhyme integrated system

In order to integrate the accent level and sentence level emotional rhyme structure into a general purpose TTS system, an emotional speech synthesis system as shown in FIG. 8 is provided.

For convenience of explanation, a brief description of the TTS system is as follows.

The construction of a general text-to-speech (TTS) system can be roughly divided into a natural language processor, a phoneme / canal extractor, and a signal processor. The natural language processing section provides basic information to be used in the phoneme / canal extraction section through morphological analysis and syntax analysis. The phoneme / union extractor generates a phonetic form and information to be used by the phoneme / union extractor based on the basic information of the natural language processor. That is, through morphological analysis and syntax analysis, the alphabet, numbers, symbols, etc. are converted into correct Hangeul forms, and pronunciation selection considering various phonological changes is made. In addition, the rhyme boundary is estimated and various parameters used as inputs of the signal processor are estimated. Finally, the signal processing unit recovers the speech using the parameters extracted from the upper layer.

Of these, the rhythm processor generates F0 trajectory (Fundamental Frequency Contour), an accent physical signal used in the signal processor. Rhyme in speech means the characteristics of the signal in terms of phonological changes such as pitch, loudness, and syllable length. This may include temporal features such as speech rate or rhythm.

It is generally thought that rhymes consist of a series of accents, and the outbreak of rhyme processing begins with the extraction of these accents. The extracted intonation is converted into the F0 trajectory, which is a physical signal, through various modeling methods.

8 is a block diagram conceptually illustrating an emotional speech synthesis system according to another aspect of the present invention.

The emotional speech synthesis system 800 shown in FIG. 8 includes an emotional rhythm structure characteristic extractor 810, a TTS system 830, an emotional speech synthesizer 850, and an emotional speech database 890. In addition, the emotional speech synthesis unit 850 includes a syllable level corrector 860, an accent level corrector 870, and a sentence level corrector 880.

The emotional rhyme structure characteristic extractor 810 analyzes the individual speech and extracts the characteristic of the individual emotional rhyme structure, and stores the extracted result in the emotional speech database 890. When the input text and the target emotion are received, the TTS system 830 converts the input text into an unaffected voice, and the emotional voice synthesizer 850 converts the unaffected voice into an individual emotional rhyme structure received from the emotional voice database 890. By correcting accordingly, an emotional voice reflecting the emotion information is generated and output.

In particular, the emotion rhythm structure feature extractor 810 extracts a general emotion rhyme structure from a dataset including voice information according to the basic emotion of the talker, and extracts the individual emotion rhyme structure of each talker from the general emotion rhyme structure. Compared with, parameterize the relative difference of individual emotional rhyme structure. Furthermore, the emotion rhythm structure feature extractor 810 extracts parameters from the sentence level, the intonation level, and the syllable level, respectively. For example, the emotion rhyme structure characteristic extractor 810 may analyze the overall pitch, the intensity, and the speech rate of the voice according to each emotion of the talker at the sentence level. In addition, the emotion rhythm structure feature extractor 810 analyzes the pause length between the intonations IP included in the speech according to each emotion of the talker at the intonation level as a parameter, and the speech according to each emotion of the talker. An accent boundary pattern of each of the intonations included in the may be analyzed at the syllable level as a parameter.

In this case, since the process of extracting each parameter at the sentence, intonation, and syllable levels is as described above, repeated description is omitted for the sake of simplicity of the specification. Table 6 shows parameters extracted by the emotion rhythm structure characteristic extractor 810 for each individual.

The syllable level correction unit 860 extracts, as a parameter, a pitch contour corresponding to the target emotion from the individual emotional rhythm structure, and corrects the pitch contour of the emotionless voice using the extracted pitch contour. In addition, the accent level corrector 870 extracts a pause length corresponding to the target emotion from the individual emotion rhyme structure as a parameter, and uses the extracted pause length to correct the pause length of the unaffected voice. Further, the sentence level correction unit 880 extracts, as parameters, the total pitch, the total intensity, and the speech rate corresponding to the target emotion from the individual emotional rhythm structure, and uses the extracted total pitch, the total intensity, and the speech rate. Modify the overall pitch, overall intensity, and speech rate of the unaffected voice.

In order to select an appropriate personal model for the unaffected speech generally synthesized by the general purpose TTS system, the emotional speech synthesis system 800 compares the IQM of the unaffected speech with the pitch value of each talker shown in FIG. Once the personal model is selected, the corresponding parameters are used successively for accent level rhyme correction and sentence level rhyme correction.

The script for this operation is as follows:

# PRAAT script for the modification of pitch contours

if emotion $ == "Anger"

Formula ... 'IQMPitch' + 'MaxPitch' * sin (x / dur * pi)

elsif emotion $ == "Fear"

Formula ... 'IQMPitch' + 'MaxPitch' * (x / dur)

elsif emotion $ == "Happiness"

Formula ... 'IQMPitch' + 'MaxPitch' * exp ((x / dur) -0.5)

elsif emotion $ == "HappinessFinal"

Formula ... 'IQMPitch' + 'MaxPitch' * sin (x / dur * pi)

elsif emotion $ == "Sadness"

Formula ... 'IQMPitch' + 'MaxPitch' * (x / dur)

The pitch contour correction function uses linear functions, sine functions, and exponential functions for modeling each K-ToBI pitch contour shown in Table 4. All functions can be implemented as PRAAT scripts (see Boersma and Weenink, 2001).

evaluation

A cognitive test was conducted to evaluate the emotional rhyme integration system and its personal model. For this purpose, the following five sentences are used, and an unaffected voice is generated using a commercial Korean TTS system having a female voice (see VoiceText of http://www.voiceware.co.kr).

Sentence 1: I closed the door saying not to go.

Sentence 2: I really mean it.

Sentence 3: I don't know.

Sentence 4: You don't know how important what we do.

Sentence 5: A stranger walked in a warm coat while the wind and the sun were arguing that they were stronger.

Then 20 emotional voices are generated (reflecting four emotions per anger, fear, happiness and sadness).

9 is a graph showing the original rhyme structure and the result of emotion synthesis.

Fig. 9 shows the rhyme structure for the sentence "I closed the door saying not to go." Continuous lines represent intensity contours and broken lines represent pitch contours.

The test was conducted on 40 female kindergarten teachers with an average age of 28.8 years. They were told to listen to 20 pieces of randomly ordered emotional synthesis voices and then to choose the voices they thought were angry, fearful, happy, and sad. Table 7 shows the recognition test results for 40 people.

Referring to Table 7, it can be seen that anger is the most recognizable among the emotions, and fear is the hardest to recognize. However, the overall success rate is higher than what might come about by chance, including feelings of happiness. Compared with the evaluation results shown in the prior art, the success rate shown in Table 7 can be said to be very desirable, because the conventional recognition rate is close to a value that can come up by chance (Lee and Park, 2009; Schroeder 2001).

As described above, the emotional speech synthesis technique according to the present invention regards each emotion as a change in rhyme characteristics, and finds a feature in which each distinct rhyme fits among other emotions. In particular, since the present invention does not regard emotion as universal information that can be obtained from analyzing a large data set, the recognition rate can be improved. In other words, the emotional speech synthesis technique according to the present invention considers the fundamental characteristics of emotions, reprints that emotions are not a kind of universal information, and analyzes four basic emotions, an emotion rhyme structure of anger, fear, happiness and sadness. The individual emotional rhyme structure obtained as a result of the analysis includes parameters such as pitch, intensity and rest length. Using this parameter, emotion information is added to the unaffected voice synthesized by the TTS system.

The emotional speech synthesis technique according to the present invention achieves an average recognition rate of up to 95.5% for all emotions, from a series of repeated recognition tests supported with sufficient prior training experience. While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art. For example, in the present invention, there are six people who spoke Korean sentences in order to generate an emotional rhyme structure, as shown in Table 8 below.

However, these talkers are provided only as an example, and of course, other talkers may be used to generate an individual emotional rhyme structure according to each emotion.

Application: Development of Korean Emotional Speech Synthesis System for Voice Interface of Assistive Robots

Today, advances in robot technology make it possible to help people in real life on many ways.

Traditionally, there have been described assisted mobile robots to assist the handicapped to work in a factory environment. However, these assistive robotic systems do not require a visual and interactive interface to human-robot interaction. There are also mobile robots that help seniors by providing information about daily life. However, it is important to use facial expressions, body movements and voice for robot-human interaction, and emotion expression techniques can be applied to them.

Therefore, the speech synthesis technique according to the present invention can be applied to a Korean emotional rhyme integration system that modifies a rhyme structure according to sentence type and emotional state. Therefore, the emotional rhyme structure is regarded as the difference in relative rhyme configured for each talker, and the individual parameterizes the difference. To identify the sentence type, a combination of form and syntax information that appear within a given sentence can be used. This emotional rhyme integration system can be used as a post-processing module for a general purpose TTS system.

Sentence Types and Rhythm Structure

In Korean, unlike sentences in English, sentences are divided into five categories according to punctuation and final endings, which are declarative, imperative, propositive, interrogative, and exclamatory. It is a door.

Based on the linguistic analysis of the morphology and syntactic implications of each sentence type, the present invention automatically identifies sentence types having morphological analysis results.

Korean Emotional Speech Synthesis System for Voice Interface of Assistive Robots

Implementing an outputted voice interface for an assistive robotic system capable of generating emotional speech synthesis results is based on sentence type and personal information. In this case, the emotional speech synthesis system according to the present invention can be used. This system allows for minor levels of rhythm correction, such as accent level, word level and syllable level.

Thus, according to one aspect of the present invention there is provided a Korean emotional speech synthesis system that can be used in the voice interface of the assistant robot designed for the elderly. The system modifies the rhyme structure of a given sentence according to sentence type and emotional state.

To identify sentence types, we use a combination of form and syntactic information that appears within a given sentence. And, in order to analyze the emotional rhyme structure, the speech synthesis method according to one aspect of the present invention is used.

In addition, the method according to the present invention can be embodied as computer readable codes on a computer readable recording medium. The computer-readable recording medium may include all kinds of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, optical data storage devices, and the like, and may also be implemented in the form of carrier waves (for example, transmission over the Internet). Include. The computer readable recording medium can also store computer readable code that can be executed in a distributed fashion by a networked distributed computer system.

Therefore, the true technical protection scope of the present invention will be defined by the technical spirit of the appended claims.

The present invention can be applied to speech synthesis techniques.

Claims (24)

delete A method for synthesizing emotional speech based on a personal rhyme model,
An emotional rhyme structure characteristic extraction step of extracting characteristics of a personal emotional prosody structure by analyzing individual speech;
Storing the extracted individual emotion rhyme structure in an emotion speech database;
A receiving step of receiving an input text and a target emotion;
Retrieving an individual emotional rhyme structure corresponding to a speaker to synthesize a voice from the emotional voice database; And
An emotion that generates and outputs an emotion voice corresponding to the talker by converting the input text into an emotionless speech and modifying the converted emotionless voice based on the individual emotion rhyme structure corresponding to the target emotion Speech synthesis steps,
The emotion rhyme structure characteristic extraction step,
A general emotion rhyme structure extraction step of extracting a general emotion rhyme structure from a data set including voice information according to the basic emotion of the talker; And
Personal rhyme model-based emotional voice synthesis method comprising the step of extracting the individual emotional rhyme structure of each utterance compared to the general emotional rhyme structure and parameterize the relative difference of the individual emotional rhyme structure. The method of claim 2, wherein the extracting of the general emotional rhyme structure comprises:
And extracting a voice characteristic corresponding to anger, fear, happiness, sadness, and neutral emotional state for each talker. The method of claim 2, wherein the extracting the individual emotional rhyme structure comprises:
A sentence level emotion rhyme structure analysis step of analyzing overall pitch, intensity, and speech rate of speech according to each emotion of the talker at a sentence level as parameters;
An accent level analysis rhyme structure analysis step of analyzing a pause length between intonation phrases (IPs) included in speech according to each emotion of the talker at the accent level as a parameter; And
Personal rhyme structure analysis step of analyzing the syllable level emotion rhyme structure to analyze at the syllable level as a parameter the IP boundary pattern of each of the accents included in the speech according to each emotion of the talker Model-based emotional speech synthesis method. The method of claim 4, wherein the sentence level emotional rhyme structure analysis step comprises:
Calculating an interquartile mean (IQM) of pitch values of speech for emotions;
Selecting an intensity of a predetermined value or more among the intensity of the emotion-specific voice;
Calculating the speech rate from the total speech length of the emotion-specific speech;
Normalizing the pitch value, intensity, and speech rate to remove disparity per sentence and discordant per speaker; And
Using the normalized result, calculating the difference between the parameters of the individual emotional rhyme for each emotion based on the individual emotional rhyme structure corresponding to the neutral emotional state, and constructing the individual emotional rhyme structure using the calculation result Personal rhyme model-based emotional speech synthesis method comprising a. The method of claim 4, wherein the intonation level emotion rhyme structure analysis step,
Detecting a pause region between emotion intonations; And
And calculating a total idle length by summing the total lengths of the idle regions. The method of claim 4, wherein the analyzing of the syllable level emotional rhyme structure comprises:
Analyzing the negative intonation boundary pattern as a pitch contour corresponding to one of L%, H%, LH%, HL%, LHL%, HLH%, HLHL%, LHLH% and LHLHL% Personal rhyme model-based emotional speech synthesis method comprising a. The method of claim 2, wherein the emotion speech synthesis step,
Converting the input text into an emotional voice using a text-to-speech system;
Retrieving an emotion rhyme structure corresponding to the target emotion of the talker from the emotion speech database; And
And a voice modification step of generating the emotional voice by modifying the emotionless voice using the retrieved emotional rhyme structure parameters. The method of claim 8, wherein the voice correction step,
Extracting, as a parameter, a pitch contour corresponding to the target emotion from the individual emotion rhyme structure; And
And a syllable level correction step of correcting the pitch contour of the emotion-free speech using the extracted pitch contour. The method of claim 8, wherein the voice correction step,
Extracting, as a parameter, a pause length corresponding to the target emotion from the individual emotion rhyme structure; And
Personal rhythm model-based emotional speech synthesis method comprising the step of modifying the intonation level using the extracted pause length to modify the pause length of the emotion-free voice. The method of claim 8, wherein the voice correction step,
Extracting, as parameters, an overall pitch, an overall intensity, and a speech rate corresponding to the target emotion from the individual emotion rhythm structure; And
And a sentence level correction step of modifying the overall pitch, the overall intensity, and the speech rate of the emotionless speech using the extracted total pitch, the overall intensity, and the speech rate. . Claim 12 is abandoned in setting registration fee. 12. A recording medium having a computer program recorded thereon, the computer program comprising instructions executable by a computer for implementing the method according to any one of claims 2 to 11. delete A system for synthesizing emotional speech based on a personal rhyme model,
An emotional rhyme structure characteristic extracting unit for analyzing the individual speech to extract characteristics of the individual emotional rhyme structure;
An emotion speech database that stores the extracted individual emotion rhyme structure;
When the input text and the target emotion are received, the input text is converted into an unaffected voice, a personal emotional rhyme structure corresponding to the talker to synthesize the voice is retrieved from the emotional voice database, and the converted unaffected voice is converted into the target emotion. And an emotional speech synthesis unit for generating and outputting an emotional voice corresponding to the talker by modifying based on the corresponding emotional rhyme structure for each individual,
The emotional rhyme structure characteristic extraction unit,
Extracting a general emotion rhyme structure from a dataset including voice information according to the basic emotion of the talker, and
And a personalized rhyme model-based emotional speech synthesis system characterized by comparing the individual emotional rhyme structure of each talker with the general emotional rhyme structure and parameterizing the relative difference of the individual emotional rhyme structure. The method of claim 14, wherein the emotion rhyme structure characteristic extracting unit extracts the general emotion rhyme structure.
And a personal rhythm model-based emotional speech synthesis system configured to extract voice characteristics corresponding to anger, fear, happiness, sadness, and neutral emotional state for each talker. The method of claim 14, wherein the emotional rhyme structure characteristic extracting unit extracts the individual emotional rhyme structure.
Sentence level emotion rhyme structure analysis operation of analyzing the overall pitch, intensity, and speech rate of speech according to each emotion of the talker at sentence level as parameters;
An intonation level emotion rhyme structure analysis operation of analyzing the rest length between the intonations IP included in the speech according to each emotion of the talker at the intonation level as a parameter, and
Personal rhyme model-based emotion, characterized in that it is adapted to perform a syllable level emotional rhyme structure analysis operation that analyzes the accent boundary patterns of each of the intonations included in the speech according to each emotion of the talker at the syllable level Speech synthesis system. The method of claim 16, wherein the emotion rhythm structure characteristic extractor is configured to analyze a sentence-level emotion rhyme structure.
Calculating an interquartile mean (IQM) of pitch values of speech by emotion,
Selecting an intensity of a predetermined value or more among the intensity of the voice for each emotion,
Calculating the speech rate from the total speech length of the emotion-specific speech;
Normalizing the pitch value, intensity, and speech rate to eliminate inconsistency and sentence inconsistency;
Using the normalized result, the difference between the parameters of the individual emotional rhyme for each emotion is calculated based on the individual emotional rhythm structure corresponding to the neutral emotional state, and the operation of configuring the individual emotional rhyme structure using the calculation result is performed. A personal rhyme model based emotional speech synthesis system, adapted to perform. The method of claim 16, wherein the emotional rhythm structure characteristic extracting unit is configured to analyze an accent rhyme structure emotional rhythm structure.
Detecting a resting region between the intonations of emotion-based speech, and
Personal rhyme model based emotional speech synthesis system, characterized in that it is adapted to perform an operation of calculating the total idle length by summing the total lengths of the idle regions. The method of claim 16, wherein the emotion rhythm structure characteristic extractor is configured to analyze a syllable level emotion rhyme structure.
Adapted to perform the operation of analyzing the negative intonation boundary pattern as a pitch contour corresponding to one of L%, H%, LH%, HL%, LHL%, HLH%, HLHL%, LHLH% and LHLHL%. Emotional speech synthesis system based on a personal rhyme model. The method of claim 14, wherein the emotional speech synthesis unit,
Converting the input text into an emotional voice using a TTS system,
Retrieving an emotional rhyme structure corresponding to the target emotion of the talker from an emotional speech database, and
Personal rhythm model-based emotional speech synthesis system, characterized in that it is adapted to perform a speech modification operation to generate the emotional speech by modifying the emotionless speech using the retrieved emotional rhyme structure parameters. The method of claim 20, wherein the emotion speech synthesizer is further configured to perform the voice correction operation.
Extracting, as a parameter, a pitch contour corresponding to the target emotion from the individual emotion rhyme structure, and
Personal rhythm model-based emotional speech synthesis system, characterized in that it is adapted to perform a syllable level correction operation to correct the pitch contour of the emotion-free speech using the extracted pitch contour. The method of claim 20, wherein the emotion speech synthesizer is further configured to perform the voice correction operation.
Extracting, as a parameter, a pause length corresponding to the target emotion from the individual emotion rhyme structure; and
A personal rhythm model-based emotional speech synthesis system, characterized in that it is adapted to perform an accent level correction operation that modifies the idle length of the unaffected speech using the extracted pause length. The method of claim 20, wherein the emotion speech synthesizer is further configured to perform the voice correction operation.
Extracting, as parameters, an overall pitch, an overall intensity, and a speech rate corresponding to the target emotion from the individual emotion rhythm structure, and
Personal rhythm model-based emotional voices adapted to perform sentence level correction operations to modify the overall pitch, overall intensity, and speech rate of the emotionless speech using the extracted overall pitch, total intensity, and speech rate Synthesis system. Claim 24 is abandoned in setting registration fee. A recording medium which can be read by a computer on which a computer program containing instructions executable by a computer for driving the system according to any one of claims 14 to 23 is recorded.

Images