Classifications

• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L17/00—Speaker identification or verification techniques
  • G10L17/26—Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices

Definitions

• the present invention relates to the field of speech signal processing and in particular to an apparatus and a method for recognizing a speech emotion change of a speaker from speech data of the speaker.
• the speech emotion recognition technology may be applied to the field of human-machine interaction, and thus may greatly improve friendliness and accuracy of human-machine interaction.
• the conventional solutions only focus on recognizing a speech emotion of a speaker by extracting speech emotion features such as pitch, energy and formant from speech data of the speaker.
• speech emotion features of different speakers are different and even speech emotion features of the same speaker are also different at different time periods, it is difficult to accurately recognize speech emotions of personalized speech data in the conventional solutions.
• the emotion change recognition from a speech of a speaker rather than the emotion recognition from a speech is more interesting in many applications.
• a time point at which an emotion of an actor is changed from "calm” to "exciting” in a video is an appropriate time point of inserting an advertisement into the video. Therefore, in such applications, it is enough to accurately recognize a speech emotion change of a speaker from speech data of the speaker.
• speech emotion recognition due to inaccuracy on speech emotion recognition in the conventional solutions, it is difficult to accurately recognize speech emotion changes of personalized speech data according to speech emotion recognition results of the conventional solutions.
• an object of the invention is to provide an apparatus and a method for recognizing a speech emotion change of a speaker from speech data of the speaker, which are capable of providing good performance on speech emotion change recognition of personalized speech data.
• an embodiment of the invention provides a method of recognizing a speech emotion change of a speaker from speech data of the speaker, which may comprise the following steps: a window dividing step of dividing the speech data of the speaker into a plurality of windows by a window width; a window speech emotion feature calculating step of calculating a speech emotion feature for each of the plurality of windows; and a speech emotion change recognizing step of recognizing the speech emotion change of the speaker for a window set consisting of at least two contiguous windows by comparing the speech emotion features of the window set with each of a plurality of speech emotion feature change templates stored in a speech emotion feature change database to find out a speech emotion feature change template which matches the speech emotion features of the window set.
• an embodiment of the invention provides an apparatus for recognizing a speech emotion change of a speaker from speech data of the speaker, which may comprise: a window dividing means for dividing the speech data of the speaker into a plurality of windows by a window width; a window speech emotion feature calculating means for calculating a speech emotion feature for each of the plurality of windows; and a speech emotion change recognizing means for recognizing the speech emotion change of the speaker for a window set consisting of at least two contiguous windows by comparing the speech emotion features of the window set with each of a plurality of speech emotion feature change templates stored in a speech emotion feature change database to find out a speech emotion feature change template which matches the speech emotion features of the window set.
• an embodiment of the invention provides a computer-readable storage medium with a computer program stored thereon, wherein said computer program, when being executed, causes a computer to execute the above method of recognizing a speech emotion change of a speaker from speech data of the speaker.
• Figure 1 is a flow chart illustrating a method of recognizing a speech emotion change of a speaker from speech data of the speaker according to an embodiment of the invention
• Figure 2 is a flow chart illustrating an implementing example of the speech emotion change recognizing step S 130 of Figure 1 ;
• Figure 3 schematically illustrates waveform graphs of two speech segments of speaker A extracted from dialogue data between speakers A and B;
• Figure 4 schematically illustrates pitch change graphs respectively extracted from two speech segments of Figure 3
• Figure 5 schematically illustrates a pitch change graph of two windows corresponding to two speech segments of Figure 3, where the window width is the minimum length of the two speech segments and the singularities are removed;
• Figure 6 schematically illustrates a pitch change graph of many windows corresponding to two speech segments of Figure 3, where the window width is 10ms and the singularities are removed;
• Figure 7 illustrates an exemplary structure of a speech emotion feature change database employed in the embodiment of the invention
• Figure 8 is a block diagram illustrating a construction of an apparatus for recognizing a speech emotion change of a speaker from speech data of the speaker according to an embodiment of the invention
• Figure 9 is a block diagram illustrating an exemplary construction of the speech emotion change recognizing means 830 of Figure 8; and Figure 10 is a block diagram illustrating an exemplary construction of a computer in which the invention may be implemented.
• Figure 1 is a flow chart illustrating a method of recognizing a speech emotion change of a speaker from speech data of the speaker according to an embodiment of the invention.
• the speech data of the speaker may be inputted via an external device such as a sound recording device, a phone, a PDA or the like.
• the speech data of the speaker may be a whole piece of continuous speech data from the speaker, for example, an oral lecture made by a lecturer.
• the speech data of the speaker may be constituted by one or more continuous speech segments of the speaker extracted from dialogue data of a plurality of speakers comprising the speaker, for example, one or more continuous speech segments of a customer extracted from telephone conversation data between the customer and a call center agent in the application of call center.
• the discrimination of different speakers may be implemented using sndpeek or the like.
• Figure 3 schematically illustrates waveform graphs of two speech segments (a) and (b) of speaker A extracted from dialogue data between speakers A and B.
• the speech data of the speaker is constituted by two speech segments (a) and (b) of the speaker A.
• the method may include a window dividing step SIlO, a window speech emotion feature calculating step S 120 and a speech emotion change recognizing step S 130.
• the window dividing step SIlO the speech data of the speaker is divided into a plurality of windows by a window width.
• the window width may be a predetermined time width such as 10ms, 100ms, Is or the like.
• the window width may be a predetermined time width such as 10ms, 100ms, Is or the like, or may be determined by a larger one of the minimum length of the one or more continuous speech segments and a predetermined time width such as 10ms, 100ms, Is or the like.
• the speech data of the speaker is constituted by one or more continuous speech segments of the speaker
• one window covers only one speech segment at most, and when one speech segment can not be fully divided, the final reminder whose length is less than the window width may be omitted.
• a speech emotion feature is calculated for each of the plurality of windows.
• the speech emotion feature may comprise one or more of speech pitch, speech energy and speech speed.
• an average value of the speech emotion features of respective feature extraction intervals in the window is calculated as the speech emotion feature of the window.
• the feature extraction interval may be set to 10ms or another value depending on a specific design.
• the speech emotion feature of the window may be calculated in another manner depending on a specific design.
• speech emotion feature singularities are removed from the speech emotion features of respective feature extraction intervals in the window.
• the speech emotion feature singularities refer to those feature values equal to or approximate to zero (for example, caused by a silence period or the like), those feature values having a large fluctuation compared with their neighboring feature values (for example, caused by a noise or the like), and so on.
• the window may be removed.
• the calculated speech emotion features of the respective windows are schematically shown in Figure 6, wherein one point in the time axis represents one window and those windows whose speech emotion features are equal to or approximate to zero are removed.
• the speech emotion change of the speaker for a window set consisting of at least two contiguous windows is recognized by comparing the speech emotion features of the window set with each of a plurality of speech emotion feature change templates stored in a speech emotion feature change database to find out a speech emotion feature change template which matches the speech emotion features of the window set.
• the window set may include a predetermined number of windows, and may be sequentially selected with a moving step whose window number is less than the predetermined number.
• the window set may include all the windows of two successive speech segments, and may be sequentially selected with a move step of one speech segment.
• one type of speech emotion change may have a predetermined number of speech emotion feature change templates, each speech emotion feature change template associates one or more representative speech emotion feature change curves (e.g., speech pitch change curve, speech energy change curve, or the like) with one type of speech emotion change, and the speech emotion feature change templates may be generated in advance through a clustering algorithm by statistical analysis of a large corpus of representative speech data from different speakers.
• each speech emotion feature change template associates one or more representative speech emotion feature change curves (e.g., speech pitch change curve, speech energy change curve, or the like) with one type of speech emotion change
• the speech emotion feature change templates may be generated in advance through a clustering algorithm by statistical analysis of a large corpus of representative speech data from different speakers.
• Figure 7 illustrates an exemplary structure of a speech emotion feature change database employed in the embodiment of the invention.
• the speech emotion feature change database includes the following two tables: a speech motion feature change type table (a) and a speech emotion feature template table (b).
• the speech motion feature change type table (a) in Figure 7 has two field of "Change type ID” and "Change type name” and schematically shows four types of exemplary speech emotion changes: "Calm -> Angry”, “Angry -> Calm”, “Calm -> Happy”, and
• the speech emotion feature template table (b) in Figure 7 has three fields of "ID”, "Feature value (pitch)" and “Change type ID” and schematically shows one exemplary speech emotion feature curve associated with the speech emotion change of "Calm -> Angry".
• Figure 2 is a flow chart illustrating an implementing example of the speech emotion change recognizing step S 130 of Figure 1.
• the speech emotion features of the window set are normalized.
• the Euclidean distance calculating step S220 an Euclidean distance between the normalized speech emotion features of the window set and each of the plurality of speech emotion feature change templates stored in the speech emotion feature change database is calculated.
• a speech emotion feature change template whose Euclidean distance with the normalized speech emotion features of the window set is the smallest and less than a predetermined threshold is determined as the matching speech emotion feature change template.
• the exemplary speech emotion change template in the speech emotion change template table (b) of Figure 7 is determined as the matching speech emotion feature change template of the speech data in Figure 3 through the above matching process, and thus the speech emotion feature change of the speech data in Figure 3 is recognized as "Calm -> Angry".
• the speech emotion change recognizing step S 130 in Figure 1 may be performed only if there is any one of speech emotion feature changes between neighboring windows in the window set exceeding a predetermined threshold.
• the method may further comprise a speech emotion recognizing step of recognizing speech emotions of respective windows in the window set according to a recognition result of speech emotion change in the window set.
• a speech emotion recognizing step of recognizing speech emotions of respective windows in the window set according to a recognition result of speech emotion change in the window set. For example, when the speech emotion feature change of the speech data in Figure 3 is recognized as "Calm -> Angry", the speech emotion features of respective windows of the speech segment (a) may be recognized as "Calm” and the speech emotion features of respective windows of the speech segment (b) may be recognized as "Angry”.
• Figure 8 is a block diagram illustrating a construction of an apparatus for recognizing a speech emotion change of a speaker from speech data of the speaker according to an embodiment of the invention.
• the apparatus 800 may include a window dividing means 810, a window speech emotion feature calculating means 820 and a speech emotion change recognizing means 830.
• the window dividing means 810 may divide the speech data of the speaker into a plurality of windows by a window width.
• the window speech emotion feature calculating means 820 may calculate a speech emotion feature for each of the plurality of windows.
• the speech emotion change recognizing means 830 may recognize the speech emotion change of the speaker for a window set consisting of at least two contiguous windows by comparing the speech emotion features of the window set with each of a plurality of speech emotion feature change templates stored in a speech emotion feature change database to find out a speech emotion feature change template which matches the speech emotion features of the window set.
• Figure 9 is a block diagram illustrating an exemplary construction of the speech emotion change recognizing means 830 of Figure 8.
• the speech emotion change recognizing means 830 may include a normalizing means 910, an Euclidean distance calculating means 920 and a determining means 930.
• the normalizing means 910 may normalize the speech emotion features of the window set.
• the Euclidean distance calculating means 920 may calculate an Euclidean distance between the normalized speech emotion features of the window set and each of the plurality of speech emotion feature change templates stored in the speech emotion feature change database.
• the determining means 930 may determine a speech emotion feature change template whose Euclidean distance with the normalized speech emotion features of the window set is the smallest and less than a predetermined threshold as the matching speech emotion feature change template.
• the apparatus 800 may further comprise a speech emotion recognizing means for recognizing speech emotions of respective windows in the window set according to a recognition result of speech emotion change in the window set.
• a speech emotion recognizing means for recognizing speech emotions of respective windows in the window set according to a recognition result of speech emotion change in the window set.
• the above apparatus and method for recognizing a speech emotion change of a speaker from speech data of the speaker may be applied to many applications.
• a speech emotion change recognition result of a customer may be provided to a call center agent in the form of speech or image during the telephone conversion between the customer and the call center agent so that the call center agent may respond to the speech emotion change of the customer appropriately and rapidly.
• the desired contents of the lecture can be extracted according to a speech emotion change recognition result of the lecturer. For example, the portions of the lecture which exhibit the speech emotion of "sad" may be filtered out so as to extract the optimistic contents of the lecture.
• the above method and apparatus may be implemented by hardware.
• Such hardware may be a single processing device or a plurality of processing devices.
• Such processing device may be a microprocessor, a microcontroller, a digital processor, a microcomputer, a part of a central processing unit, a state machine, a logic circuit and/or any device capable of manipulating a signal.
• the above method and apparatus may be implemented by either software or firmware.
• a program that constitutes the software is installed, from a storage medium or a network, into a computer having a dedicated hardware configuration, e. g., a general - purpose personal computer 1000 as illustrated in Figure 10, that when various programs are installed therein, becomes capable of performing various functions, or the like.
• a central processing unit (CPU) 1001 performs various processes in accordance with a program stored in a read only memory (ROM) 1002 or a program loaded from a storage section 1008 to a random access memory (RAM) 1003.
• ROM read only memory
• RAM random access memory
• data required when the CPU 1001 performs the various processes or the like is also stored as required.
• the CPU 1001, the ROM 1002 and the RAM 1003 are connected to one another via a bus 1004.
• An input / output interface 1005 is also connected to the bus 1004.
• the following components are connected to input / output interface 1005:
• An input section 1006 including a keyboard, a mouse, or the like;
• An output section 1007 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), or the like, and a loudspeaker or the like;
• the storage section 1008 including a hard disk or the like; and a communication section 1009 including a network interface card such as a LAN card, a modem, or the like.
• the communication section 1009 performs a communication process via the network such as the internet.
• a drive 1010 is also connected to the input / output interface 1005 as required.
• a removable medium 1011 such as a magnetic disk, an optical disk, a magneto - optical disk, a semiconductor memory, or the like, is mounted on the drive 1010 as required, so that a computer program read therefrom is installed into the storage section 1008 as required.
• the program that constitutes the software is installed from the network such as the internet or the storage medium such as the removable medium 1011.
• the network such as the internet or the storage medium such as the removable medium 1011.
• this storage medium is not limit to the removable medium 1011 having the program stored therein as illustrated in Figure 10, which is delivered separately from the device for providing the program to the user.
• the removable medium 1011 examples include the magnetic disk (including a floppy disk (register trademark )), the optical disk (including a compact disk - read only memory (CD-ROM) and a digital versatile disk (DVD)), the magneto - optical disk
• the storage medium may be the ROM 1002, the hard disk contained in the storage section 1008, or the like, which have the program stored therein and is delivered to the user together with the device that containing them.

Landscapes

• Engineering & Computer Science (AREA)
• Health & Medical Sciences (AREA)
• Audiology, Speech & Language Pathology (AREA)
• Human Computer Interaction (AREA)
• Physics & Mathematics (AREA)
• Acoustics & Sound (AREA)
• Multimedia (AREA)
• User Interface Of Digital Computer (AREA)
• Management, Administration, Business Operations System, And Electronic Commerce (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Priority Applications (2)

Applications Claiming Priority (1)

Publications (1)

Family

ID=42739098

Family Applications (1)

Country Status (2)

Cited By (3)

Families Citing this family (8)

Citations (3)

• 2009
  • 2009-03-16 WO PCT/CN2009/070801 patent/WO2010105396A1/en active Application Filing
  • 2009-03-16 CN CN2009801279599A patent/CN102099853B/zh active Active

Patent Citations (3)

Non-Patent Citations (2)

Also Published As

Similar Documents

Legal Events