KR101480668B1 - Mobile Terminal Having Emotion Recognition Application using Voice and Method for Controlling thereof - Google Patents

Abstract

The present invention relates to a terminal and a method for controlling the same, wherein the terminal can recognize an emotion of a user using the voice of the user, and output an appropriate response corresponding to an emotional state of the user. According to an embodiment of the present invention, the terminal comprises a microphone, a memory, an emotion recognition unit, an output part, and an input part. The microphone obtains a first voice including the voice of a user. The memory stores sound model data related to a plurality of emotions. The emotion recognition unit receives the first voice, and can recognize an emotion of the user from the plurality of emotions using the sound model data and the first voice. The output part comprises: a display part to display visual information; a sound output module to output auditory information; a haptic module to generate vibrations; and a fragrance part to generate olfactory information. The input part receives a command from the user for the operation of an emotion recognition application installed in the terminal. The sound model data is information about characteristic parameters of the voice corresponding to each emotion. If the user inputs a command to execute the emotion recognition application through the input part, a predetermined execution window related to the emotion recognition unit is displayed on the display part and guides the recording of the first voice. The output part can output a response corresponding to the emotion of the user recognized based on the first voice.

Description

[0001] The present invention relates to a terminal equipped with an emotion recognition application using voice, and a control method thereof. [0002]

The present invention relates to a terminal equipped with an emotion recognition application and a control method thereof. More particularly, the present invention relates to a terminal capable of recognizing a feeling of a user using a voice of a user and outputting an appropriate response according to a feeling state of the user, And a control method thereof.

Multimodal emotion recognition means grasping emotions using various kinds of information such as facial expression, speech, gesture, gaze, head movement, context, etc. When multimodal information through a multimodal interface is input The information input in each of the modalities can be fused and analyzed. In addition, various learning algorithms can be used to extract and classify the characteristics of the information input in the multi-modality. At this time, depending on which learning algorithm is applied, the error rate to be recognized can be changed by analyzing the result.

The ability to recognize the user 's emotions is an important element of the intelligent interface. For this purpose, emotion recognition technology using characteristics of the user' s facial expression and voice has been attempted.

However, most of the emotion recognition technology based on voice has a problem that the recognition rate can not reach a satisfactory level.

In addition, there has been a problem that frequent cases of not matching with actual events occur frequently by using object reaction data of an artificial emotional state without inducing natural emotions and measuring object reactions.

Accordingly, there is a need to develop a terminal equipped with an emotion recognition application that can recognize a user's emotion more accurately by using a user's voice and can provide appropriate feedback in consideration of the emotion state of the user .

Korean Patent Publication No. 10-2002-0018541

F. Samaria and S. Young, "HMM based architecture for face identification ", Image and Computer Vision, vol. 12, pp. 537-543, October 1994

Disclosure of Invention Technical Problem [8] The present invention has been made to solve the above problems, and it is an object of the present invention to provide a terminal capable of recognizing a user's emotions using a voice of a user and outputting an appropriate response according to a user's emotional state, To provide it to the public.

Specifically, the present invention provides a terminal and a control method thereof that can more accurately recognize a user's emotions by utilizing feature parameters of a voice corresponding to a plurality of emotions felt by a user.

It is another object of the present invention to provide a terminal capable of providing feedback corresponding to emotion felt by a user by using a terminal and considering the emotional state of a user, and a control method thereof.

It is to be understood that both the foregoing general description and the following detailed description of the present invention are exemplary and explanatory and are not intended to limit the invention to the precise form disclosed. It can be understood.

There is provided a terminal capable of recognizing a plurality of emotions classified according to a preset reference using a user's voice, the terminal relating to an example of the present invention for realizing the above-mentioned problem, A memory for storing acoustic model data related to the plurality of emotions; a memory for storing the acoustic model data related to the plurality of emotions; and a processor for receiving the first voice and receiving the emotion of the user among the plurality of emotions using the acoustic model data and the first voice A display unit for displaying visual information, an acoustic output module for outputting auditory information, a haptic module for generating vibration, and an orientation unit for generating smell information, And an input unit for receiving a command related to the operation of the installed emotion recognition application, Wherein when the user inputs an execution command of the emotion recognition application by using the input unit, the acoustic model data is information on a feature parameter of the voice corresponding to each of the plurality of emotion, An execution window of the first voice may be displayed on the display unit to induce the recording of the first voice and the output unit may output a response corresponding to the emotion of the user recognized based on the first voice.

The reaction may include at least one of a first data display in the display unit, an acoustic output in the acoustic output module, a generation of vibration having a predetermined intensity and pattern in the haptic module, and a perfume generated in the direction unit And the first data is at least one of predetermined text, background, and emoticon.

The execution window receives the recording information from the user and uses the recording information to record the first voice. The recording information includes at least one of a sex of the user, a recording start point of the first voice, And / or < / RTI > time.

In the memory, acoustic model data different from each other according to the gender may be stored.

The emotion recognizing means may include a sensing unit for sensing the voice of the user recorded in the first voice based on the signal energy of the first voice, An extraction unit for extracting a parameter and a recognition unit for comparing the first characteristic parameter with the acoustic model data to determine the emotion of the user.

In addition, the feature parameter extracted by the extracting unit may include a Mel-Frequency Cepstral Coefficient (MFCC) and a log energy.

In addition, the MFCC applies a Fast Fourier Transform (FFT) to the user's voice sensed by the sensing unit, obtains a power spectrum by applying a Mel-scale filter bank to the result of applying the FFT, , And applying DCT (Discrete Cosine Transform) to the result of taking the log.

The recognition unit may calculate the observation probability value of the acoustic model data and the first feature parameter stored in the memory and may further include an emotion corresponding to the acoustic model data having the highest observation probability value among the calculated observation probability values, It can be judged by emotion.

Also, the acoustic model data stored in the memory may be generated based on a hidden Markov model.

Also, when the user inputs the update execution command displayed in the execution window using the input unit, acoustic model data for the user may be generated based on the hidden Markov model and additionally stored in the memory.

In addition, the plurality of emotions classified according to the predetermined criteria may include at least one of neutral, joy, anger, and grief.

In addition, when the user is in communication with the other party using the terminal, the terminal recognizes the emotion of the other party using the emotion recognition unit, and the output unit outputs a response corresponding to the emotion of the recognized party .

There is provided a method of controlling a terminal capable of recognizing a plurality of emotions classified according to a predetermined reference by using a user's voice, A second step of displaying a predetermined execution window associated with the emotion recognition means on the display unit, and a second step of displaying a first execution result of the first voice A fourth step in which the first voice is recorded, a fifth step in which the first voice is transmitted to the emotion recognition means, the third step in which the emotion recognition means acquires the acoustic model data and the first A sixth step of recognizing the emotion of the user among the plurality of emotions by using a voice, And a seventh step of outputting a response corresponding to the user's emotion recognized on the basis of the plurality of emotions, wherein the acoustic model data is information on a feature parameter of the voice corresponding to each of the plurality of emotions, A sound output module for outputting auditory information, a haptic module for generating vibration, and a direction portion for generating smell information.

The reaction may include at least one of a first data display in the display unit, an acoustic output in the acoustic output module, a generation of vibration having a predetermined intensity and pattern in the haptic module, and a perfume generated in the direction unit And the first data is at least one of predetermined text, background, and emoticon.

The sixth step may include detecting a voice of the user recorded on the first voice based on a signal energy of the first voice, And comparing the first feature parameter with the acoustic model data to determine an emotion of the user.

In addition, the plurality of emotions classified according to the predetermined criteria may include at least one of neutral, joy, anger, and grief.

The present invention can provide a user with a terminal capable of recognizing a user's emotion using a user's voice and outputting an appropriate response according to a user's emotional state and a control method thereof.

Specifically, the present invention can provide a user with a terminal and a control method thereof that can more accurately recognize a user's emotions by utilizing characteristic parameters of a voice according to a plurality of emotions felt by the user.

In addition, the present invention can provide a user with a terminal capable of providing feedback in consideration of a user's emotional state by outputting feedback corresponding to emotion felt by the user using the terminal and a control method thereof.

It should be understood, however, that the effects obtained by the present invention are not limited to the above-mentioned effects, and other effects not mentioned may be clearly understood by those skilled in the art to which the present invention belongs It will be possible.

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate a preferred embodiment of the invention and, together with the description, serve to provide a further understanding of the technical idea of the invention, It should not be construed as limited.
1 shows an example of a block diagram of a terminal equipped with an emotion recognition application of the present invention.
2A and 2B illustrate a front perspective view and a rear perspective view of a terminal related to the present invention.
3 is a front view of a terminal for explaining an operation state of a terminal related to the present invention.
4 shows an embodiment of an execution window by execution of the emotion recognition application according to the present invention.
5A and 5B schematically show an embodiment of a process of extracting a feature parameter of an extracting section of the emotion recognition means.
6A and 6B schematically illustrate an embodiment of a process of recognizing and training the emotion of the emotion recognition means.
FIG. 7 shows experimental results on recognition rate and recognition time according to the emotion recognition means of the present invention.
8 is a schematic view illustrating a process of recognizing a user's emotion by the emotion recognition means of the present invention.

Hereinafter, a preferred embodiment of the present invention will be described with reference to the drawings. In addition, the embodiment described below does not unduly limit the contents of the present invention described in the claims, and the entire configuration described in this embodiment is not necessarily essential as the solution means of the present invention.

Hereinafter, a terminal equipped with an emotion recognition application using voice to be proposed by the present invention will be described in detail.

A mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player), a PDA Navigation, and the like. However, it will be apparent to those skilled in the art that the configuration according to the embodiments described herein may be applied to a fixed terminal such as a digital TV, a desktop computer, and the like, unless the configuration is applicable only to a portable terminal.

1 shows an example of a block diagram of a terminal equipped with an emotion recognition application of the present invention.

The terminal 1100 includes a wireless communication unit 1110, an audio / video input unit 1120, an input unit 1130, a sensing unit 1140, an output unit 1150, a memory 1160, 1170, a control unit 1180, a power supply unit 1190, and the like. The components shown in FIG. 1 are not essential, and a terminal having more or fewer components may be implemented.

Hereinafter, the components will be described in order.

The wireless communication unit 1110 may include one or more modules that enable wireless communication between the terminal 1100 and the wireless communication system or between the terminal 1100 and the network in which the terminal 1100 is located. For example, the wireless communication unit 1110 may include a broadcast receiving module 111, a mobile communication module 1112, a wireless Internet module 1113, a short range communication module 1114, a location information module 1115, .

The broadcast receiving module 1111 receives broadcast signals and / or broadcast-related information from an external broadcast management server through a broadcast channel.

The broadcast channel may include a satellite channel and a terrestrial channel. The broadcast management server may refer to a server for generating and transmitting broadcast signals and / or broadcast related information, or a server for receiving broadcast signals and / or broadcast related information generated by the broadcast management server and transmitting the generated broadcast signals and / or broadcast related information. The broadcast signal may include a TV broadcast signal, a radio broadcast signal, a data broadcast signal, and a broadcast signal in which a data broadcast signal is combined with a TV broadcast signal or a radio broadcast signal.

The broadcast-related information may refer to a broadcast channel, a broadcast program, or information related to a broadcast service provider. The broadcast-related information may also be provided through a mobile communication network. In this case, it may be received by the mobile communication module 1112.

The broadcast-related information may exist in various forms. For example, an EPG (Electronic Program Guide) of DMB (Digital Multimedia Broadcasting) or an ESG (Electronic Service Guide) of Digital Video Broadcast-Handheld (DVB-H).

For example, the broadcast receiving module 1111 may be a Digital Multimedia Broadcasting-Terrestrial (DMB-T), a Digital Multimedia Broadcasting-Satellite (DMB-S), a Media Forward Link Only A digital broadcasting system such as DVB-CB, OMA-BCAST, or Integrated Services Digital Broadcast-Terrestrial (ISDB-T). Of course, the broadcast receiving module 1111 may be adapted to other broadcasting systems as well as the digital broadcasting system described above.

The broadcast signal and / or broadcast related information received through the broadcast receiving module 1111 may be stored in the memory 1160.

The mobile communication module 1112 transmits and receives radio signals to at least one of a base station, an external terminal, and a server on a mobile communication network. The wireless signal may include various types of data depending on a voice call signal, a video call signal or a text / multimedia message transmission / reception.

The wireless Internet module 1113 refers to a module for wireless Internet access, and may be built in or externally attached to the terminal 1100.

WLAN (Wi-Fi), Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access) and the like can be used as the technology of the wireless Internet.

The short-range communication module 1114 refers to a module for short-range communication. Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, and the like can be used as the short range communication technology.

The position information module 1115 is a module for obtaining the position of the terminal 1100, and a representative example thereof is a Global Position System (GPS) module. According to the current technology, the GPS module 1115 calculates distance information and accurate time information from three or more satellites, and then applies a trigonometric method to the calculated information to generate a three-dimensional string of latitude, longitude, The location information can be accurately calculated. At present, a method of calculating position and time information using three satellites and correcting an error of the calculated position and time information using another satellite is widely used. In addition, the GPS module 1115 can calculate speed information by continuously calculating the current position in real time.

1, an A / V (Audio / Video) input unit 1120 is for inputting an audio signal or a video signal, and may include a camera module 1121, a microphone 1122, and the like. The camera module 1121 processes image frames such as still images or moving images obtained by the image sensor in the video communication mode or the photographing mode. The processed image frame can be displayed on the display portion 1151. [

The image frame processed by the camera module 1121 can be stored in the memory 1160 or transmitted to the outside via the wireless communication unit 1110. [

At least two camera modules 1121 may be provided depending on the use environment.

For example, the camera module 1121 may include first and second cameras 1121a and 1121b for capturing 3D images on the opposite side of the display unit 1151 of the terminal 1100, A third camera 1121c for self-photographing of the user may be provided in a part of the surface of the surface of the display unit 1151 where the display unit 1151 is provided.

In this case, the first camera 1121a is for capturing a left eye image, which is a source image of a 3D image, and the second camera 1121b may be for capturing a right eye image.

The microphone 1122 receives an external sound signal by a microphone in a communication mode, a recording mode, a voice recognition mode, or the like, and processes it as electrical voice data. The processed voice data can be converted into a form that can be transmitted to the mobile communication base station through the mobile communication module 1112 and output when the voice data is in the call mode. The microphone 1122 may be implemented with various noise reduction algorithms for eliminating noise generated in the process of receiving an external sound signal.

In particular, the microphone 1122 can be used to record the voice of a user using the terminal of the present invention in the voice recognition mode. The voice recorded by the microphone 1122 can be used by the emotion recognition means of the control unit 1180 to recognize the user's emotion.

The input unit 1130 can generate input data for controlling the operation of the terminal and generate commands related to the overall operation of the emotion recognition application installed in the terminal 1100 of the present invention.

For example, when the user inputs an execution command of the emotion recognition application using the input unit 1130, a predetermined execution window is displayed on the display unit 1151 for progressing a series of processes for emotion recognition by the emotion recognition unit Can be displayed.

The input unit 1130 can receive from the user a signal designating two or more contents among the displayed contents according to the present invention. A signal for designating two or more contents may be received via the touch input, or may be received via the hard key and soft key input.

The input unit 1130 may receive an input from the user for selecting the one or more contents. Also, an input may be received from a user to generate an icon associated with a function that the terminal 1100 can perform.

The input unit 1130 may include a direction key, a keypad, a dome switch, a touch pad (static / static), a jog wheel, a jog switch, and the like.

The sensing unit 1140 senses the current state of the terminal 1100 such as the open / close state of the terminal 1100, the position of the terminal 1100, the presence of a user, the orientation of the terminal 1100, And generates a sensing signal for controlling the operation of the terminal 1100. For example, when the terminal 1100 is in the form of a slide phone, it is possible to sense whether the slide phone is opened or closed. It is also possible to sense whether the battery is powered on, whether the interface unit 1170 is connected to an external device, and the like. Meanwhile, the sensing unit 1140 may include a proximity sensor 1141. The proximity sensor 1141 will be described later in relation to the touch screen.

The output unit 1150 is for generating output related to visual, auditory or tactile sense and includes a display unit 1151, an acoustic output module 1152, an alarm unit 1153, a haptic module 154 and a projector module 1155, and the like.

The display unit 1151 displays (outputs) information to be processed by the terminal 1100. For example, when the terminal is in the call mode, a UI (User Interface) or GUI (Graphic User Interface) associated with the call is displayed. When the terminal 1100 is in the video communication mode or the photographing mode, the photographed and / or received video or UI and GUI are displayed.

In addition, the display unit 1151 according to the present invention supports 2D and 3D display modes.

That is, the display unit 1151 according to the present invention may have a configuration in which a switch liquid crystal 1151b is combined with a general display device 1151a as shown in FIG. 1 below. Then, by using the switch liquid crystal 1151b, the optical parallax barrier can be operated to control the traveling direction of light, thereby separating the light so that different lights reach the right and left eyes. Therefore, when a combined image of the right eye image and the left eye image is displayed on the display device 1151a, the user can see the image corresponding to each eye and feel as if the image is displayed as a three-dimensional object.

That is, under the control of the control unit 1180, the display unit 1151 drives only the display device 1151a without driving the switch liquid crystal 1151b and the optical parallax barrier in the 2D display mode, And performs an operation.

Under the control of the controller 1180, the display unit 1151 drives the switch liquid crystal 1151b, the optical parallax barrier, and the display device 1151a in the 3D display mode to perform a 3D display operation.

The display unit 1151 may be a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED) A flexible display, and a three-dimensional display (3D display).

Some of these displays may be transparent or light transmissive so that they can be seen through. This can be referred to as a transparent display, and a typical example of the transparent display is TOLED (Transparent OLED) and the like. The rear structure of the display portion 1151 may also be of a light transmission type. With this structure, the user can see an object located behind the terminal body through the area occupied by the display portion 1151 of the terminal body.

There may be two or more display units 1151 according to the embodiment of the terminal 1100. For example, in the terminal 1100, a plurality of display units may be spaced apart from one another or may be disposed integrally with each other, or may be disposed on different surfaces.

(Hereinafter, referred to as a 'touch screen') in which a display unit 1151 and a sensor (hereinafter, referred to as 'touch sensor') that detects a touch operation form a mutual layer structure, It can also be used as an input device. The touch sensor may have the form of, for example, a touch film, a touch sheet, a touch pad, or the like.

The touch sensor may be configured to convert a change in a pressure applied to a specific portion of the display portion 1151 or a capacitance generated in a specific portion of the display portion 1151 into an electrical input signal. The touch sensor can be configured to detect not only the position and area to be touched but also the pressure at the time of touch.

If there is a touch input to the touch sensor, the corresponding signal (s) is sent to the touch controller (not shown). The touch controller processes the signal (s) and transmits corresponding data to controller 1180. Thus, the control unit 1180 can know which area of the display unit 1151 is touched or the like.

The proximity sensor 1141 may be disposed within an area of the terminal 1100 that is wrapped by the touch screen or near the touch screen. The proximity sensor refers to a sensor that detects the presence or absence of an object approaching a predetermined detection surface or a nearby object without mechanical contact using the force of an electromagnetic field or infrared rays. The proximity sensor has a longer life span than the contact sensor and its utilization is also high.

Examples of the proximity sensor include a transmission type photoelectric sensor, a direct reflection type photoelectric sensor, a mirror reflection type photoelectric sensor, a high frequency oscillation type proximity sensor, a capacitive proximity sensor, a magnetic proximity sensor, and an infrared proximity sensor. And to detect the proximity of the pointer by the change of the electric field along the proximity of the pointer when the touch screen is electrostatic. In this case, the touch screen (touch sensor) may be classified as a proximity sensor.

Hereinafter, for convenience of explanation, the act of recognizing that the pointer is positioned on the touch screen while the pointer is not in contact with the touch screen is referred to as "proximity touch & The act of actually touching the pointer on the screen is called "contact touch. &Quot; The position where the pointer is proximately touched on the touch screen means a position where the pointer is vertically corresponding to the touch screen when the pointer is touched.

The proximity sensor detects a proximity touch and a proximity touch pattern (e.g., a proximity touch distance, a proximity touch direction, a proximity touch speed, a proximity touch time, a proximity touch position, a proximity touch movement state, and the like). Information corresponding to the detected proximity touch operation and the proximity touch pattern may be output on the touch screen.

The audio output module 1152 can output audio data received from the wireless communication unit 1110 or stored in the memory 1160 in a call signal reception mode, a call mode or a recording mode, a voice recognition mode, a broadcast reception mode, The sound output module 1152 also outputs sound signals related to functions (e.g., call signal reception tones, message reception tones, etc.) performed in the terminal 1100. The sound output module 1152 may include a receiver, a speaker, a buzzer, and the like.

The alarm unit 1153 outputs a signal for notifying the occurrence of an event of the terminal 1100. Examples of events generated in the terminal 1100 include call signal reception, message reception, key signal input, touch input, and the like. The alarm unit 1153 may output a signal for informing occurrence of an event in a form other than a video signal or an audio signal, for example, a vibration. The display unit 1151 and the sound output module 1152 may be a kind of the alarm unit 1153 .

The haptic module 1154 generates various tactile effects that the user can feel. A typical example of the haptic effect generated by the haptic module 1154 is vibration. The intensity and the pattern of the vibration generated by the hit module 1154 are controllable. For example, different vibrations may be synthesized and output or sequentially output.

In addition to vibration, the haptic module 1154 may be configured to perform various functions such as a pin arrangement vertically moving with respect to the contact skin surface, a spraying force or suction force of the air through the injection port or the suction port, a touch on the skin surface, contact with an electrode, And various tactile effects such as an effect of reproducing a cold sensation using an endothermic or exothermic element can be generated.

The haptic module 1154 can be implemented not only to transmit a tactile effect through direct contact, but also to allow a user to feel a tactile effect through a muscular sensation such as a finger or an arm. The haptic module 1154 may include two or more haptic modules according to the configuration of the terminal 1100.

The projector module 1155 is an element for performing an image project function using the terminal 1100 and is identical to the image displayed on the display unit 1151 according to a control signal of the controller 1180 At least some of which can display other images on an external screen or wall.

Specifically, the projector module 1155 includes a light source (not shown) that generates light (for example, laser light) for outputting an image to the outside, a light source And a lens (not shown) for enlarging and outputting the image at a predetermined focal distance to the outside. Further, the projector module 1155 may include a device (not shown) capable of mechanically moving the lens or the entire module to adjust the image projection direction.

The projector module 1155 can be divided into a CRT (Cathode Ray Tube) module, an LCD (Liquid Crystal Display) module and a DLP (Digital Light Processing) module according to the type of the display means. In particular, the DLP module may be advantageous in downsizing the projector module 1155 by enlarging and projecting an image generated by reflecting light generated by the light source on a DMD (Digital Micromirror Device) chip.

Preferably, the projector module 1155 may be provided longitudinally on the side, front or back side of the terminal 1100. Needless to say, the projector module 1155 may be provided at any position of the terminal 1100 as needed.

In addition, the output unit 1150 may further include a direction unit capable of generating smell information. The directional portion can generate a predetermined smell and odor from the outside of the terminal 1100.

The memory 1160 may store a program for processing and controlling the control unit 1180 and may store the input / output data (e.g., telephone directory, message, audio, still image, electronic book, History, and the like). The memory 1160 may also store the frequency of use of each of the data (e.g., each telephone number, each message, and frequency of use for each multimedia). In addition, the memory 1160 may store data related to vibration and sound of various patterns that are output upon touch input on the touch screen.

The memory 1160 may also store acoustic model data, such as a hidden Markov model, trained from speech data for each emotion for a plurality of emotions. Such acoustic model data is used by the emotion recognition means to determine the emotion of the user.

The memory 1160 can also be used to update acoustic model data by additionally storing feature parameters for the user's voice. The feature parameter represents acoustic parameters such as Mel-Frequency Cepstral Coefficient.

The memory 1160 may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory, etc.) ), A random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read- A magnetic disk, an optical disk, a memory, a magnetic disk, or an optical disk. The terminal 1100 may operate in association with a web storage that performs the storage function of the memory 1160 on the Internet.

The interface unit 1170 serves as a path for communication with all external devices connected to the terminal 1100. The interface unit 1170 receives data from an external device or supplies power to each component in the terminal 1100 or transmits data in the terminal 1100 to an external device. For example, a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device having an identification module, an audio I / O port, A video input / output (I / O) port, an earphone port, and the like may be included in the interface unit 1170.

The identification module is a chip for storing various information for authenticating the usage right of the terminal 1100 and includes a user identification module (UIM), a subscriber identity module (SIM), a universal user authentication module A Subscriber Identity Module (USIM), and the like. Devices with identification modules (hereinafter referred to as "identification devices") can be manufactured in a smart card format. Accordingly, the identification device can be connected to the terminal 1100 through the port.

When the terminal 1100 is connected to an external cradle, the interface unit 1170 may be a path through which power from the cradle is supplied to the terminal 1100 or various command signals input from the cradle by the user The terminal 1100 may be a channel to be transmitted to the terminal 1100. The various command signals input from the cradle or the power source may be operated as a signal for recognizing that the terminal 1100 is correctly mounted on the cradle.

A controller 1180 typically controls the overall operation of the terminal 1100. For example, voice communication, data communication, video communication, and the like. The control unit 1180 may include a multimedia module 1181 for multimedia playback. The multimedia module 1181 may be implemented in the control unit 1180 or may be implemented separately from the control unit 1180.

The control unit 1180 may perform a pattern recognition process for recognizing handwriting input or drawing input performed on the touch screen as characters and images, respectively.

In particular, the control unit 1180 may include emotion recognition means for recognizing the emotion using the user's voice by the terminal 1100 of the present invention. The emotion recognizing means may include a sensing unit for sensing voice of a user as a voice activity detection based on signal energy, an extracting unit for extracting characteristic parameters, and a recognizing unit for determining emotion of the user.

When the display unit 1151 is formed of an organic light-emitting diode (OLED) or a TOLED (Transparent OLED), the control unit 1180 controls the display unit 1151 When the preview image is pulled up on the screen of the organic light-emitting diode (OLED) or the TOLED (Transparent OLED) and the size of the preview image is adjusted according to the user's operation, The power consumption of the power source supplied from the power supply unit 1190 to the display unit 1151 can be reduced by turning off the driving of the pixels in the second area except for the first area in which the preview image is adjusted.

The power supply unit 1190 receives external power and internal power under the control of the controller 1180 and supplies power required for operation of the respective components.

The various embodiments described herein may be implemented in a recording medium readable by a computer or similar device using, for example, software, hardware, or a combination thereof.

According to a hardware implementation, the embodiments described herein may be implemented as application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays May be implemented using at least one of a processor, controllers, micro-controllers, microprocessors, and other electronic units for performing other functions. In some cases, The embodiments described may be implemented by the control unit 1180 itself.

According to a software implementation, embodiments such as the procedures and functions described herein may be implemented with separate software modules. Each of the software modules may perform one or more of the functions and operations described herein. Software code can be implemented in a software application written in a suitable programming language. The software code is stored in the memory 1160 and can be executed by the control unit 1180. [

2A and 2B illustrate a front perspective view and a rear perspective view of a terminal related to the present invention.

Referring to FIG. 2A, the disclosed terminal 1100 includes a bar-shaped body. However, the present invention is not limited thereto, and can be applied to various structures such as a slide type, a folder type, a swing type, and a swivel type in which two or more bodies are relatively movably coupled.

The body includes a case (a casing, a housing, a cover, and the like) which forms an appearance. In this embodiment, the case may be divided into a front case 1101 and a rear case 1102. [ Various electronic components are embedded in the space formed between the front case 1101 and the rear case 1102. [ At least one intermediate case may be additionally disposed between the front case 1101 and the rear case 1102. [

The cases may be formed by injection molding a synthetic resin, or may be formed to have a metal material such as stainless steel (STS) or titanium (Ti) or the like.

The front body 1101 mainly includes a display unit 1151, an audio output module 1152, a third camera 1121c, input units 1130, 1131 and 1132, a microphone 1122, an interface unit 1170, Can be arranged.

The display portion 1151 occupies most of the main surface of the front case 1101. An acoustic output module 1151 and a camera module 1121 are disposed in an area adjacent to one end of both ends of the display unit 1151 and an input unit 1131 and a microphone 1122 are disposed in an area adjacent to the other end. The input unit 1132 and the interface unit 1170 may be disposed on the side surfaces of the front case 1101 and the rear case 1102.

The third camera 1121c has a photographing direction substantially opposite to that of the first and second cameras 1121a and 1121b and may be a camera having the same or different pixels as the first and second cameras 1121a and 1121b have.

A flash and a mirror may be additionally disposed adjacent to the third camera 1121c. The flash illuminates the subject when the subject is photographed by the third camera 1121c. The mirror enables the user to illuminate the user's own face or the like in the case where the user desires to photograph (self-photograph) himself / herself using the third camera 1121c.

The input unit 1130 is operated to receive a command for controlling the operation of the terminal 1100 and may include a plurality of operation units 1131 and 1132. The operation units 1131 and 1132 may be collectively referred to as a manipulating portion.

Contents input by the first or second operation units 1131 and 1132 can be variously set. For example, the first operation unit 1131 receives commands such as start, end, scroll, and the like, and the second operation unit 1132 controls the size of the sound output from the sound output module 1152, The touch recognition mode can be activated or deactivated.

The microphone 1122 may be used to receive an external acoustic signal and process it as electrical voice data. In particular, the microphone 1122 may be used to record user's voice, which can be used for user's emotional recognition that can be applied to the present invention .

Referring to FIG. 2B, first and second cameras 1121a and 1121b for photographing the reverse direction of the third camera 1121c may be further mounted on the rear surface of the terminal body, that is, the rear case 1102. FIG.

An acoustic output module 1152 'may be additionally disposed on the rear surface of the terminal body. The sound output unit 1152 'may implement the stereo function together with the sound output module 1152 (see FIG. 2A) and may be used for the implementation of the speakerphone mode during the call.

In addition to the antenna for communication, a broadcast signal receiving antenna 1116 may be additionally disposed on the side of the portable terminal body. An antenna 1116 constituting a part of the broadcast receiving unit 1111 (see FIG. 1) may be installed to be able to be drawn out from the terminal body.

A power supply unit 1190 for supplying power to the terminal 1100 is mounted on the terminal body. The power supply unit 1190 may be built in the terminal body or may be detachable from the outside of the terminal body.

The rear case 1102 may further include a touch pad for sensing a touch. The touch pad may be of a light transmission type for the display portion 1151. [ In this case, if the display unit is configured to output the time information on both sides (i.e., in the directions of both the front and rear sides of the mobile terminal), the time information can be recognized through the touch pad. The information output on both surfaces may be controlled by the touch pad.

Meanwhile, the touch screen may be disposed in the rear case 1102 by separately mounting the touch pad dedicated display.

The touch pad operates in correlation with the display portion 1151 of the front case 1101. The touch pad may be disposed parallel to the rear of the display unit 1151. [ The touch pad may have the same size as the display portion 1151 or a small size.

Hereinafter, a related operation of the display unit 1151 and the touch pad will be described with reference to FIG. 3 is a front view of a terminal for explaining an operation state of a terminal related to the present invention.

Various types of time information can be displayed on the display unit 1151. [ These pieces of information can be displayed in the form of letters, numbers, symbols, graphics, or icons.

At least one of the letters, numbers, symbols, graphics, or icons may be displayed in a predetermined arrangement for inputting such information, thereby being implemented as a keypad. Such a keypad may be referred to as a so-called " virtual keypad ".

3 illustrates inputting of a touch applied to the virtual keypad through the front surface of the terminal body.

The display unit 1151 may operate as an entire area or may be divided into a plurality of areas and operated. In the latter case, the plurality of areas can be configured to operate in association with each other.

For example, an output window 1151a and an input window 1151b are displayed at the top and bottom of the display unit 1151, respectively. The output window 1151a and the input window 1151b are areas allocated for outputting or inputting information, respectively. In the input window 1151b, a virtual keypad 1151c is displayed in which a number for inputting a telephone number or the like is displayed. When the virtual keypad 1151c is touched, a number or the like corresponding to the touched virtual keypad is displayed on the output window 1151a. When the first operation unit 1131 is operated, call connection to the telephone number displayed on the output window 1151a is attempted.

In addition to the input method described in the above embodiments, the display unit 1151 or the touch pad may be configured to receive a touch input by scrolling. The user can move a cursor or a pointer located on an object displayed on the display unit 1151, for example, an icon or the like, by scrolling the display unit 1151 or the touch pad. Further, when the finger is moved on the display portion 1151 or the touch pad, the path along which the finger moves may be visually displayed on the display portion 1151. [ This may be useful for editing the image displayed on the display unit 1151. [

One function of the terminal may be executed in response to a case where the display portion 1151 (touch screen) and the touch pad are touched together within a predetermined time range. In the case of being touched together, there may be a case where the user clamps the terminal body using the thumb and index finger. The one function may be, for example, activation or deactivation of the display portion 1151 or the touch pad.

On the other hand, an arrow or finger graphic for pointing a specific object or selecting a menu in the display unit 1151 is referred to as a pointer or a cursor. However, in the case of a pointer, it often means a finger or a stylus pen for a touch operation or the like. Therefore, in this specification, a graphic displayed on the display unit is referred to as a cursor, and a physical means for performing a touch, a proximity touch, and a gesture, such as a finger or a stylus pen, is referred to as a pointer.

The emotion recognizing means using speech according to the present invention classifies four emotions (for example, neutral, joy, anger and sadness) felt by a target object by sensing an object to be emotionally recognized, that is, can do. However, the four types of emotions classified are merely examples for application of the present invention, and emotions can be classified into a plurality of emotions by different criteria.

The emotion recognizing means that can be applied to the present invention can recognize the voice of the object by receiving the object information of the object responding to the presented stimulus. At this time, the object is the object to which the emotion is to be recognized. An object may be a person, or it may be an animal that wants to recognize emotion in order to recognize the emotion of a particular animal.

The object information is data of a subject to be measured, which is mainly recorded with the voice of the subject to be measured.

In one embodiment of the object information, when the voice of the object is recorded through the microphone, the digital data generated from the object may be the object information.

In addition, the emotion recognition means can extract the feature parameter of the speech of the object using the speech of the object, and can determine the emotion of the object by using the information about the feature parameter of the speech corresponding to each emotion.

Hereinafter, the emotion recognition application applied to the present invention will be described in detail. 4 shows an embodiment of an execution window by execution of the emotion recognition application according to the present invention.

When a user of the terminal 1100 inputs an execution command of the emotion recognition application using the input unit 1130, a predetermined execution window associated with the emotion recognition means is displayed on the display unit 1151. [

4, the name of the application called "OnlineEmotion Recognition" appears at the top of the execution window.

The "Start Record" button is a button for starting user's voice recording. When the "Start Record" button is pressed, voice recording is started. When a silent (non-voice) is detected, the recognized emotion of the user is displayed on the screen based on the voice inputted until then.

The "Play Sound" button is a button for playing back the recorded voice so far. When the "Play Sound" button is pressed, a list of the recorded voice is displayed, and the voice selected by the user is played back.

"Male" and "Female" are means for selecting the gender of a user who inputs a voice. The present invention uses a gender emotion model to improve the performance of emotion recognition because emotional recognition characteristics are different between men and women. That is, the emotion is recognized using the male emotion model while the voice recorded while the "Male" is selected, and the voice recorded while the "Female" is selected is recognized using the female emotion model.

The execution window of the emotion recognition application may additionally include an option to set the start point of the voice recording and the voice recording time in addition to the buttons. In addition, the execution window has a button for an update command, so that voice data for the user can be added to the memory 1160. [

Hereinafter, a method of recognizing an emotion using the voice of an object through the emotion recognition means will be described in detail.

As described above, the emotion recognition means may be constituted by a sensing unit, an extracting unit, and a recognizing unit.

The sensing unit performs voice activity detection based on signal energy and can detect the user's voice from the first voice recorded by the user.

5A and 5B schematically show an embodiment of a process of extracting a feature parameter of an extracting section of the emotion recognition means.

5A and 5B, the extracting unit extracts a feature parameter from a sound source including a user's voice recorded by the microphone 1122, and the feature parameter includes a Mel-Frequency Cepstral Coefficient (MFCC) and a log-energy .

Referring to FIG. 5A, the MFCC applies Fast Fourier Transform (FFT) to a voice of a user through a Hamming Window, applies a Mel-scale filter bank to the result of applying the FFT to obtain a power spectrum, Can be obtained by taking a log and applying DCT (Discrete Cosine Transform) to the result of taking the log.

Referring to FIG. 5B, the feature parameter is obtained by extracting a 39-th real number column for every frame of the recorded voice. Here, the length of the frame may be set to 30 ms.

The real column of 39th column consists of three 13th real columns. The first 13th real number sequence is obtained by using MFCC (12th order) and Log-energe (1st order) extracted from the current frame, and the second 13th real number sequence is obtained by using the element-specific difference between the current frame and the first- And the 13th real number sequence in the third round is obtained using the element-by-element difference between the current frame and the second previous frame.

6A and 6B schematically illustrate an embodiment of the process of training and recognizing emotions by the recognition unit of the emotion recognition means, and FIG. 7 shows experimental results of recognition rate and recognition time according to the emotion recognition means of the present invention .

Referring to FIG. 6A, in the training process of the emotion recognition means, an emotion model is constructed using the feature parameters extracted by the extraction unit. In this way, an acoustic model based on a plurality of emotions can be constructed, which can be stored in the memory 1160 and used for user's emotion judgment. These emotional models use feelings expressed naturally while many experiment participants indirectly experience a specific situation.

The construction of the emotion model according to the present invention is useful for practical use because it uses a voice similar to a voice that is naturally expressed in a real situation, and is advantageous for generalization because of a large number of speakers. Experimental results of the recognition rate and the recognition time of the emotion recognition means according to the present invention are shown in FIG. As shown in FIG. 7, the terminal equipped with the emotion recognition application of the present invention shows a very high emotion recognition rate.

Referring to FIG. 6B, in the recognition process of the emotion recognition unit, the feature parameters extracted by the extraction unit are used. The recognition unit compares the feature parameter extracted from the input speech with the acoustic model of each emotion stored in the memory 1160, and determines the emotion corresponding to the acoustic model representing the highest probability value as the emotion of the user.

Meanwhile, FIG. 8 schematically shows a process in which the emotion recognition means of the present invention recognizes a user's emotion.

Referring to FIG. 8, the emotion recognizing means senses the user's voice from the original voice, and the signal energy can be used in the process of sensing the user's voice. After detecting the user's voice, the feature extraction unit extracts feature parameters including MFCC and Log-energy. The feature parameter of the user voice is compared with the acoustic model of each emotion stored in the memory 1160, and the emotion of the user can be accurately determined based on the recognition result obtained in this process.

The plurality of emotions classified here may be neutral, joyful, anger, sad, etc., as described above. However, the contents of the present invention are not limited thereto, and it is possible to classify them into a plurality of emotions (for example, surprise, disgust, fear, etc.) according to other criteria.

When a terminal equipped with the emotion recognition application of the present invention receives an execution command of an emotion recognition application built in the terminal 1100 from a user, a predetermined execution window associated with the emotion recognition means is displayed on the display unit 1151. [ The execution window may appear as a pop-up on the display unit 1151. [ The user inputs the recording information related to the recording of the user's voice through the execution window and records the first voice including the user's voice using the microphone 1122. [

When the first voice is recorded, the first voice is transmitted to the emotion recognizing means, and the emotion recognizing means recognizes the emotion recognized by the user among the plurality of emotions using the acoustic model data stored in the memory 1160 and the first voice, .

The emotion recognizing process of the emotion recognizing means includes sensing the voice of the user recorded on the first voice based on the signal energy of the first voice, Extracting the parameter, and comparing the first characteristic parameter with the acoustic model data.

Then, the output unit 1150 outputs a response corresponding to the emotion of the user recognized based on the first voice. The reaction may be performed using a display portion 1151, an acoustic output module 1152, a haptic module 1155, a directional portion, or the like. The outputs according to the display unit 1151, the sound output module 1152, the haptic module 1155, and the like are reactions having a specific pattern stored in advance in the memory 1160.

The response of the output unit 1150 may include a text, a backdrop or an emoticon display on the display unit 1151, a sound output set by the sound output module 1152, a vibration of a vibration having a predetermined intensity and pattern in the haptic module 1155 And fragrance generated in the direction portion.

Also, when the user is talking with the other party using the terminal 1100 according to the present invention, the terminal 1100 can not only recognize the emotion of the user but also recognize the emotion of the other party. In this case, the output unit 1150 of the terminal 1100 can output an appropriate response corresponding to the emotion of the recognized other party, thereby allowing the user to better understand the emotion of the calling party. Of course, the user may be provided with a reaction according to his / her emotional state, thereby helping to alleviate emotions of the user himself / herself.

The present invention can also be embodied as computer-readable codes on a computer-readable recording medium. A computer-readable recording medium includes all kinds of recording apparatuses in which data that can be read by a computer system is stored. Examples of the computer-readable recording medium include a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like, and may be implemented in the form of a carrier wave (for example, transmission via the Internet) . The computer readable recording medium may also be distributed over a networked computer system so that computer readable code can be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the present invention can be easily inferred by programmers of the technical field to which the present invention belongs.

It should be understood that the above-described apparatus and method are not limited to the configuration and method of the embodiments described above, but the embodiments may be modified so that all or some of the embodiments are selectively combined .

Claims (16)

A terminal capable of recognizing a plurality of emotions classified according to a preset reference using a user's voice,
A microphone for acquiring a first voice including the voice of the user;
A memory for storing acoustic model data related to the plurality of emotions;
Emotion recognition means for receiving the first voice and recognizing the emotion of the user among the plurality of emotions using the acoustic model data and the first voice;
An output unit including a display unit for displaying time information, an acoustic output module for outputting auditory information, a haptic module for generating vibration, and an orientation unit for generating smell information; And
And an input unit for receiving a command related to an operation of an emotion recognition application installed in the terminal from the user,
Wherein the acoustic model data is information on a feature parameter of the speech corresponding to each of the plurality of emotions,
When the user inputs an execution command of the emotion recognition application using the input unit, a predetermined execution window associated with the emotion recognition unit is displayed on the display unit to induce the recording of the first sound, Outputting a response corresponding to the emotion of the user recognized based on the first voice,
Wherein the emotion recognition means comprises:
A sensing unit for sensing a voice of the user recorded on the first voice based on a signal energy of the first voice;
An extracting unit for extracting a first feature parameter from the detected voice of the user; And
And a recognition unit for comparing the first characteristic parameter with the acoustic model data to determine an emotion of the user,
Wherein the recognition unit calculates an observation probability value of the acoustic model data and the first characteristic parameter stored in the memory and stores the emotion corresponding to the acoustic model data having the highest observation probability value among the calculated observation probability values in the emotion of the user However,
The acoustic model data stored in the memory is generated based on a hidden Markov model,
Wherein acoustic model data for the user is generated based on the hidden Markov model and is additionally stored in the memory when the user inputs the update execution command displayed in the execution window using the input unit. The method according to claim 1,
The above-
A first data display in the display unit, an acoustic output in the acoustic output module, a generation of vibration having a predetermined intensity and pattern in the haptic module, and a perfume generated in the direction unit,
Wherein the first data is at least one of preset text, a backdrop, and an emoticon. The method according to claim 1,
Wherein the execution window receives recording information from the user and uses the recording information to record the first audio,
Wherein the recording information includes at least one of a gender of the user, a recording start time of the first audio, and a time of recording the first audio. The method of claim 3,
In the memory,
Wherein different acoustic model data are stored according to the gender. delete The method according to claim 1,
Wherein the feature parameter extracted by the extraction unit includes a Mel-Frequency Cepstral Coefficient (MFCC) and a log energy. The method according to claim 6,
The MFCC,
A Fast Fourier Transform (FFT) is applied to the user's voice sensed by the sensing unit, a power spectrum is obtained by applying a Mel-scale filter bank to the FFT applied result, and a log is input to the power spectrum , And applying DCT (Discrete Cosine Transform) to the result of taking the log. delete delete delete The method according to claim 1,
Wherein the plurality of emotions classified in accordance with the preset criteria include:
Neutral, joy, anger and grief. ≪ Desc / Clms Page number 13 > The method according to claim 1,
When the user is in a call with the other party using the terminal,
The terminal recognizes the emotion of the other party using the emotion recognition means,
Wherein the output unit outputs the response in response to the emotion of the recognized other party. A method of controlling a terminal capable of recognizing a plurality of emotions classified according to a predetermined reference using a user's voice,
A first step of receiving an execution command of an emotion recognition application installed in the terminal from the user;
A second step of displaying a predetermined execution window associated with the emotion recognition means on the display unit;
A third step of the user inputting the recording information of the first voice including the voice of the user;
A fourth step of recording the first voice;
A fifth step of transmitting the first voice to the emotion recognizing means;
A sixth step of recognizing the emotion of the user among the plurality of emotions using the acoustic model data stored in the memory and the first voice; And
And an output unit outputting a response corresponding to the user's emotion recognized based on the first voice,
Wherein the acoustic model data is information on a feature parameter of the speech corresponding to each of the plurality of emotions,
Wherein the output unit comprises a display unit for displaying time information, an acoustic output module for outputting auditory information, a haptic module for generating vibration, and a direction unit for generating smell information,
In the sixth step,
Detecting a user's voice recorded on the first voice based on a signal energy of the first voice;
A sixth step of extracting a first feature parameter from the detected voice of the user; And
And (6-3) comparing the first characteristic parameter with the acoustic model data to determine the emotion of the user,
In step 6-3, the acoustic model data stored in the memory and the observation probability value of the first characteristic parameter are calculated, and the emotion corresponding to the acoustic model data having the highest observation probability value among the calculated observation probability values is stored in the memory Judging the user's emotions,
The acoustic model data stored in the memory is generated based on a hidden Markov model,
Wherein when the user inputs the update execution command displayed in the execution window using the input unit, acoustic model data for the user is generated based on the hidden Markov model and is additionally stored in the memory. Control method. 14. The method of claim 13,
The above-
A first data display in the display unit, an acoustic output in the acoustic output module, a generation of vibration having a predetermined intensity and pattern in the haptic module, and a perfume generated in the direction unit,
Wherein the first data is at least one of preset text, a backdrop, and an emoticon. delete 14. The method of claim 13,
Wherein the plurality of emotions classified in accordance with the preset criteria include:
Wherein the at least one of the at least one of the plurality of terminals comprises at least one of a neutral, joy, anger, and sadness.

Images