WO2016043570A1 - Terminal device, method for driving terminal device, and computer readable recording medium - Google Patents

Abstract

The present invention relates to a terminal device, a method for driving a terminal device, and a computer readable recording medium. A terminal device according to an embodiment of the present invention may comprise: a storage unit for storing sensory information associated with an emotional state of a user of the terminal device, expressed in a content or with an atmosphere of an object present in a content; a sensory information extraction unit for extracting the sensory information associated with the emotional state or the atmosphere when the content is input; and a control unit for performing a haptic-related operation using the extracted sensory information.

Description

Terminal device, method of driving terminal device and computer readable recording medium

The present invention relates to a terminal device, a method of driving the terminal device, and a computer-readable recording medium, and more particularly, to a terminal device such as a smartphone or a wearable device, which is represented in content such as video, audio, and text. A terminal device, a method of driving the terminal device, and a computer-readable recording medium, which can express a user's emotional state or an atmosphere of an object as a haptic, for example, vibration.

Recent research in the field of emotional computing has focused on building emotionally intelligent computers. Most of this research has focused on developing ways in which machines perceive users' emotions. Typical examples in the field of emotional computing are cognitive tutors that can warn of the display of content based on assistive devices that help individuals in the learner's emotional state and autism spectrum to detect the emotional state of the communication partner. tutors). Such systems may be seen to have emotional perceptual intelligence that recognizes the user's emotional state and is adapted accordingly.

The presence of expressive intelligence, the ability to provide emotionally intelligent feedback, is equally important because the system is emotionally intelligent. This emotional feedback can be either monomodal, such as visual, auditory, or tactile, or multimodal, such as visual-haptic, auditory-haptic. Can be.

Current research on multiple mode interfaces mainly focuses on how effectively multiple information can be presented and delivered to end users simultaneously using different methods. Multimodal interfaces can also be used to provide emotionally intelligent emotional feedback to users.

Cognitive psychology research in the field of multisensory integration requires that the cues provided in all modes be congruent with each other so that multimodal feedback is effective, as well as the cues provided in one mode are effective in the cues belonging to the other. It may be distorted or invalidated.

The study of multimodal interfaces was inspired by the fact that in the real world people interact with their environment in a multimodal fashion. Multimodal interfaces can allow for intuitive and efficient real world-like interactions. Such an interface creates immersion (e.g. in virtual reality), enables efficient attention management (e.g., alerts on a mobile phone), and (eg, ultrasonic fracturing of data) It has been explored for other uses, such as sonification, to better understand complex information.

A vast amount of research on multimodal interfaces has focused on how effectively information can be encoded in different perceptual modalities. The most influential theory in this area is the Multiple Resource Theory (MRT), which has the position that there are fixed capacity resources available for each approach. Meta-analysis of visual and auditory and tactile feedback shows that visual and auditory feedback is most effective when a single task is performed, and that tactile feedback is effective in a multitasking environment. have.

Research in cognitive psychology has been focused on how information is integrated in the aggregated perception of the human brain unified from multiple input channels. A collision paradigm in which the two approaches receive incongruent inputs has been used to study how humans integrate information. Interactions in the field of vision-haptic, hearing-hearing and hearing-haptic have been explored using this paradigm. Some of the classic results from this paradigm are discussed below.

In the Ventriloquist effect, the spatial location of the voice stimulus may be affected by synchronized (corresponding to) visual stimuli. Participants could hear many flashes of sound bursts and see flashing lights (synchronized with many sounds) coming from spatially distinct resources. They are asked to point toward the sound source while ignoring the flashes of light. The results show that participants typically change their attitude towards the position of the light strobe.

Rubber hand illusion is about how cognition of touch can be affected by visual feedback. When subjects see rubber hands swung, they report feelings in rubber hands while their invisible hands are also swung at the same time. This illusion occurs because the human brain misunderstands tactile perception based on visual perception.

Parchment skin illusion is an example of the interaction between voice and tactile stimuli. Participants are asked to rub their palms. This sound is recorded, accentuated and corrected to frequencies above 2 KHz, and played back to the subjects via headphones. Participants report that they feel very dry, like parchment paper, when they hear the stressed feedback.

These results demonstrate that multimodal sensory integration is not a directly related process. This assumes that humans tend to behave like maximum-likelihood integrators and combine information from multiple sensory modalities in a statistically optimal way. Similar results were shown when participants were asked to integrate emotion cues from multiple ways.

The emotional McGurk effect demonstrates how humans integrate facial expressions and rhyme cues to recognize affection. Participants provided static faces representing multiple emotions with speech sounds changing from rhymes to express different emotions. The results of the emotional confirmation show that both visual bias and negative bias depend on the reliability of the stimulus.

Cognitive psychology research in multisensory affect integration has mostly focused on speech-visual integration, especially facial expressions and voice rhymes. Touch is a very powerful way to convey emotions, and how lean is the affection transmitted through visual and haptic ways or voice and haptic interactions.

However, there are few studies on how the cues represented by the multiple methods interact emotionally. With this in mind, there is a need for interaction designers to combine emotional cues in tactile ways in order to consistently elicit the emotional responses desired by end users.

Also, to design an emotional multisensory interface, knowledge of how emotionally integrated cues from these approaches are integrated into a unified perception is essential. In this regard, it is necessary to examine how emotionally the visual and haptic methods interact, and there is a need for an authoring tool that interaction designers can use to generate haptic feedback given visual information.

An embodiment of the present invention is a terminal device capable of expressing a user's emotional state or an atmosphere of an object expressed in content such as video, audio, and text in a terminal device such as a smartphone or a wearable device as a haptic, for example, a vibration. It is an object of the present invention to provide a method of driving a device and a computer-readable recording medium.

The terminal device according to an embodiment of the present invention is a storage unit for storing the sensory information related to the emotional state of the user of the terminal device expressed in the content or the atmosphere of the object present in the content, the content is to be input And a sensory information extractor configured to extract sensory information related to the emotional state or the mood, and a controller configured to perform a haptic related operation using the extracted sensory information.

The terminal device further includes a surrounding situation recognition unit for recognizing a surrounding situation to generate the content, and a haptic operation unit performing the haptic related operation, and the controller controls the haptic operating unit based on the sensory information. can do.

The surrounding situation recognition unit may include at least one of an image capturing unit photographing the surrounding situation and a voice obtaining unit obtaining the voice of the surrounding situation.

The terminal device further comprises a user interface for receiving a user command for adjusting the emotional state of the user expressed in the content, and a communication interface for transmitting the sensory information to an external device for performing the haptic-related operation, The storage unit may store state control information corresponding to the user command by matching the sensory information, and the controller may control the communication interface to transmit the sensory information to the external device.

The user interface may receive a user command for expressing an emotion that matches the emotional state or a user command for expressing an emotion opposite to the emotional state.

The content may include at least one of video content, audio content, and text content.

The sensory information extractor may include a content feature recognizer that recognizes the emotional state or the mood, a mapper that performs a mapping operation to find sensory information matching the recognized emotional state or the mood, and the found sensory information. It may include a parameter extraction unit for extracting the parameters.

The parameter extractor may extract the sensory information by further considering characteristics of an external device that receives the sensory information when the sensory information is extracted.

The sensory information may include haptic information for the haptic related operation, and the haptic information may include at least one parameter of intensity, frequency, wavelength, and rhythm.

Each of the parameters may include a plurality of levels, and the haptic information may include a parameter corresponding to one of the plurality of levels.

In addition, the method of driving a terminal device according to an embodiment of the present invention, storing the sensory information related to the emotional state of the user of the terminal device expressed in the content or the atmosphere of the object present in the content, the content is input And extracting sensory information related to the emotional state or the mood, and performing a haptic related operation using the extracted sensory information.

The driving method of the terminal device further includes a step of recognizing a surrounding situation and performing the haptic related operation to generate the content, wherein the processing comprises the haptic related operation based on the sensory information. Can be controlled.

The step of recognizing the surrounding situation may include at least one of photographing the surrounding situation and acquiring a voice of the surrounding situation.

The driving method of the terminal device further includes receiving a user command for adjusting the emotional state of the user expressed in the content, and transmitting the sensory information to an external device performing the haptic-related operation, The storing may include storing state control information corresponding to the user command with the sensory information, and the processing may be controlled to transmit the sensory information to the external device.

The receiving of the user command may include receiving a user command to express an emotion corresponding to the emotional state or a user command to express an emotion opposite to the emotional state.

The content may include at least one of video content, audio content, and text content.

The extracting of the sensory information may include: recognizing the emotional state or the mood, performing a mapping operation to find sensory information matching the recognized emotional state or the mood, and parameters of the found sensory information. And extracting the parameter may extract the sensory information by further considering characteristics of an external device that receives the sensory information when the sensory information is extracted.

The sensory information may include haptic information for the haptic related operation, and the haptic information may include at least one parameter of intensity, frequency, wavelength, and rhythm.

Each of the parameters may include a plurality of levels, and the haptic information may include a parameter corresponding to one of the plurality of levels.

Furthermore, a computer-readable recording medium including a program for executing a method of driving a terminal device, wherein the method of driving the terminal device includes an emotion state of a user of the terminal device expressed in content or an object present in the content. Setting sensory information related to an atmosphere of the user, extracting sensory information related to the emotional state or the mood when the content is input, and performing a haptic related operation using the extracted sensory information The step of outputting the sensory information may be performed.

-

1 is a diagram showing an emotional service system according to an embodiment of the present invention;

2 is a diagram showing an emotional service system according to another embodiment of the present invention;

3 is a block diagram illustrating a detailed structure of a terminal device 1 of FIG. 1;

4 is a block diagram showing a detailed structure of a terminal device 1 shown in FIG. 1 or a terminal device shown in FIG. 2;

5 is a diagram illustrating an example of the sensory information extraction unit of FIGS. 3 and 4;

6 is a view for explaining a service process related to the system of FIG.

7 is a diagram illustrating a specific operation of the terminal device 1 shown in FIG. 6;

8 is a flowchart illustrating a driving process of a terminal device according to an embodiment of the present invention;

9 is a flowchart illustrating a driving process of the terminal device 1 shown in FIG. 1;

10 is a flowchart illustrating a driving process of the terminal device shown in FIG. 2;

11 is a diagram illustrating a driving process of a terminal device 1 of FIG. 1 or a terminal device of FIG. 2 according to another embodiment of the present invention; and

12 is an exemplary view for explaining the authoring tool constructed in accordance with the guidelines generated by the auditory tactile experiment.

-

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram illustrating an emotional service system according to an exemplary embodiment of the present invention, which relates to adjusting an emotion of a receiving user who receives content.

As shown in FIG. 1, the emotional service system 90 according to an embodiment of the present invention may include a terminal device 1 (100), a terminal device (110), a communication network (120), and a service providing device (130). Can be.

Here, the terminal device 1 100 and the terminal device 2 110 include a mobile device such as a smart phone, a notebook computer, an MP3, and a PDP, and a wearable device such as a gearbox. Of course, it is not intended to be particularly limited to such a device, and any device may be used as long as it can express tactile senses such as haptic motion as well as multiple senses such as vision and hearing.

For example, the terminal device 1 100 may transmit video, audio, or text content to the terminal device 2 110, which may well express its emotional state as a calling terminal. For example, the terminal device 1 (100) may provide a captured image that can express its own happy state, or select and provide other video content that can replace it to the terminal device 2 (110). In this process, the user of the terminal device 1 (100) may allow the emotional state expressed in the content to be delivered to the terminal device 2 (110) as it is, but, for example, when the terminal device 2 (110) transmits a depressed or bad state. ) So that the emotional state may be transmitted in a state in which the user of) does not panic.

In other words, the terminal device 2 110 may perform a haptic operation such as vibration before receiving the content from the terminal device 1 100 and expressing the received content. Therefore, if the haptic motion expresses a depressed or bad state, the user of the terminal device 2 110 may be very nervous. In this regard, the user of the terminal device 1 (100) may specify that an emotional state opposite to the specific content is expressed by the haptic motion.

In addition, the terminal device 1 (100) may be a content created by the user, and of course, a content that can replace the same, but receives a user command to control and express the emotional state expressed in the content and the emotional state. Then, the emotional state is recognized from the content, and the sensory information for the haptic operation, that is, the parameters are extracted based on the state control information for adjusting the emotional state based on the recognized emotional state and the user command. The extracted parameter information may be transmitted to the terminal device 2 110 that performs the haptic operation.

In order to perform this function, the terminal device 1 100 may store a program for processing sensory information, for example, an application (hereinafter, referred to as an app), and execute the same. For example, a user of the terminal device 1 (100) touches an icon displayed on a screen to execute a corresponding app, and through this, a content for expressing his or her emotional state and a user command for adjusting the emotional state expressed in the content. You will be able to enter For example, even if the content indicates the same emotional state, it may be regarded that other sensory information is transmitted to the terminal device 2 110 according to a user command related to the state control information.

However, the sensory information may be different from each other depending on the characteristic information of the terminal device 110, for example, the specification. For example, even if they have the same content and the same state control information, the parameters of the sensory information output may be somewhat different, since the specification of the receiving terminal device 2 (110) should be considered. To this end, in the process of extracting the sensory information, the terminal device 1 (110) can be stored in advance and used therein, but the service providing device (130) for the characteristic information of the receiving terminal device (110), for example, for the specification. You can request and use the information. Alternatively, all extracted parameters may be transmitted to the terminal device 2 110, and the terminal device 2 110 may extract and use the parameter corresponding to the terminal device 110. This may be possible because the device information about the characteristics of the terminal devices are included therein.

The terminal device 2 110 is not significantly different from the terminal device 1 100. However, there is a difference in that it is a receiving terminal. The terminal device 2 110 may perform a haptic operation according to the sensory information provided by the terminal device 1 100, which is a calling terminal. For example, this haptic action can be made in a vibrating manner. When the user of the terminal device 2 110 wants to check the received content, the corresponding content may be executed. For example, in the case of a video, the corresponding video will be displayed on the screen, and in the case of a voice file, the corresponding video file may be executed and output to a voice output unit such as a speaker. Also, in the case of a text message, the text message can be displayed on the screen.

In addition, the terminal device 2 110 may analyze the received parameters and perform a haptic operation according to the analysis result. In addition, when the terminal device 1 (100) fails to select and provide parameters suitable for its characteristics, it may be additionally performed to extract a parameter corresponding to itself from among many parameters.

The communication network 120 includes both wired and wireless communication networks. Here, the wired network includes an internet network such as a cable network or a public telephone network (PSTN), and the wireless communication network includes a CDMA, WCDMA, GSM, Evolved Packet Core (EPC), Long Term Evolution (LTE), WiBro network, and the like. to be. Of course, the communication network 120 according to an embodiment of the present invention is not limited thereto, and may be used as, for example, a cloud computing network under a cloud computing environment as an access network of a next generation mobile communication system to be implemented. For example, when the communication network 120 is a wired communication network, an access point in the communication network 120 may access an exchange or the like of a telephone company, but in the case of a wireless communication network, data may be connected to an SGSN or a gateway GPRS support node (GGSN) operated by a communication company. Or, it can be connected to various repeaters such as BTS (Base Station Transmission), NodeB, e-NodeB and process data.

The communication network 120 may include an access point. Access points include small base stations, such as femto or pico base stations, which are frequently installed in buildings. Here, the femto or pico base station is classified according to the maximum number of terminal devices 1 (100) and the terminal device 2 (110) can be connected according to the classification of the small base station. Of course, the access point includes a short-range communication module for performing short-range communication such as Zigbee and Wi-Fi with the terminal 1 (100) and the terminal 2 (110). The access point may use TCP / IP or Real-Time Streaming Protocol (RTSP) for wireless communication. Here, the short-range communication may be performed in various standards such as Bluetooth, Zigbee, Infrared (IrDA), Ultra High Frequency (UHF) and Very High Frequency (VHF) and Ultra Wideband Communication (UWB). Can be. Accordingly, the access point extracts the location of the data packet, specifies the best communication path for the extracted location, and transmits the data packet along the designated communication path to the next device, such as the terminal device 1 (100) and the terminal device 2 (110). Can be delivered to. An access point may share several circuits in a general network environment, and may include, for example, a router, a repeater, and a repeater.

The service providing apparatus 130 may store an app for providing an emotional service according to an embodiment of the present invention, and provide an app stored at the user's request. In addition, the service providing device 130 may provide device characteristic information of the receiving terminal device 2 (110) at the request of the terminal device 1 (100). Furthermore, the service providing apparatus 130 may store and execute a program, ie, a tool, for processing all information related to an app executed in the terminal device 1 (100) as a concept of a cloud device. For example, the terminal device 1 110 may receive a user command for adjusting content and an emotional state expressed in the content. Then, the service providing device 130 may extract the sensory information based on this, and provide the sensor information to the terminal device 2 (110).

On the other hand, the service providing device 130 may provide an open API (Application Program Interface) to the third party. In other words, the service providing apparatus 130 may operate as a third party platform. Accordingly, the service providing device 130 may store various types of Apple provided by a third party. Such an apple may be related to devices compatible with the terminal device 1 (100) and the terminal device 2 (110), and may also be an apple that interworks with an existing service or content.

2 is a diagram illustrating an emotional service system according to another embodiment of the present invention, which relates to operating in a standalone state.

As shown in FIG. 2, the emotional service system 90 ′ according to another embodiment of the present invention may include some or all of the terminal device 200 and the peripheral device 210.

Here, some or all means that the peripheral device 210 may be omitted and configured, for example, the terminal device 200 may detect the state of the surrounding environment, not the state of the peripheral device 210. In order to fully understand the invention, it will be described as including all.

The terminal device 200 is not significantly different from the terminal device 1 100 described with reference to FIG. 1. However, the terminal device 200 of FIG. 2 does not use video, audio content, etc. that can photograph itself or express its emotional state like the terminal device 1 (100) of FIG. As shown in FIG. 210, an image of an image displayed on a screen of a movie theater may be an emotional state expressed in the captured image, more precisely, an atmosphere of an object, which is expressed by any direct haptic motion. Of course, the terminal device 1 100 of FIG. 1 may also use contents related to the surrounding situation or the situation of the peripheral device 210 as in the terminal device 200 of FIG. 2.

However, the terminal device 1 100 of FIG. 1 is intended to express its emotional state by transmitting the content to the receiving terminal device 2 110 that receives the content, and the terminal device 200 of FIG. The difference is that you can feel this haptic motion.

3 is a block diagram illustrating a detailed structure of the terminal device 1 and the terminal device of FIG.

For convenience of description, referring to FIG. 3 together with FIG. 1, the terminal device 1 100 according to an exemplary embodiment of the present invention may partially or entirely include the control unit 300, the storage unit 310, and the sensory information extraction unit 320. It may further include a user interface unit.

Here, the inclusion of some or all includes some components such as the user interface unit or the storage unit 310 being omitted, or some components such as the storage unit 310 are integrated with the sensory information extracting unit 320. It means that things, etc., and to include a telegram in order to help a full understanding of the invention.

In consideration of the situation of FIG. 1, the user interface includes a button input unit and a display unit, through which a request for execution of an emotional service for multisensory feedback according to an embodiment of the present invention may be requested. For example, the associated program may be executed through the button input unit, or the associated program may be executed by touching an icon displayed on the screen of the display unit.

In addition, the user interface unit adjusts its emotional state to any content, for example, video content, voice content, or text content that is pre-stored in the storage unit 310 or provided by the user, in some form. User commands as to whether to deliver to the user at 110 can be specified. For example, the user command may be a user command corresponding to an emotional state expressed in the content, or may be a user command for transmitting an opposite emotional state. In other words, even though the content may indicate a bad state, the haptic gesture may input a user command to indicate an opposite state of joy.

The controller 300 may control overall operations of the user interface unit, the storage unit 310, and the sensory information extraction unit 320 in the terminal device 1 (100). For example, when a user command for arbitrary content is input through the user interface, the sensory information extracting unit 320 receives a user command related to the corresponding content and the inputted user command, that is, state control information for adjusting an emotional state. Can be provided as

In addition, the controller 300 may provide the information requested by the sensory information extractor 320 by controlling the storage 310. For example, when sensory information (or state information) representing various emotional states of the content and state control information matched thereto are stored in the storage unit 310, the sensory information mapped to the sensory information and state control information is stored. May be provided to the sensory information extraction unit 320. In this process, the controller 300 may further output sensory information in consideration of device characteristic information of the terminal device 2 110 and provide the sensory information to the sensory information extracting unit 320.

The controller 300 may include an image processor. In other words, the controller 300 may be involved in not only transmitting and receiving general data and information through the image processor, but also transmitting sensory information extracted through the sensory information extractor 320 to the terminal device 2 110. In this case, various operations may be additionally performed, such as being involved in data conversion processes such as encoding and decoding.

The storage unit 310 may store various information processed by the controller 300. In addition, the storage unit 310 may store sensory information representing various emotional states that can be expressed in content and state control information corresponding thereto. Here, the sensory information refers to information related to sight, touch, and hearing, which means a direct emotional state such as 'happy' and 'sad' expressed in the content. Also, the sensory information may mean haptic information including various levels of vibration, frequency, rhythm, intensity, waveform, and the like, when the haptic state is expressed in the tactile sense. Such haptic information may be a parameter of sensory information.

It may be preferable that the sensory information including the haptic information is substantially set and processed in the sensory information extractor 320. Alternatively, the above operation may be performed through a separate storage unit directly interworking with the sensory information extracting unit 320, and thus the present invention will not be limited to the above description in the embodiment of the present invention.

The sensory information extracting unit 320 may extract sensory information based on the state control information designated by the user through the user interface unit and the emotional state expressed in the content. To this end, the sensory information extracting unit 320 may perform a process of extracting sensory information by using the data stored in the storage 310 in cooperation with the controller 300 or by directly accessing the storage 310. have.

To this end, the sensory information extracting unit 320 may store a program for performing an operation of extracting sensory information therein, and execute the program when the user requests it.

4 is a block diagram illustrating another detailed structure of the terminal device 1 shown in FIG. 1 or the terminal device shown in FIG. 2.

For convenience of description, referring to FIG. 4 together with FIG. 2, the terminal device 200 according to an embodiment of the present invention includes a communication interface 400, a user interface 410, a context recognizer 420, and a controller 430. ), The sensory information extraction unit 440, the haptic operation unit 450, and the storage unit 460 may include some or all of the above, in which part or all is the same as described above.

Referring to FIG. 1, the communication interface 400 may be involved in downloading an app according to a user's request, and transmit the sensory information extracted according to the operation of the sensory information extractor 440 to an external device. It is used to send and receive various information. In this regard, since it has been described above sufficiently, further description thereof will be omitted.

The user interface unit 410 is not significantly different from the user interface unit in FIG. 3 and will be replaced by the contents thereof. However, in the case of FIG. 2, the user interface unit 410 may input a separate user command for executing the situation recognition unit 420.

The context recognizer 420 may typically include an image capture unit for capturing an image, a voice acquirer for voice recognition, and the like. For example, when a user of the terminal device 200 captures image content displayed on a screen of a movie theater or receives audio provided with the corresponding image, the recognized data is recognized by the controller 430. It may be stored in the storage unit 460 under control.

The control unit 430 may include a communication interface unit 400, a user interface unit 410, a situation recognition unit 420, a control unit 430, a sensory information extractor 440, and a haptic operation unit 450 in the terminal device 200. And controls the overall operation of the storage unit 460. For example, referring to FIG. 2, the storage unit 460 stores the image and audio content captured by the situation recognition unit 420. In addition, the controller 430 may extract the sensory information using the user command related to the state control information set by the user in the user interface 410 and the content stored in the storage 460. To operate. In other words, when the user presets the haptic operation to be performed on the content that is currently stored in the storage unit 460, the captured image may be provided to the sensory information extractor 440. The haptic operator 450 may be controlled using the sensory information extracted from the provided captured image.

In this regard, the controller 430 may include a determination unit for determining the extracted sensory information. In other words, since the sensory information may include parameters such as rhythm, frequency, and waveform, and may include various haptic information indicating levels for respective parameters, the controller 430 operates the haptic operator 450. A process for analyzing sensory information, that is, haptic information, may be performed.

The sensory information extraction unit 440 and the storage unit 460 have been described above with reference to FIG. 3, and thus, further description thereof will be omitted.

The haptic operator 450 may include, for example, a vibration motor. Therefore, under the control of the controller 430, the state of the surrounding situation can be expressed to the user of the terminal device 200 in a sense, that is, a tactile sense. The haptic operator 450 may operate to reflect various haptic information such as rhythm, frequency, and waveform under the control of the controller 430.

5 is a diagram illustrating a sensory information extraction unit of FIGS. 3 and 4.

For convenience of description, referring to FIG. 5 together with FIG. 1, the sensory information extractor 440 according to an embodiment of the present invention may include a content characteristic recognizer 500, a mapper 510, a storage 520, and parameter extraction. It may include some or all of the portion 530.

Here, the inclusion of some or all may be omitted by some components such as the storage unit 520, or may be integrated into other components such as the mapping unit 510, these components may be configured in part or all Means that it may be included in the recording medium to operate in one program form, etc., and all of them are included to aid a sufficient understanding of the present invention.

The content characteristic recognizer 500 may determine the content characteristic by analyzing the input content. Here, the content characteristic refers to recognizing the emotional state expressed in the content by analyzing video, audio, and text messages. For example, if a smiley face of the user is detected in the video, the user may be recognized as happy. If a smiley voice is detected in the audio, the user may be recognized as happy. The specific state or context of the text message may be analyzed to recognize the happy state. Therefore, the content according to an embodiment of the present invention can be seen that the visual emotional state in the video and the auditory emotional state in the audio as intrinsic stimulus. The content characteristic recognizer 500 may be called an emotion estimator in that it estimates a factor for stimulating human emotion.

In addition, the mapping unit 510 performs a process of mapping the data stored in the storage unit 520 based on the emotional state recognized by the content characteristic recognition unit 500, that is, the state information and the state control information designated by the user. Here, mapping may be understood as a process of finding a block in which state information and state control information exist in common. Such a block may contain various haptic information described later. The sensory information extracted through the mapping process may be, for example, various information classified according to characteristics of the terminal device, that is, specifications. For example, low- and high-end terminal devices can be considered to have common state control information and state information. Therefore, the mapping unit 510 may be regarded as finding various sensory information for each specification.

Subsequently, the parameter extractor 530 may extract haptic parameters corresponding to one piece of sensory information, that is, haptic parameters related to intensity, frequency, wavelength, and rhythm, based on the acquired device characteristics of the receiving terminal device 2 (110). .

6 is a view for explaining a service process related to the system of FIG.

Referring to FIG. 6 together with FIG. 1, the terminal device 1 100 according to an exemplary embodiment of the present invention performs a setting for adjusting an emotional state expressed in content (S600). In other words, this setting may be regarded as related to whether the terminal device 2 110 transmits the information to the terminal device 110 or the opposite state as the emotional state expressed in the content.

In addition, the terminal device 1 (100) extracts the sensory information related to the emotional state expressed in the content based on the set state control information (S610). Here, the extraction of sensory information may be understood to include all the processes described with reference to FIG. 5.

Subsequently, the terminal device 1 100 transmits the extracted sensory information to the terminal device 2 110 (S620), and the terminal device 2110 performs a haptic operation based on the received sensory information (S630). For example, the terminal device 110 may analyze the received sensory information and perform a vibration operation as a haptic motion according to the analysis result.

FIG. 7 is a diagram illustrating a specific operation of the terminal device 1 shown in FIG. 6.

As illustrated in FIG. 7, the terminal device 1 100 may execute the service-related Apple by touching an icon displayed on the screen (S700).

Subsequently, the user of the terminal device 1 100 may prepare content in which a visual stimulus, an auditory stimulus, or an audio-visual stimulus is embedded, and determine how to express an emotional state (S710). For example, it may be set to show a haptic response to the opponent as opposed to the emotional state expressed in the content.

After this process, the terminal device 1 (100) performs a process for extracting the sensory information using a tool for extracting the sensory information that is part of the app, or a separate tool (S720). This process may be understood as the process of FIG. 5.

Thereafter, the terminal device 1 (100) performs its own haptic operation using the extracted sensory information, more precisely, haptic parameters included in the sensory information, or transmits it to the receiver terminal 2 (110) of FIG.

8 is a flowchart illustrating a driving process of a terminal device according to an embodiment of the present invention, which may be related to a recording medium.

For convenience of description, referring to FIG. 8 together with FIG. 1, the terminal device 1 100 according to an exemplary embodiment of the present invention has a sense related to an emotional state of a user of a terminal device expressed in content or an atmosphere of an object present in the content. Set the information (S800).

For example, the terminal device 1 100 may set sensory information related to the emotional state of the user expressed in the content, and set state control information for adjusting the emotional state to match the sensory information. In this case, setting means not only matching and storing in the storage unit but also including the same in one program.

Subsequently, when the content is input, the terminal device 1 100 extracts sensory information related to an emotional state or an atmosphere (S810).

In this process, the terminal device 1 (100) may receive a user command for adjusting the emotional state of the user expressed in the content as the state control information, and extracts sensory information related to the content based on the received user command. Can be. In other words, the sensory information may be extracted using the state control information corresponding to the user command and the emotional state expressed in the content. Here, the sensory information is meant to include haptic information.

Furthermore, the terminal device 1 100 may output sensory information to perform a haptic related operation using the extracted sensory information (S820).

9 is a flowchart illustrating a driving process of the terminal device 1 of FIG. 1 or the terminal device of FIG. 2.

For convenience of description, referring to FIG. 9 together with FIG. 1, the terminal device 1 100 according to an embodiment of the present invention stores sensory information related to an emotional state of a user expressed in content or an atmosphere of an object present in the content. (S900).

For example, in the case of FIG. 1, the terminal device 1 100 may store state control information for adjusting an emotional state with sensory information.

Subsequently, when the content is input, the terminal device 1 100 extracts sensory information related to an emotional state or an atmosphere (S910).

In this process, the terminal device 1 100 may receive a user command for adjusting the emotional state of the user expressed in the content as the state control information.

Accordingly, the terminal device 1 100 may extract sensory information related to the content based on the received user command. In other words, the sensory information may be extracted using the state control information corresponding to the user command and the emotional state expressed in the content. Here, the sensory information is meant to include haptic information.

Further, the terminal device 1 100 processes the sensory information to perform the haptic related operation on the extracted sensory information (S920). For example, the terminal device 1 100 may transmit sensory information to the receiving terminal device, that is, the terminal device 2 110 of FIG. 1.

10 is a flowchart illustrating a driving process of the terminal device illustrated in FIG. 2.

Referring to FIG. 10 along with FIG. 2, the terminal device 200 according to an exemplary embodiment of the present invention may recognize a surrounding situation by using, for example, a camera provided in the imaging unit, a microphone provided in the voice recognition unit, or the like. S1000). Here, the surrounding situation refers to the surrounding environment or the peripheral device 210. Therefore, recognition refers to photographing an image displayed on a screen of a movie theater, such as a movie theater, acquiring audio together, and recognizing it.

Prior to this recognition process, if the terminal device 200 needs to input state control information for adjusting the emotional state as a user command as shown in FIG. 9, since the terminal device 200 is related to its own emotional state, the emotional state represented in the content It would be desirable to set it to correspond to.

Subsequently, the terminal device 200 extracts sensory information related to the content of the recognized surrounding situation (S1010). In other words, the terminal device 200 analyzes the content to detect the emotional state or mood of the object included in the image, and extracts the sensory information associated with the emotional state or mood. For example, such sensory information may refer to the haptic information mentioned above, but, for example, when only video content having only visual stimuli is input, corresponding sensory information is extracted, and audio-visual stimuli such as video and audio are extracted. In the case of the content at the same time it may be understood to extract the corresponding sensory information. In practice, the sensory information may have different types of haptic information included therein and specific parameter values may vary.

In addition, the terminal device 200 may perform a haptic operation based on the extracted sensory information (S1020).

11 is a diagram illustrating a driving process of the terminal device 1 of FIG. 1 or the terminal device of FIG. 2 according to another embodiment of the present invention.

For convenience of description, referring to FIG. 11 along with FIG. 1, the terminal device 1 100 according to an embodiment of the present invention generates a set of arousal mapping data (S1110). Here, generating the awakening mapping data set may correspond to storing the aforementioned sensory information and state control information.

Subsequently, the terminal device 1 100 may prepare content and perform an emotional state adjustment process for the content (S1110).

In operation S1120, the terminal device 1 100 may estimate arousal evaluation values related to the emotional state of the content. This process may correspond to the process of determining the emotional state of the content.

Subsequently, the terminal device 1 100 determines whether the state control information corresponds to the content or is opposite (S1130). If it is determined that the result corresponds, the mapping operation corresponding to the emotional state is performed (S1140), and if it is the opposite, the mapping operation of the emotional state opposite to the emotional state is performed (S1150). For example, in the case of contradictory, if the arousal increases in the correspondence, it decreases, and if the awakening decreases in the correspondence, the mapping operation is performed.

When the mapping data is output according to the mapping operation as described above, the device characteristic may be additionally determined (S1160).

As a result of the determination, when the information on the device characteristic is included, mapping data corresponding to the device characteristic, that is, the haptic parameter is output (S1170).

On the other hand, if there is no device characteristic, the extracted corresponding mapping data may be transmitted to the receiving terminal device 2 (110), so that the corresponding device may extract the parameter based on its own device information (S1180).

Continuing, we will look at behavioral experiments of visual haptic interactions in connection with embodiments of the present invention.

We conducted behavioral experiments to investigate the interactions between emotional tactile stimuli. The first step in this process was to identify the emotions that led to the tactile data set. The International Affective Picture System (IAPS) is a very well-cited data set that contains 1182 images with associated significance and arousal ratings. We chose a representative sample of 120 images from the IAPS that included all categories (objects, people, scenes and animals) and emotional evaluations of the original data set. The LAPS assessments reported in the literature were then conducted a pilot to confirm that they could be reproduced with the target participants. The assessments obtained were found to be within the standard deviation of one of the assessments reported in the literature, which participants could then use to validate the dataset.

There are no standardized tactile data sets available. So, we constructed a tactile data set by sampling the tactile design space with dimensions of intensity, frequency, rhythm, and waveform. Conduct experiments were conducted to confirm that the tactile stimuli selected for the embodiment of the present invention could be cognitively distinguishable. Emotional assessments were then collected from participants for 180 stimuli in this data set. This tactile data set and IAPS subset were used in this experiment.

We performed two behavioral experiments to study tactile interactions. The first experiment was performed with 40 pairs of tactile stimuli and two types of conditions were evaluated. Under the corresponding conditions, visual tactile stimuli delivered similar emotional information. In conflicting (or opposing) conditions, tactile stimuli conveyed other emotional information. This is shown in Table 1 below.

Table 1

We used a two-dimensional emotional circulation model to obtain these estimates. Emotional experiences in this model were characterized in two-dimensional spaces, described by significance (how pleasant or unpleasant stimuli) and arousal (how excited or calmed by stimuli).

19 participants participated in this experiment. Participants were represented visually, tactilely, and visually tactilely. They were asked to evaluate these stimuli on a Self-Assessment Manikin (SAM) scale. The SAM grade is a Likert scale designed specifically for obtaining significance and arousal assessments. Indications of stimulation were randomly and counter-balanced across participants.

What we found from these behavioral experiments was that significance assessments were dominated by poetry stimuli, and tactile stimuli tended to increase the arousal of poetry stimuli.

Next, let's look at the tactile interaction for low level (or level) parameters.

The results from previous experiments help us to understand the interaction between tactile stimuli in the conditions of significance and arousal evaluations. However, in order for information about the visual stimulus to be able to produce a given tactile stimulus, we need to understand this interaction under conditions of low level features of the tactile stimulus. So we performed the following experiment.

We selected 108 tactile stimuli from the tactile data set. These stimuli were divided into two groups (HapA and HapB) based on intensity. We also selected three groups of stimuli from the IAPS dataset of 54 images, each of which was VisA, VisB, and VisC. These groups were selected for low, medium and high arousal ratings while the significance rating was neutral. Participants presented visual, tactile and visual tactile stimuli according to the following scheme. This is shown in Table 2. Table 2 shows the visual-haptic coupling.

TABLE 2

24 participants participated in this study. Participants were shown to have visual tactile stimuli only visually, tactilely. They were asked to evaluate these stimuli at the SAM grade. Based on the assessments provided by the participants, a tactile mapping data set was generated. This data set provides a mapping for tactile parameters given the emotional content of the visual stimulus.

An app or authoring tool according to an embodiment of the present invention provides a method for interaction designers to create tactile stimuli, given knowledge of the emotional content of the poetry stimulus. The emotional content of the visual stimulus may be explicitly provided as significance and arousal evaluations or emotion keywords. Alternatively, the emotional content of the visual stimulus may be extracted from the features of the image itself using methods similar to those described in the emotion-based image retrieval literature. Authoring tools based on guidelines generated by behavioral experiments generate a set of parameters for tactile stimulation.

The architecture of the authoring tool was discussed with reference to FIG. 5. The system handles two inputs, a stimulus value and a modulation value that tells the interaction designers that they want to modulate the emotion conveyed by the stimulus. There are two possible values that this input can be treated as <Table 3>. Table 3 shows the adjustment input / output values.

TABLE 3

For example, a significance and arousal estimator such as the content characteristic recognizer 500 of FIG. 5 handles an emotional keyword that carries emotional content of a visual stimulus as an input. 'Keywords' may be low features of visual data such as simple emotional words or facial expressions, color distribution in the scene, contrast, difference values between temporarily adjacent display data, and the like. Using a sentimental model of emotion, the given emotional keyword is converted into corresponding significance and arousal ratings. The mapping model uses a tactile mapping database and guidelines generated using behavioral experiments to generate tactile parameters. If an adjustment is specified as "correspondence", it creates tactile parameters corresponding to visual emotion. If specified as a “collision,” it creates tactile parameters that can increase or decrease the significance or arousal of the visual stimulus. If a correct mapping for a given significance and arousal assessment cannot be used, a module such as mapping section 510 of FIG. 5 infers the closest significance and arousal assessments that can be used to generate the parameters. A parameter estimator such as the parameter extractor 530 of FIG. 5 compares the device characteristics with those of the prototype (or prototype) on which the experiment was performed. The characteristics of the device of interest are the bandwidth and intensity range of the device. It maps the output from the mapping module into the parameter space of the device and outputs the tactile parameters needed to create an emotional stimulus.

Experiments have shown how tactile stimuli can influence the emotional information delivered by visual stimuli. We also proposed how to build an authoring tool to create an emotional tactile stimulus that could correspond to or collide with the visual stimulus. This method has already been described with reference to FIG. 11.

On the other hand, another similar approach is possible for visual input or for input comprising both visual and sound data. The architecture of the tactile authoring tool for visual and sound stimulation is shown in FIG. 7. Therefore, further description will be omitted.

Furthermore, the behavior experiments of audio haptic interactions in relation to the embodiment of the present invention will be described.

We conducted experiments investigating the interaction between emotional hearing and emotional tactile stimuli. Prior to this experiment, the validity of the auditory and tactile data sets was verified as follows. The International Affective Digitized Sounds (IADS) system is a frequently cited data set that includes 167 different environment sounds with associated significance and arousal ratings. We divided these datasets into nine categories (e.g., low significance-low arousal, high significance-neutral arousal), taking samples from each category, taking into account the significance and the degree of arousal, and subtracting 40 emotional auditory stimulus subsets from IADS. selected. We conducted a pilot to ensure that the IADS assessments reported in the literature could be reproduced well in our set of participants. We found that the significance or arousal assessments obtained from the participants did not differ significantly from those reported in the IADS literature, thereby validating the auditory dataset and continuing to use it in subsequent experiments.

We then constructed a tactile data set containing 180 distinct stimuli with four different parameters, rhythm, waveform, frequency and intensity.

We performed two experiments to study audio-haptic interactions with emotional states. In the first experiment, we examined how the corresponding or conflicting emotional information in the auditory and tactile aspects interacts. All the stimuli used in this experiment were obtained from the IADS subset, and an emotional haptic data set was generated. We paired the auditory and tactile stimuli with each other under two conditions, namely those that conflict with the corresponding conditions. In the corresponding conditions, auditory-tactile stimuli conveyed emotional information consistent with each other. In conflicting conditions, auditory and tactile stimuli delivered discordant emotional information. Table 4 shows the auditory-haptic stimulus pair design.

Table 4

Forty auditory-haptic stimulus pairs were given to 19 participants (aural and haptic stimuli appeared simultaneously) and asked participants to assess the significance and arousal of auditory-tactile stimuli on a SAM grade.

The results showed that, where appropriate, the presence of the tactile stimulus affects both the significance and arousal evaluations of the auditory stimulus. High significance auditory stimulation with high significance tactile stimulation was found to decrease significance, in which case the auditory stimulus felt less pleasant, which is counterintuitive. Arousal assessments were increased when low-aural auditory and tactile senses were given simultaneously in pairs, compared to the case where only auditory stimuli were given. In the case of high arousal, the arousal evaluation of auditory-tactile pair stimuli was not statistically different from that of auditory stimuli alone. In the case of conflict, the significance and arousal assessment of auditory-tactile pair stimuli were governed only by the auditory stimulus, except in the case of the paired stimulus of high awake tactile-low awake auditory.

As a second experiment to study auditory-tactile interactions with emotional states, low-level tactile characteristics (tactile stimulus intensity, frequency, rhythm, waveform, etc.) are used to understand the factors that affect emotional assessments. An experiment was conducted to determine how it affects the emotional assessment (similarity and arousal assessment) of emotional auditory-tactile stimuli.

If we can figure out how the low-level characteristic affects the evaluations, we can only adjust the change by selecting the appropriate low-level haptic characteristics. 22 participants participated in this study.

We performed auditory stimuli in an IADS dataset with five groups, Aud-LA, Aud-NN, Aud-HA, Aud-LV, and Aud-HV, and haptic stimuli in an emotional haptic dataset with three groups. selected. We took five pair groups, Groups A, B, C, B and E. Group A included the pairs we obtained by combining Aud-LA stimuli with all three types of haptic group stimuli. Groups B, C, D and E included pairs obtained by combining Aud-NN, Auid-HA, Aud-LV and Aud-HA with each of all three types of haptic group stimuli. All three haptic groups can have two levels, one having low intensity and the other having high intensity. Thus, each group has one auditory stimulus paired with a low intensity haptic stimulus (AHL), and another auditory stimulus paired with a high intensity haptic stimulus (AHH). Each pair group has 36 stimulus pairs. Where Aud is an auditory stimulus, LA is low arousal, NN is neutral significance / wake, HA is high arousal, LV is low significance, and HV is high significance.

Participants were randomly assigned to two groups-group A and group B. Each group included 11 participants each. Participants in group A were all represented by AHL level pairs, while participants in group B were all represented by AHH level pairs. Since both AHL and AHH levels have five pair groups, we represent each pair group in one session. So all participants had to go through five sessions. In each session, 18 stimulus pairs were presented and asked participants to assess how they felt stimuli in the SAM grade.

As a result, we obtained the following results.

The intensity parameter affects both significance and arousal assessments.

High intensity haptic stimuli lead to reduced auditory significance at greater degrees. For example, at 35 ㏈, the significance test decreased more than 20 ,, and the arousal test was vice versa.

As much as frequency influenced auditory significance assessments, the 165 Hz frequency resulted in the largest negative change in significance. High significance hearing was mostly affected by frequency.

Rhythm affected both significance and arousal assessments.

The waveform did not affect significance or arousal evaluations.

12 is an exemplary view for explaining the authoring tool constructed in accordance with the guidelines generated by the auditory tactile experiment.

For example, the authoring tool may provide a method for interaction designers to generate haptic stimuli once emotional auditory (or audio) stimulus content is recognized. Emotional auditory stimulus content may be provided explicitly as significance and arousal assessments or emotion keywords. Emotional content can also be extracted from the characteristics of the stimulus itself. Based on the guidelines and models generated by behavioral experiments, the authoring tool can generate a set of parameters for the haptic stimulus.

The system as in FIG. 12 may input two pieces of information about the auditory stimulus and whether the interaction designer (or user) wishes to adjust the emotions delivered by the stimulus. The two values that can be input will be described later.

The significance and arousal estimator 1200 obtains an emotional keyword that conveys the input emotional auditory content. Using an emotional circular model, the significance and arousal estimator 1200 converts a given emotional keyword into corresponding significance and arousal assessments.

The mapping module 1210 uses a model and guidelines generated using behavioral experiments to generate haptic parameters. If the adjustment entered by the user is specified as Correspond, the mapping module 1220 generates haptic parameters corresponding to auditory emotions. If specified as a conflict, it creates a haptic parameter that can increase or decrease the significance or arousal of the auditory stimulus. If a precise mapping for a given significance and arousal assessment is not possible, then the mapping module 1210 can estimate the closest possible significance and arousal assessments to generate the parameter. For the above operation, the mapping module 1210 may interwork with the storage unit 1220.

The parameter estimator 1230 compares the device characteristics with those of the prototype device performed in the experiment. Characteristics of the device of interest are device bandwidth and intensity range. Mapping the output from the mapping module 1210 into the device parameter space and outputting the haptic parameters needed to generate the emotional stimulus.

For example, if the mapping module 1210 searches the storage unit 1220 and extracts haptic parameters associated with various device characteristics, the parameter estimator 1230 selects and outputs only haptic parameters that match the input device characteristics. Can be.

 12 is not substantially different as compared to FIGS. 5 and 7. In other words, the authoring tool of FIG. 12 may be stored and operated in the sensory information extracting units 320 and 440 of FIGS. 3 and 4. However, there is a difference that the content source is auditory stimulus content. In addition, there is a difference in outputting the stored haptic parameter from the storage unit 1220 according to the guidelines derived by the above experiment.

Therefore, except for this point, the contents are not significantly different from those in FIGS. 5 and 7, and therefore, the contents will be replaced by those contents. In other words, the significance and arousal estimator 1200 of FIG. 12 may be the content characteristic recognizer 500 of FIG. 5, the mapping module 1210 may be a mapping unit 510, and the storage unit 1220. The parameter estimator 1230 may be the storage 520 and the parameter extractor 530 of FIG. 5, respectively.

Based on this, an embodiment of the present invention stores various values according to guidelines, that is, parameter values obtained in an experiment with respect to visual stimulus, auditory stimulus and visual-acoustic stimulus, and then inputs the input stimulus. Depending on the type of content, appropriate parameter values may be generated (or extracted) and output as haptic parameters.

Embodiments of the present invention relate to emotional computing in a human-computer interface (HCI). Embodiments of the present invention have a commercial potential for controlling user emotional states in wearable devices or mobile devices in the future.

On the other hand, even if all the components constituting the embodiment of the present invention is described as being combined or operated in combination, the present invention is not necessarily limited to these embodiments. In other words, within the scope of the present invention, all of the components may be selectively operated in combination with one or more. In addition, although all of the components may be implemented in one independent hardware, each or some of the components of the components are selectively combined to perform some or all functions combined in one or a plurality of hardware. It may be implemented as a computer program having a. Codes and code segments constituting the computer program may be easily inferred by those skilled in the art. Such a computer program may be stored in a computer-readable non-transitory computer readable media and read and executed by a computer, thereby implementing an embodiment of the present invention.

Here, the non-transitory readable recording medium refers to a medium that stores data semi-permanently and can be read by a device, not a medium storing data for a short time such as a register, a cache, or a memory. . Specifically, the above-described programs may be stored and provided in a non-transitory readable recording medium such as a CD, a DVD, a hard disk, a Blu-ray disk, a USB, a memory card, a ROM, or the like.

While the above has been shown and described with respect to preferred embodiments of the invention, the invention is not limited to the specific embodiments described above, it is usually in the art to which the invention belongs without departing from the spirit of the invention claimed in the claims. Various modifications can be made by those skilled in the art, and these modifications should not be individually understood from the technical spirit or the prospect of the present invention.

-

-

Claims (15)

1. In the terminal device,

   A storage unit for storing sensory information related to an emotional state of a user of the terminal device expressed in content or an atmosphere of an object present in the content;

   A sensory information extractor configured to extract sensory information related to the emotional state or the mood when the content is input; And

   A controller configured to perform a haptic related operation using the extracted sensory information;

   Terminal device comprising.
2. The method of claim 1,

   Peripheral situation recognition unit for recognizing the surrounding situation to generate the content; And

   And a haptic operation unit which performs the haptic related operation.

   The control unit, the terminal device, characterized in that for controlling the haptic operation unit based on the sensory information.
3. The method of claim 2,

   The surrounding situation recognition unit,

   And at least one of an image capturing unit for photographing the surrounding situation and a voice obtaining unit obtaining a voice of the surrounding situation.
4. The method of claim 1,

   A user interface unit configured to receive a user command for adjusting an emotional state of a user expressed in the content; And

   And a communication interface for transmitting the sensory information to an external device that performs the haptic related operation.

   The storage unit stores the state control information corresponding to the user command by matching the sensory information,

   The control unit, characterized in that for controlling the communication interface to transmit the sensory information to the external device.
5. The method of claim 4, wherein

   The user interface unit is characterized in that for receiving a user command to express an emotion that matches the emotional state or a user command to express an emotion opposite to the emotional state.
6. The method of claim 1,

   And the content includes at least one of video content, audio content, and text content.
7. The method of claim 1,

   The sensory information extraction unit,

   A content characteristic recognition unit recognizing the emotional state or the mood;

   A mapping unit that performs a mapping operation to find sensory information matching the recognized emotional state or the mood; And

   A parameter extraction unit for extracting parameters of the found sensory information;

   Terminal device comprising a.
8. The method of claim 7, wherein

   The parameter extractor extracts the sensory information by further considering characteristics of an external device that receives the sensory information when the sensory information is extracted.
9. The method of claim 1,

   The sensory information includes haptic information for the haptic related operation,

   The haptic information includes at least one parameter of intensity, frequency, wavelength, and rhythm.
10. The method of claim 9,

    Each said parameter comprising a plurality of levels,

    The haptic information terminal device characterized in that it comprises a parameter corresponding to one of the plurality of levels.
11. In the driving method of the terminal device,

    Storing sensory information related to an emotional state of a user of the terminal device expressed in content or an atmosphere of an object present in the content;

    Extracting sensory information related to the emotional state or the mood when the content is input; And

    Performing a haptic-related operation using the extracted sensory information;

    Method of driving a terminal device comprising.
12. The method of claim 11,

    Recognizing a surrounding situation to generate the content; And

    And performing the haptic related operation.

    The processing may include controlling the haptic related operation based on the sensory information.
13. The method of claim 12,

    Recognizing the surrounding situation,

    And at least one of photographing the surrounding situation and acquiring a voice of the surrounding situation.
14. The method of claim 11,

    Receiving a user command for adjusting an emotional state of a user expressed in the content; And

    And transmitting the sensory information to an external device that performs the haptic related operation.

    The storing may include storing state control information corresponding to the user command with the sensory information,

    The processing may include controlling the sensor device to transmit the sensory information to the external device.
15. The method of claim 14,

    The receiving of the user command may include receiving a user command for expressing an emotion corresponding to the emotional state or a user command for expressing an emotion opposite to the emotional state.

Images