KR20130094383A - Recognition method of location based emotion state and system adopting the method - Google Patents

Abstract

PURPOSE: A location-based emotion recognizing method and a system applied with the same are provided to grasp emotion state visually, by mapping the emotion state on an electronic map. CONSTITUTION: A physiological signal of an experimentee is measured at the arbitrary time interval. The physiological signal moves the location. Coordinate information of the corresponding position where the physiological signal is measured is acquired. Sensibility per location of the experimentee is recognized. Database of the sensibility per location is structured. [Reference numerals] (AA) Collect a physiological signal and location data; (BB) Recognize sensibility; (CC) Apply location based sensibility information

Description

Recognition method of location based emotion state and system adopting the method}

The present invention relates to a location-based emotion recognition method and a system applying the same, and more particularly, to a method and a system for applying a database of emotions of a subject varying for each location.

It is not easy to recognize human emotions through menstrual signals. Emotional structure and development process is still unclear. However, the physiological signal indicating the physiological change of the human body may be used as an index for objectively determining the workload or emotion compared to other emotion recognition methods. Thus, physiological signals can be used for personalization services beyond medical and ergonomics.

Most of the physiological signal measurement studies are performed in a randomly set virtual environment. A study of fatigue evaluation in nine city bus drivers [1] measured and analyzed EMG signals by creating a real road image and a virtual bus operating environment. In addition, in the study on driving method of car [2], a virtual highway driving environment is created for a group of adults and EMG, electrocardiogram, eye movement and skin according to driving using Steering Wheel and Joystick The electrical resistance was measured and analyzed.

In order to measure mobile physiological signals, portable and wearable wireless-based measuring devices are required. The Sensor Stack system [3], developed at the MIT Media Lab, analyzes in real time via a wireless interface whether each dancer shows a consistent movement in a dance ensemble that must exhibit the same movement. The EXMOCARE BT2 [4] is a wrist watch-type biorecognizer that infers people's psychological anxiety and stress from physiological signals and sends them to mobile phones or the Internet.

  Kwon Dae-gyu, Kim Jae-jun, Kim Kyung, Lee Chan-ki, Kim Dong-won (2009). Objective evaluation of the fatigue level through city bus driver's physiological signals. Lectures and Papers at the Korean Society of Mechanical Engineers 2009 Fall Conference, 2917-2918.   Kim Bae-young, Gu Tae-yoon, Bae Chul-ho, Park Jung-hoon, and Signature Won (2010). A Study on HMI Evaluation of Joystick Operation Using Physiological Signal Measuring Technique. Korean Society of Automotive Engineers Papers, 18 (3), 1-7.   R. Aylward, S. D. Lovell, and J. A. Paradiso (2006) A compact, wireless, wearable sensor network for interactive dance ensembles. International Workshop on wearable and implantable Body Sensor Networks, 65 70.   Exmocare. Bt2, exmocare's second generation wireless wearable sensor device. available at http://www.exmovere.com/bt2/.   J. Posner, A. Russell, and B. S. Peterson, "The circumplex model of affect: An integrative approach to affective neuro science, cognitive development and psychopathology," Cambridge University Press, vol. 17, pp. 715-734, 2005.

The present invention provides a location-based emotion recognition method for evaluating the emotion change of the subject changing by region and a system applying the same.

Emotion recognition method according to the present invention:

Measuring the physiological signal of the subject moving position at any time interval;

Obtaining coordinate information of a corresponding position at which the physiological signal of the subject is measured;

Recognizing (evaluating) a subject's positional sensitivity by processing the physiological signal; And

Constructing a database of emotions for each location of the subject; Includes.

Location based emotion recognition system according to the present invention:

A sensor device for detecting a physiological signal and coordinates of the corresponding position at an arbitrary time interval from a subject moving the position;

An emotion recognition device that recognizes a subject's emotion using the physiological signal; And

A system for establishing a database of emotions of the subject; .

One embodiment of the method according to the present invention further includes a visualization step of displaying the subject's positional sensitivity on the corresponding coordinates of the electronic map, the system comprises the subject's positional sensitivity on the corresponding coordinates of the electronic map It further includes a visualization device for displaying.

According to another embodiment of the present invention, the physiological signal may be detected from a plurality of subjects.

According to another embodiment of the present invention,

The physiological signal comprises a PPG signal,

Recognizing the emotion may use a PPG signal, it is possible to recognize the emotion by setting a reference value to the PPG average value of a certain period obtained from the subject, and comparing the subject's real-time PPG value with respect to the reference value.

According to another embodiment of the present invention,

Emotion may be recognized by setting a reference value to a PPG mean value of a certain period obtained from the subject and comparing the subject's real-time PPG value with respect to the reference value.

Various types or forms of application are possible by databaseting the subject's location-based emotion according to the present invention. For example, by mapping the emotional state on the electronic map, it is possible to visually identify the emotional state of a subject or a group of subjects that are differentiated by region, and effectively construct an alternative corresponding thereto.

1 is a flowchart illustrating a process of a location-based emotion evaluation method according to the present invention.

Hereinafter, with reference to the accompanying drawings will be described in detail the embodiments of the location-based emotion evaluation method and a system applying the same.

1 is a flowchart illustrating a process of a location-based emotion evaluation method according to the present invention. Location-based emotion recognition method according to an embodiment of the present invention comprises the steps of collecting the physiological signal and location information of the subject on the move at any time interval; Recognizing (evaluating) the emotional state of the subject by analyzing the collected physiological signals; And applying the evaluated emotional state by location.

In the application phase of data on the emotional state, an example of the application, "visualization" may use a server system that collects the subject's emotional state and location information, the server system expresses the subject's emotional state by location on the electronic map According to the display of the emotional state on the electronic map, the trend of the emotional state according to the change of the position of the subject can be expressed on the electronic map, while the emotional state is obtained from the physiological signals obtained from the subject. It is then documented (databaseed) and can be used for other forms of "application." The database contains information about the sensitivity and location of one or more subjects, and provides data for the application (or use) of the subject's location change data.

As shown in FIG. 1, in a first step, collection of a physiological signal and location data by a terminal device is performed. The collection of the physiological signal is performed on one or more subjects, and the position data of the subject, that is, the latitude and longitude information, is also collected. The position data obtained from the subject is used as coordinates for displaying the subject's emotional state by position on the electronic map, for example. In the second step, physiological signals are analyzed to assess (recognize) the subject's emotional state. The evaluation of this emotional state is performed by conventional methods. For example, in emotional evaluation, emotional recognition is carried out by applying Russell's prototype model [5] to arouse, relaxation, pleasure, displeasure, arousal-pleat, arousal-pleasant, relaxation-pleat, relaxation-displeasure, neutral gun 9 emotions can be recognized. However, the following embodiment of the present invention recognizes and applies awakening and relaxation. In the third step, the analyzed database is then applied for appropriate use. Application of location-based emotional data includes visualization applications and non-visualization applications. Visualization applications include "visualizations" that indicate the emotional state of a subject for each location on an electronic map, and in addition to this visualization, a non-visualization application using emotional state data may be performed. Analysis of regional emotional distribution of a group consisting of a large number of subjects, and based on this, may include preparing regional countermeasures.

The terminal device used in the collecting step includes a sensing device for detecting a physiological signal from the subject and obtaining the position information of the subject. The terminal device also includes an emotion recognition device for analyzing a physiological signal. The sensing device may include a sensor for detecting a physiological signal from a subject, for example, a photoplethysmograph (PPG) sensor, a galvanic skin response (GSR) sensor, or a skin temperature (SKT) sensor. have. It may also include a Global Positioning System (GPS) receiver for obtaining location information.

As a physiological signal used for evaluation of the emotional state, PPG, GSR, SKT signals may be used. The physiological signal and location information may be detected using a wrist watch type sensing device as shown in FIG. 2. In some cases, the GPS receiver for detecting the location information may be provided separately from the sensing device for detecting the physiological signal, or may be embedded in the emotion recognition device. The sensing device may include a wired or wireless communication interface for transmitting a physiological signal and location information to the emotion recognition device. The emotion recognition device preferably has a form of a portable terminal carried by a subject, and may transmit the analyzed result to a server system for application such as visualization that can be accessed through an internet network. The wireless communication interface may be implemented by various methods, for example, may include a communication module such as Bluetooth. The wireless communication interface transmits a physiological signal to the emotion recognition device at random time intervals, periodically or aperiodically.

3 illustrates a schematic structure of an emotion evaluation system through the Internet network 2. As shown in FIG. 3, a plurality of emotion recognition devices 3 are connected to the server 1 via the Internet network 2 as described above. Each emotion recognition device 3 is provided to a plurality of subjects together with the sensing device 3a. The sensing device 3a detects the physiological signal and the location information at arbitrary time intervals periodically or aperiodically and transmits the physiological signal and the longitude and latitude through a wired or wireless communication interface, for example, Bluetooth, to the emotion recognition device. Transmit information (physiological signal: 4 bits / 4 ms, longitude information: every 1 second). The emotion recognizing apparatus evaluates the emotional state by using the physiological signal from the subject from the sensing device, and transmits the result of the position information of the subject to the server system 1.

In one embodiment of the present invention, PPG (frequency, amplitude) signal of PPG, GSR, SKT is selected and used for emotion recognition by the emotion recognition device. Emotional analysis was performed by determining the increase and decrease in the reference value (Vref) to be processed in real time. The reference value was set to an average value of the initial 30 seconds (7500 pieces, about 7.3 KB). The received physiological signal value (Vcur) was then compared with the reference value (Vref) (2500 seconds 10 seconds, about 2.4KB). Emotion was recognized according to the difference value (difference = (PPG frequency, PPG amplitude) _criterion- (PPG frequency, PPG amplitude) _present ). Recognition as awakening is when the current frequency and amplitude are increased or only the frequency is increased. Relaxation, on the other hand, is when the frequency decreases and the amplitude increases or only the frequency decreases. The emotion analysis result obtained through this process (emotion recognition value, difference value, location information) was transmitted to the server system 3 at an arbitrary time interval, preferably periodically.

The emotion analysis result of the subject or group of subjects collected in the server system (emotional visualization server 3) may be applied in various forms. One example of the application is to display the emotional state for each subject's location on the electronic map. That is, by visualizing a change in the emotional state of the moving subject, for example, as shown in FIG. 5, the emotional change and distribution in a specific region may be visualized. FIG. 5 illustrates the change in the physiological signal (sensitivity state) measured during the movement from Bundang to Gwanghwamun (Gyeonghwa-do, Gyeonggi-do) on a Google Earth browser (screen), which is a kind of electronic map. According to the change, ie emotion, high and low on the map. Here, the browser refers to all terminals or screens of terminals connected to the emotion visualization server through the internet network. To this end, physiological signals and location data were measured through a wristwatch-type measuring device and a GPS receiver as shown in FIG. Delivered. Based on this, the server system generates a keyhole markup language (KML) file in real time, and the Google Earth browser periodically or aperiodically, preferably, KML having information (refreshInterval object) that periodically updates a script module in the server system. Called periodically to confirm the perceived sensitivity from the subject's physiological signals.

The present invention shows the possibility of sharing the emotions of subjects that change when moving, and shows that the present invention can be applied in various forms or forms such as tools and media art of personalized emotional content service.

Hereinafter, the application of visualization using the method and system of the present invention as described above will be described in more detail. Table 1 below shows the specifications of the emotional visualization server system.

division Specifications hardware Unbuntu 11.4 software Google Earth Version 6.1 Standard technology Use KML 2.2 and OGC KML 2.2 standard extensions based on XML Physiological signal PPG (frequency, amplitude) Emotion Recognition Range “Awakening”, “relaxation”

The emotion visualization server corresponding to the hardware receives a PPG value (including a reference value), coordinate values, and visual information from the emotion recognition device. When a CGI program is called from the Earth browser, it generates KML.

The Earth browser loads the “SEED KML” file, repeatedly calls the CGI program of the server at a specific time interval, and outputs the returned KML on the map on the Google Map. The Earth browser is connected to a PC connected to the server. Same terminal device.

6 is a sequence diagram for a visualization application.

Referring to FIG. 6, the emotion recognition apparatus transmits a PPG reference value to a server at system initialization, and then transmits PPG (frequency, amplitude), longitude, and time information in an infinite loop until emotion recognition ends. To store these data.

The emotional visualization server also generates a visualization KML file using the data within the infinite loop of the emotional visualization service period, has a CGI program call in the Earth browser, and returns the visualization KML file to the Earth browser. The Earth browser loads “SEED KML”, a program file for requesting a visualization KML file by calling a CGI program of the emotional visualization server as a KML format file, and at a time interval arbitrarily defined in SEED KML. Call the CGI program The emotional visualization server returns the visualization KML file as described above in response to the CGI call, and the Google browser outputs the visualization KML on the Google map.

Here, look at the data as follows.

division Explanation PPG threshold 1) Reference value for determining high and low physiological signal in visualization
Set the periodic average value of the period signal as the reference value (for example, average value of PPG value accumulated for 30 seconds)
2) Receive from sensing device (Bluetooth)
3) Processed by Emotion Recognition Device and sent to Emotion Visualization Server PPG value 1) Value for recognizing emotion (wake, relaxation)
2) Emotion recognition as 'Awakening' when it is higher than PPG standard value and 'Relaxing' when low
3) Receive with sensing device (Bluetooth)
4) Processed by Emotion Recognition Device and sent to Emotion Visualization Server Position value, time 1) Latitude, longitude data, time
2) Used as emotional position value and emotional expression time of Earth browser
3) Receive from GPS receiver (Bluetooth)
4) Processed by Emotion Recognition Device and sent to Emotion Visualization Server

The following example illustrates the code for a CGI program for creating a visualization KML file.

CGI program code #! / usr / bin / python
beforeLatitude = previous value of 'latitude value' currently received
beforeLongitude = previous value of 'longitude value' currently received
presentLatitude = 'Latitude value' currently received
presentLongitude = 'longitude value currently received'
presentTime = 'time value' currently received
referencePPG = initially received 'PPG reference value'
beforePPG = previous value of 'PPG value' currently received
presentPPG = 'PPG value' currently received

gxTrack_when = Array of 'time values' saved by previous call
gxTrack_coord = 'latitude / longitude value' array saved by previous call
visual_coordinates = Array of visualization coordinates saved by previous call
when = ''''''<when>% s </ when>''''''''
coord = ''''''<gx:coord>% d% d 40.1999 </ gx: coord>''''''
coordinates = '''''''<Placemark>
<styleUrl># multiTrack_n </ styleUrl>
<TimeStamp>
<when>% s </ when>
</ TimeStamp>
<LineString>
<extrude> 1 </ extrude>
<gx: altitudeMode> relativeToSeaFloor </ gx: altitudeMode>
<coordinates>
% d,% d,% d
% d,% d,% d
</ coordinates>
</ LineString>
</ Placemark>''''''
gxTrack_when + = [when% (presentTime)]
gxTrack_coord + = [coord% (presentLongitude, presentLatitude)]
visual_coordinates + = [coordinates% (presentTime, beforeLongitude, beforeLatitude, beforePPG, presentLongitude, presentLatitude, presentPPG)]
visual_kml = (
'<? xml version = "1.0" encoding = "UTF-8"?> \ n'
'<kml xmlns = "http://www.opengis.net/kml/2.2" xmlns: gx = "http://www.google.com/kml/ext/2.2"> \ n "
'<Document> \ n'
'<Style id = "multiTrack_n"> \ n'
'<IconStyle> \ n'
'<Icon> \ n'
'<href> icon URL </ href> \ n'
'</ Icon> \ n'
'</ IconStyle> \ n'
'<LineStyle> \ n'
'<color> 7f0000ff </ color> \ n'
'<width> 6 </ width> \ n'
'</ LineStyle> \ n'
'</ Style> \ n'
'<Folder> \ n'
'<name> Tracks </ name> \ n'
'<Placemark> \ n'
'<name> me </ name> \ n'
'<styleUrl>#multiTrack</styleUrl> \ n'
'<gx: Track> \ n'
'% s \ n'
'% s \ n'
'</ gx: Track> \ n'
'</ Placemark> \ n'
'% s \ n'
'</ Folder> \ n'
'</ Document> \ n'
'</ kml>
)% (gxTrack_when, gxTrack_coord, visual_coordinates)
print 'Content-Type: application / vnd.google-earth.kml + xml \ n'
print visual_kml

The following is an example of a visualization KML file.

Visualization KML <? xml version = "1.0" encoding = "UTF-8"?>
<kml xmlns = "http://www.opengis.net/kml/2.2" xmlns: gx = "http://www.google.com/kml/ext/2.2">
<Document>
...
<Style id = "multiTrack_n">
<LineStyle>
<color> 7f0000ff </ color>
<width> 6 </ width>
<gx: labelVisibility> 1 </ gx: labelVisibility>
</ LineStyle>
</ Style>
<Folder>
<name> Tracks </ name>
<Placemark>
<name> me </ name>
<styleUrl>#multiTrack</styleUrl>
<gx: Track>
<when> 2011-10-10T08: 35: 09Z </ when>
<when> 2011-10-10T08: 35: 19Z </ when>
<when> 2011-10-10T08: 35: 29Z </ when>
<when> 2011-10-10T08: 35: 39Z </ when>
<when> 2011-10-10T08: 35: 49Z </ when>
<when> 2011-10-10T08: 35: 59Z </ when>
...
<gx: coord> 127.124380 37.384058 40.1999 </ gx: coord>
<gx: coord> 127.126815 37.385878 40.1999 </ gx: coord>
<gx: coord> 127.126611 37.386475 40.1999 </ gx: coord>
<gx: coord> 127.122502 37.390175 40.1999 </ gx: coord>
<gx: coord> 127.119219 37.392050 40.1999 </ gx: coord>
<gx: coord> 127.110400 37.391794 40.1999 </ gx: coord>
...
</ gx: Track>
</ Placemark>
<Placemark>
<name> me </ name>
<styleUrl>#multiTrack</styleUrl>
<TimeStamp>
<when> 2011-10-10T08: 35: 30Z </ when>
</ TimeStamp>
<LineString>
<extrude> 1 </ extrude>
<gx: altitudeMode> relativeToSeaFloor </ gx: altitudeMode>
<coordinates>
127.124380,37.384058,200
127.126815,37.385878,300
</ coordinates>
</ LineString>
</ Placemark>
<Placemark>
<name> me </ name>
<styleUrl>#multiTrack</styleUrl>
<TimeStamp>
<when> 2011-10-10T08: 36: 44Z </ when>
</ TimeStamp>
<LineString>
<extrude> 1 </ extrude>
<gx: altitudeMode> relativeToSeaFloor </ gx: altitudeMode>
<coordinates>
127.126815,37.385878,300
127.126611,37.386475,500
</ coordinates>
</ LineString>
</ Placemark>
...
</ Folder>
</ Document>
</ kml>

Above, we have described a visualization application that represents a subject's emotional state on an electronic map, for example Google Earth.

The following are examples of virtual use of the method and apparatus using the present invention as described above; That is, the scenario will be described as an example.

Application Scenario I 1. Emergency Control Center Operator A checks the physiological signs of the patient on the move and controls the patient in danger.
2. Patient B is on the move wearing a physiological signal measuring device and a recognition device.
3. B is on the bus. For some time, the bus driver's violent driving and bad road conditions have made him constantly feeling nervous.
4. A checks the state of B on the map. B has been awakening for some time. If the awakening state persists, B is in danger. After some time, there is no change in B's condition, so A recognizes it as a danger and identifies B's current position on the map.
5. A sends an emergency wait signal to the first aid center closest to B's position.
6. Patients in the above scenarios are limited to those who are unable to communicate their situation or who are unable to communicate or who cannot judge their situation due to mental disorders (eg speech impaired, polio, physically handicapped, elderly).
7. Benefits: The patient can be aware of the dangerous situation and give first aid or control before the patient becomes dangerous. Application scenario II 1. Used to improve urban and road environment
2. Department A's Environment Division Department A collected physiological signal data and locations for N citizens to improve Seoul's environment.
3. A printed out the N data on the map. On the map, Seoul's emotions are visualized (N) by season, time zone, and region.
4. In the areas where the 'wake up' value is high, the environment improvement policy was implemented.
5. On the roads with high awakening values, there is a high likelihood of traffic weight.
6. In late evening, when I examined alleyways where the 'wake up' value was high, it was found that there was a shortage of street lamps. In response, the government increased the budget for installing street lights.

1: emotion visualization server 2: Internet network 3: emotion recognition device

Claims (15)

Measuring the physiological signal of the subject moving position at any time interval;
Obtaining coordinate information of a corresponding position at which the physiological signal of the subject is measured;
Recognizing (evaluating) a subject's positional sensitivity by processing the physiological signal; And
Constructing a database of emotions for each location of the subject; Emotion recognition method comprising a. The method of claim 1,
A visualization step of displaying the subject-specific emotion on the electronic map; Emotion recognition method further comprising. The method of claim 1,
Wherein the physiological signal is detected from a plurality of subjects. The method of claim 1,
The physiological signal comprises a PPG signal,
Recognizing the emotion is emotion recognition method, characterized in that using the PPG signal. 5. The method of claim 4,
Emotion recognition method, characterized in that the reference value is set to the average value of the PPG of a certain period obtained from the subject, and the emotion is compared by comparing the subject's real-time PPG value with respect to the reference value. 3. The method of claim 2,
Emotion recognition method, characterized in that the reference value is set to the average value of the PPG of a certain period obtained from the subject, and the emotion is compared by comparing the subject's real-time PPG value with respect to the reference value. The method according to claim 5 or 6,
Wherein the physiological signal is detected from a plurality of subjects. The method according to claim 5 or 6,
And a visualization step of displaying the subject-specific emotion on the electronic map. Sensing device for detecting the physiological signal and the coordinates of the position at any time interval from the subject moving the position;
An emotion recognition device that recognizes a subject's emotion using the physiological signal; And
A server system for constructing a database on the emotion of the subject; Emotion recognition system having a. The method of claim 9,
Emotion recognition system, characterized in that to visualize the subject-specific emotion on the electronic map. 11. The method of claim 10,
A browser for displaying an electronic map by connecting to the server system,
The server system generates a visualization file for the visualization,
And the browser displays the emotions for each location of the subject on the electronic map using the visualization file. The method according to any one of claims 9 to 11,
Emotion recognition system comprising a plurality of the sensing device and the emotion recognition device provided to a plurality of subjects. 13. The method of claim 12,
The emotion recognizing apparatus sets a reference value to a PPG average value of a predetermined period obtained from the subject, and compares the subject's real-time PPG value with respect to the reference value to perform emotion recognition. The method according to any one of claims 9 to 11,
The emotion recognizing apparatus sets a reference value to a PPG average value of a predetermined period obtained from the subject, and compares the subject's real-time PPG value with respect to the reference value to perform emotion recognition. 15. The method of claim 14,
Emotion recognition system comprising a plurality of the sensing device and the emotion recognition device provided to a plurality of subjects.

Images