KR20150137307A - The method and system for providing user optimized information, recording medium for performing the method - Google Patents

Abstract

Disclosed are a method and a system for providing user customized information, and a recording medium for performing the same. The method for providing the user customized information from a device for augmenting user cognition which communicates with a server, comprises the following steps: recognizing emotional state of a user by sensing bio-signals of the user; generating external environment information by sensing the external environment of the user; transmitting the emotional state and the external environment information of the user to a server; receiving user customized information searched based on the emotional state and the external environment information of the user from a user customized information database of the server, and outputting the same to the user; and generating feedback information on the outputted user customized information, and transmitting the same to the server to update the user customized information database.

Description

TECHNICAL FIELD [0001] The present invention relates to a method and system for providing user-customized information, a recording medium for performing the method, and a recording medium for performing the method.

The present invention relates to a method and system for providing user-customized information and a recording medium for performing the same, and more particularly, to a method and system for providing user-customized information for providing a user's perception enhancement service to a user, The present invention relates to a recording medium for carrying out the present invention.

With the recent development of IT technology, a lot of eyeglass products are being released for user's convenience. However, most of the products provide a one-way unidirectional service to the user, and adaptive services considering the user environment or the user's perception are not provided.

Also, even if an adaptive service considering the user environment is provided, the intelligent service product providing such a service has a high product unit price.

Therefore, there is a need for a new conceptual interface device that can use the existing eyeglass type display platform as an electronic device capable of user adaptive service.

KR10-2013-0130740A

According to an embodiment of the present invention, a user emotion state and external environment information are transmitted to a server, and user-customized information corresponding to a user's emotional state and external environment information is received and output from a server and feedback information on user- And provides a method for providing user-customized information to be transmitted.

According to another aspect of the present invention, there is provided a method for generating feedback information by transmitting user's emotional state and external environment information based on a user's bio-signal and an external environment and outputting user-customized information corresponding thereto, And a server for searching the information database for user-customized information according to the user's situation and updating the user-customized information database based on the feedback information.

According to an aspect of the present invention, there is provided a method of providing user-customized information in a user's augmentation apparatus communicating with a server, the method comprising: recognizing a user's emotional state by sensing a user's bio- A user emotion state and external environment information are transmitted to the server, and the user emotion state and the external environment information are retrieved from the user-customized information database of the server, And outputs the generated feedback information to the user, and generates feedback information on the output user-customized information to update the user-customized information database, and transmits the generated feedback information to the server.

On the other hand, the user-customized information database is managed by an Incremental Principal Component Analysis (IPCA), an Incremental Linear Discriminant Analysis (ILDA), an Incremental Non-Negative Matrix Factorization (INMF), an Incremental Support Vector Machine (ISVM) And can be progressively learned using the feedback information.

The feedback information may be generated based on at least one of the user emotion state and the external environment information as a result that the user-customized information is output to the user.

The recognition of the user's emotional state may include extracting at least one emotional feature variable from the bio-signal, performing a performance evaluation on the extracted emotional feature variable, and storing the emotional feature variables having an evaluation performance of a threshold value or more By individually reducing the dimension of the emotional characteristic variables to the region where the emotional characteristic parameter is set, and recognizing the emotional state of the user with the optimized emotional characteristic variables.

In addition, the biological signal may include at least one of the user's pulse wave (PPG), skin conductivity (GSR), and brain wave signal.

The generating of the external environment information may include detecting the user's gaze path to the external environment and detecting the user's interested object in the external environment.

The detecting of the user's interest object may include detecting a user's gaze path to the external environment, detecting the user's gaze area for the external environment using the detected gaze path, And detecting the object of interest.

The detection of the region of interest may include extracting the line feature information from the detected line of sight of the user and detecting the region of interest of the user with the line feature information.

The gaze feature information may be at least one of a time of staying in a sight line in the same area and a number of times a sight line in the same area stays.

In addition, the user-customized information database may include a similar tendency user cluster in which the plurality of users are clustered according to the measured similarity after measuring the similarity among the plurality of users.

In addition, the user-customized information database may be periodically updated based on user-customized information databases of a group of similar-like users retrieved from the similarity-oriented user cluster.

In addition, the external environment information may include at least one of an actual external environment image viewed by the user, a sound of an actual external environment heard by the user, a location and a temperature of the user.

Also, outputting the user-customized information to the user may output the user-customized information in the form of at least one of visual image, sound, and vibration based on the external environment information.

In addition, when the external environment information matches the specific external environment, it may further include preferentially outputting a warning alarm.

In addition, it may be a computer-readable recording medium on which a computer program is recorded, for providing user-customized information.

According to another aspect of the present invention, there is provided a user-customized information providing system for sensing a user's emotional state by sensing a user's biological signal, sensing external environment of the user to generate external environment information, Wherein the user-customized information database outputs the user-customized information corresponding to the information to the user, generates feedback information on the user-customized information as a result of the output, And a server for searching the user-customized information corresponding to the environment information, transmitting the user-customized information to the user awareness enhancing device, and updating the learned user-customized information database based on the feedback information.

On the other hand, the server can acquire the user-customized information by an Incremental Principal Component Analysis (IPCA), an Incremental Linear Discriminant Analysis (ILDA), an Incremental Non-Negative Matrix Factorization (INMF), an Incremental Support Vector Machine (ISVM) The feedback information can be progressively learned in the database.

In addition, a plurality of user awareness enhancing apparatuses may be provided, and the server may generate a similar tendency user cluster in which users of a plurality of user awareness enhancing apparatuses are clustered based on a predetermined criterion.

In addition, the server may calculate similarities between the plurality of users, and may cluster the plurality of users whose calculated similarity is equal to or greater than a set value to create a similar-tendency user cluster.

Also, the server confirms whether the user of the user awareness enhancing apparatus requesting the user-customized information is a new user who is not registered in the similarity user cluster, and if the new user is an existing user registered in the similar- It is possible to calculate the similarity and update the new user to the cluster of the existing user having a degree of similarity higher than the set value.

In addition, the server may periodically update the user-customized information database by reflecting the user-customized information databases of the similarity-oriented users group retrieved from the similarity-oriented user cluster.

According to one aspect of the present invention, it is possible to provide important information that the user is not aware of, or user-customized information corresponding to the user's situation, using the existing eyeglass platform, The intelligent cognitive enhancing service capable of real-time growth can be provided by progressively learning the user-customized information database through the user.

In addition, it is possible to provide a user-customized information providing system using collective intelligence by reflecting the provision of user-customized information according to the change of the user situation of the similar user group.

1 is a diagram illustrating a user-customized information providing system according to an embodiment of the present invention.
Fig. 2 is a flowchart showing the user-customized information providing system shown in Fig. 1. Fig.
FIG. 3 is a control block diagram of the UE or the UE shown in FIG. 1. FIG.
FIG. 4 is a diagram illustrating an embodiment of the UE or the UE shown in FIG. 3. Referring to FIG.
5 is a flowchart illustrating a method of providing user-customized information in the UE or UE shown in FIG.
FIG. 6 is a control block diagram of the server shown in FIG. 1. FIG.
7 is a diagram illustrating a user-customized information providing system according to another embodiment of the present invention.
8 is a control block diagram of the server shown in FIG.
FIG. 9 is a flowchart showing a method of providing user-customized information in the server shown in FIG.

The following detailed description of the invention refers to the accompanying drawings, which illustrate, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that the various embodiments of the present invention are different, but need not be mutually exclusive. For example, certain features, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in connection with an embodiment. It is also to be understood that the position or arrangement of the individual components within each disclosed embodiment may be varied without departing from the spirit and scope of the invention. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is to be limited only by the appended claims, along with the full scope of equivalents to which such claims are entitled, if properly explained. In the drawings, like reference numerals refer to the same or similar functions throughout the several views.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

1 is a diagram illustrating a user-customized information providing system according to an embodiment of the present invention.

The user-customized information providing system 1000 according to the present embodiment includes the user awareness enhancing apparatus 100 and the server 200. [

The user awareness enhancing apparatus 100 is a device that communicates with the server 200 through a communication network to receive a user adaptive service and provides the user with the adaptive service. In this case, the communication network is a high speed backbone network of a communication network capable of a large-capacity, long-distance voice / data service, for example, a general Internet network. In addition, the communication network is a next generation wired network for providing high-speed data services based on ALL-IP (ALL Internet Protocol), and mediates signals and data transmission / reception between the systems.

In particular, the user awareness enhancing apparatus 100 of the present embodiment generates current user context information including a user's emotional state and external environment information by sensing the user's bio-signal and the external environment, and transmits the generated current user context information to the server 200. In addition, the server 200 receives the user-customized information corresponding to the current user context information and outputs the user-customized information to the user and generates feedback information based on the change in the user context information resulting from the output.

The server 200 may be a general type of server that provides predetermined information in response to a client connection request and communicates predetermined information with the user awareness enhancing apparatus 100. [

In particular, the server 200 of the present embodiment stores the learned user-customized information database in advance, searches user-customized information corresponding to the current user context information received from the user awareness-enhancing apparatus 100, 100).

In addition, the user-customized information database is updated by reflecting the received feedback information on the user-customized information that has been transmitted, thereby growing a user-customized information database in real time.

Fig. 2 is a flowchart showing the user-customized information providing system shown in Fig. 1. Fig.

Referring to FIG. 2, the user's augmentation apparatus senses a user's bio-signal and an external environment to generate user's current situation information and transmits the generated information to the server. At this time, the user current state information may include the user's emotion state and external environment information.

The server retrieves user-customized information corresponding to the user's current situation information from the learned user-customized information database, and transmits the retrieved user-customized information to the user's augmenting device.

The user's augmentation apparatus outputs the received user-customized information to the user, re-senses the user's biometric signal and the external environment as a result of the output, generates feedback information based on the sensed user's biometric signal and the external environment, and transmits the feedback information to the server.

The server updates the user-customized information database by reflecting the feedback information to perform progressive learning of the database.

FIG. 3 is a control block diagram of the UE or the UE shown in FIG. 1. FIG.

3, the user awareness enhancing apparatus 100 of the present embodiment includes a communication unit 110, a sensing unit 120, an input unit 130, a memory unit 140, an output unit 150, and a control unit 160. [ . Figure 3 illustrates a user perceptual enhancement device having various components. However, not all illustrated components are required. A user perception enhancing device may be implemented by more components than the illustrated components, and a user perception enhancing device may be implemented by fewer components. Hereinafter, the above-mentioned components will be described in detail.

The communication unit 110 may include one or more components for performing wireless communication with the server.

For example, the communication unit 110 may be configured as a wireless Internet module.

The wireless Internet module is a module for wireless Internet access. The wireless Internet module can be built in or externally. Wireless Internet module Wi-fi (wireless-fidelity) , LTE (Long Term Evolution), 3GPP2 (3 rd Generation Partnership Project 2), 3GPP (3 rd Generation Partnership Project), WiMAX (Worldwide Interoperability for Microwave Access), IEEE ( Institute of Electronics Engineers) < RTI ID = 0.0 > 802.16m. ≪ / RTI >

The communication unit 110 may transmit the user's emotional state and external environment information to the server, and receive user-customized information in response to the user's emotional state and external environment information. In addition, the communication unit 110 may transmit the generated feedback information to the server side upon reception of the user-customized information.

The sensing unit 120 includes at least one or more components for sensing a user's bio-signal and an external environment. The sensing unit 120 includes a biological signal sensing unit 121 and an external environment sensing unit 122.

The biological signal detection unit 121 includes a sensor unit 121-1 for measuring the user's pulse wave PPG, skin conductivity GSR and body temperature and a preprocessor 121-2 connected to the sensor unit 121-1. ).

The sensor unit 121-1 is composed of one or more sensors and measures biological signals such as a user's pulse wave, skin conductivity, and body temperature. The sensor unit 121-1 uses a sensor of a commercial product and can be attached to a predetermined helmet such as a body temperature and a pulse sensor in FIG. 4 and attached to a palm or a finger.

The preprocessing unit 121-2 amplifies the bio-signal measured by the sensor unit 121-1 and performs filtering for noise elimination to extract only the pure bio-signal and transmits it to the controller 160. [

In addition, the bio-signal detection unit 121 includes an EEG measuring unit 121-3 for measuring a user's EEG. The EEG measuring unit 121-3 may be implemented as a helmet-like structure as shown in FIG. 4, and may be attached to the head of a user. In FIG. 4, the brain wave measuring unit is implemented in the form of a helmet or a cap, but it is also possible to realize the band-shaped measuring unit.

The EEG measuring unit 121-3 includes an electrode unit (not shown), a signal processor (not shown), and a transmitter (not shown).

The electrode unit includes at least one electrode, and each electrode detects an EEG output from the user. The electrode of the electrode part may be composed of either a wet electrode to be attached to the body using a conductive gel or an adhesive tape when used as an Ag-AgCl electrode, and a dry electrode not requiring an additional conductive gel.

The electrode portion is preferably capable of detecting a signal having an amplitude of a voltage range of 10 to 200 mV in the frequency range of the EEG, for example, about 0 to 60 Hz. Meanwhile, the EEG may include an amplitude of an alpha wave and a beta wave, and a relative ratio of an alpha wave to a beta wave.

The signal processing unit performs various analog signal processing processes such as computation, amplification, and filtering of the brain waves detected by the electrode unit having characteristics of low power, high noise ratio, and fine signal. The signal processing unit performs an analog-to-digital converting (ADC) on the brain waves input from the electrode unit, converts the analog brain waves into a digital type signal, and performs signal processing for computation, amplification, and filtering.

4, the EEG transmitting unit transmits EEG signals processed by the signal processing unit to the controller 160. When the EEG measurement unit 135 is implemented separately from the controller 160, the transmitter transmits the EEG including the duplexer and the antenna . ≪ / RTI > Such a transmitter can perform digital to analog conversion (DAC) on a signal-processed digital form of EEG to convert it into an analog signal. The transmitter can perform the intermediate frequency operation or the RF frequency operation on the analog type signal, and accordingly, the EEG can be transmitted using an appropriate carrier wave.

The external environment sensing unit 122 includes one or more components for sensing the external environment of the user.

The external environment sensing unit 122 includes an image sensing unit 122-1, a microphone 122-2, a position sensing unit 122-3, and a temperature / humidity measurement unit 122-4.

The image pickup section 122-1 picks up an image of the actual external environment viewed by the user and the pupil of the user (specifically, the eye region). For example, in the case where the eyeglass type interface device is applied to a user or an augmentation device as shown in Fig. 4, the imaging section 122-1 includes an external camera for photographing an actual external environment image, an internal camera . ≪ / RTI >

The microphone 122-2 senses the sound of the actual external environment that the user hears. For example, the microphone may be disposed on the front face of the spectacles-type display if the eye-like interface device is applied to the user-intensifier as in Fig.

The position sensing unit 122-3 is a module for confirming or acquiring the position of the user using the user awareness enhancing apparatus 100. [

The position sensing unit 122-3 may be implemented as a GPS module, for example. The GPS module receives position information from a plurality of satellites. The location information may include coordinate information indicated by latitude and longitude. For example, the GPS module can accurately calculate the current position according to the triangulation method by measuring the precise time and distance from three or more satellites and measuring three different distances. A method of obtaining distance and time information from three satellites and correcting the error from one satellite may be used. In particular, the GPS module can obtain latitude, longitude and altitude as well as three-dimensional velocity information and accurate time from the position information received from the satellite.

The position sensing unit 122-3 may be a sensor capable of sensing the position, orientation, and acceleration / deceleration of the user holding the user recognition apparatus.

The temperature / humidity measurement unit 122-4 is configured to measure the temperature and humidity of the user's external environment, and may be configured with a temperature / humidity meter.

The input unit 130 receives a feedback signal for user-customized information output from the user. In the case of voice input, the input unit 130 may have a mutual layer structure with the microphone 122-2. In the case of inputting characters, the input unit 130 may include a key pad, a touch pad Or the like.

The output unit 150 may be a display unit 151, an audio output module 152, an alarm unit 153, or the like for outputting a voice signal, a video signal, or a notification signal.

The display unit 151 displays user-customized information to the user in the form of a video signal. 4, the display unit 151 may be implemented in the form of a lens mounted on the glasses, which is a see-through display, when the interface device in the form of a glass is applied to the user-assisted intensifier 100. [

The audio output module 152 outputs a voice signal received from the communication unit 110 or stored in the memory unit 140, and outputs user-customized information to the user in the form of a voice signal. The sound output module may be implemented in the form of a bone conduction earphone attached to the inner side of the helmet, as shown in FIG.

The alarm unit 153 outputs a signal for informing the user of the occurrence of an event of the specific external environment when a specific external environment is sensed by the user recognition and augmentation apparatus 100 and includes at least one of a vibration type, Alarm can be output.

The controller 160 controls the overall operation of the user perception-enhancing device 100.

The control unit 160 includes an emotion state recognition unit 161, an external environment information generation unit 162, an output control unit 163, and a feedback information generation unit 164.

The emotion state recognition unit 161 recognizes the emotion state of the user based on the bio-signal detected by the bio-signal detection unit 121. [

The emotion state recognition unit 161 includes the emotion feature variable extraction unit 161-1, the optimizing unit 161-2 and the emotion recognition unit 161-3, Extracts a plurality of emotion feature parameters from the signal, performs individual optimization through performance evaluation of the extracted emotion feature parameters and individual feature dimension reduction, and recognizes the emotion state of the user with the optimized emotion feature parameters .

The emotion characteristic parameter extracting unit 161-1 extracts the emotion characteristic variable 161-1 from the pulse wave PPG signal, the skin conductivity GSR, the body temperature and the EEG signal from the biological signal sensing unit 121, (PWV), tension, and frequency of EEG.

For example, emotional characteristic variables for HRV prediction are HRV mean, standard deviation of the normal-to-normal intervals (RMN), root mean square of the successive differences between consecutive NN intervals (NN50), total power, nHF (high frequency band), nLF (low frequency band), LF / HF (radio high / low) frequency bands. These emotional feature variables include frequency interpolation (eg, 4 Hz interpolation) and time domain analysis (HRV), frequency domain analysis (HRV) for the interval between peak values of the pulse wave (PPG) Lt; / RTI >

In addition, the emotion feature parameter extractor 161-1 extracts a plurality of emotion feature parameters that can predict the degree of arousal of each individual from the skin conductivity (GSR) signal using the gradient technique and the magnitude of the Y-axis variation. These emotional characteristic variables include the down-sampling of the skin conductivity (GSR) signal, the mean and standard deviation through noise removal filtering, phasic detection, fluctuation through yonic detection and differential processing ) Count value.

The emotion feature parameter extractor 161-1 amplifies the EEG signal to a predetermined gain, performs fast Fourier transform, etc., converts the EEG type of the EEG into a digital signal form, component analysis algorithms, and selects the time range of EEG analysis. In addition, the emotion feature parameter extracting unit 161-1 removes unnecessary noise in the EEG analysis and extracts the frequency of the EEG for determining the emotional state.

The optimizing unit 161-2 performs performance evaluation on the plurality of emotion feature variables extracted by the emotion feature extracting unit 161-1 and outputs the emotion feature variables as the region including the emotion feature variables having the evaluation capability equal to or higher than the threshold value Reduce the dimension of variables. Here, the performance evaluation of the plurality of emotion feature parameters can be performed through a sequential floating forward search algorithm, and as a result, the dimension of the emotion feature variable is reduced to the area including the variables with excellent evaluation performance Individual optimization can be performed.

The emotion recognition unit 161-3 processes the emotion feature variables in the dimension of the selected emotion feature variables through the optimizing unit 161-2 according to a preset emotion pattern recognition rule to determine the current emotion state of each individual . Here, the emotion feature parameters in the dimension of the selected emotion feature parameters are used as input signals of the neural network algorithm learned according to the emotion pattern recognition rule, and the neural network algorithm is composed of one or more hidden layers, And generates a judgment result as an output. The individual's emotional state can be distinguished, for example, by a negative emotional state such as fear, sadness, joy, satisfaction, moderate, and positive emotional state.

Meanwhile, the emotion recognition unit 161-3 can generate emotion state determination results of the user by using the emotion feature parameters as input signals of a binary classifier including a weak classifier and a strong classifier.

Here, the weak classifier can be implemented as SVM (Support Vector Machine).

SVM is based on the Statistical Learning Theory (SLT) proposed by Vapnik in 1979. We use a method to minimize errors through SRM (Structural Risk Minimize), which is different from Empirical Risk Minimize (ERM) in general statistical learning methods.

The SVM provides an optimal separation boundary to solve the classification problem. Training data {(x i, d i), i = 1,. . . , N}, xi belongs to one of two classes, and di? {- 1, 1} serves as a label for the class. The SVM maximizes the distance between the separation interface and the support vector adjacent to the separation interface to obtain an optimal separation plane separating each class. When the optimal linear separation interface is set to f (x) = wx + b, the distance between support vector and f (x) is 1 / || w < / RTI > SVM || We minimize the w || ² and maximize the separation interval to find the optimal separation plane. This optimal separation surface is referred to as OSH (Optimal separating hyperplane).

The strong classifier can also be implemented as an Adaboosted SVM.

Adaboosted SVM is a classifier generated by applying Adaboost technique to SVM which is weak classifier. The Adaboost technique is an algorithm for generating a strong classifier with high detection performance through linear combination of weak classifiers.

The Adaboost technique performs a weak classifier multiple times on a slightly modified training sample and combines the hypothesis generated as a result of each run to produce one final hypothesis. This leads to a hypothesis with a higher accuracy than that of the weak classifiers. The main concept of the Adaboost technique is to assign weights to each example of a given training sample. Initially, all weights are the same, but each time the weak classifier returns a hypothesis, the weight of all instances that are misclassified by that hypothesis is increased. In this way, the weak classifier focuses on the difficult views of the learning data. The final hypothesis is the combination of hypotheses generated in the previous run. That is, hypotheses with low classification errors have higher weights.

The external environment information generating unit 162 generates external environmental information based on the external environment sensed by the external environment sensing unit 122. [ The external environment information generating unit 162 generates external environment information including at least one of an actual external environment image that the user sees, a sound of an actual external environment heard by the user, a user's location, and a temperature and humidity.

The external environment information generation unit 162 includes an interest object detection unit 162-1 and an emergency environment detection unit 162-2.

The object of interest detection unit 162-1 detects an object of interest to which the user is focused in the external environment image based on the actual external environment image and the user's pupil captured by the image pickup unit 122-1.

More specifically, the object of interest detection unit 162-1 includes a region of interest detection unit (not shown) and an object recognition unit (not shown).

The ROI detecting unit detects the ROI of the user and detects the ROI of the user on the ROI using the detected ROI. Specifically, the ROI detecting unit can extract the eye feature information from the detected eye path.

Herein, the eye feature information includes at least one of a change in a pupil, an eye flicker, a gaze point, a 'time to first fixation', a 'fixation length', a 'fixation count' Information such as 'observation length', 'observation count', 'fixation before', and 'participation rate'.

Here, the 'time to first fixation' is the time until the user's gaze is fixed after the stimulus (visual image) is provided, that is, the time until the user's gaze is fixed for the first time, The 'fixation length' is a time at which the user's eyes stay in a specific area of interest (AOI) of the input image, and the 'fixation count' It is the number of times the gaze stays. The 'observation length' is the total time the user's gaze stays in a specific area, the 'observation count' is the number of times the user's gaze stays in the specific area again, and the 'fixation before' ) Is the number of times that the gaze is stopped until the first gaze remains within a certain region, and the participant percentage is the percentage of users who have gazed at least once within the specific region, that is, the fixed frequency of gazing the user to a specific region. The method of extracting each of the line-of-sight characteristic information from the user's line of sight is widely known, and a description thereof will be omitted.

The region of interest detection unit detects the region of interest of the user based on at least one of the time during which the line of sight remains in the same region and the number of times the line of sight remains in the same region among the extracted line feature information.

The interest region detection unit may be configured as an NN learning apparatus that receives a plurality of input items and uses a neural network algorithm.

The object recognition unit recognizes at least one object of interest on the region of interest. The interest object recognition unit includes an image information extracting unit, a CSD processing unit, an ICA processing unit, and an extracting unit.

The image information extracting unit extracts image information on brightness (I), edge (E), and complementary colors (RG, BY) of the input image. Specifically, the image information extracting unit extracts image information of at least one of brightness, edge, symmetry, and complementary color of the input image based on R (Red), G (Green), and B Can be extracted.

The CSD processing unit may generate a brightness feature map, a direction feature map, a symmetric feature map, and a color feature map by performing a center-surround difference (CSD) and a normalization process on the extracted image information.

The ICA processing unit performs an independent component analysis on the output feature map to generate a salient map (SM).

The extracting unit recognizes the protruding area on the protruding map as an object of interest. Specifically, the extracting unit performs a reinforcement process or an inhibition process on the plurality of protruding points included in the protruded map output from the ICA processing unit using the learned information, assigns a priority to a plurality of protruding points, The region having the rank is detected as the object of interest.

The emergency environment detecting unit 162-2 detects whether the sensed external environment is a specific external environment and controls the alarm unit 153 to preferentially output the warning alarm through the alarm unit 153 when the specific external environment is detected.

Specifically, the emergency environment detecting unit 162-2 checks whether the sensed external environment meets a condition of a specific external environment stored in advance in the memory unit 140. If the sensed external environment satisfies the condition of the specific external environment, And outputs a warning alarm to the user.

The particular external environment may be an external situation that may immediately affect the user, for example, when the vehicle is approaching the user at high speed, the user's location may be in an accident frequent area.

The output control unit 163 outputs the user-customized information received through the communication unit 110 to fit the current situation of the user. The output control unit 163 controls the output control unit 163 based on the type of the user-customized information received from the communication unit 110, the emotion state of the user recognized from the emotion state recognition unit 161 and the external environment information generated by the external environment information generation unit 162 And controls the output unit 150.

For example, the output control unit 163 may output user-customized information as a voice signal through the sound output module 152 of the output unit 150 when the user is operating. The output control unit 163 may output the user-customized information as a video signal through the display unit 151 of the output unit 150 when the ambient noise is not less than a predetermined value.

Accordingly, the output control unit 163 can control the user to output the user-customized information to the user so as to meet the current situation of the user and improve the acceptability.

The feedback information generating unit 164 generates feedback information on the output user customized information and controls the communication unit 110 to transmit the feedback information to the server side.

The feedback information generating unit 164 generates feedback information based on the user's reaction to the result of outputting the user-customized information received through the input unit 130, that is, the explicit feedback of the user. The feedback information generating unit 164 determines the degree of affirmity index in a predetermined manner in a user response in the form of a voice or a character received through the input unit 130. If the user- It is judged whether the service is suitable for the service.

More specifically, the feedback information generation unit 164 searches for a keyword from a user voice or a character, and calculates a positive degree based on the searched keyword. At this time, the feedback information generating unit 164 searches for a keyword matched to the positive / negative keyword database stored in the memory unit 140, and calculates a positive degree by adding the affinity weight or irregularity weight associated with the searched keyword .

In addition, the feedback information generating unit 164 re-detects the biological signal and / or the external environment through the sensing unit 120, and determines whether the emotion state and / And generates feedback information. That is, the feedback information generating unit 164 performs a self-evaluation on whether the user-customized information is a service suitable for the user during the outputting of the user-customized information or the changed state of the user's emotional state and / or the external environment information after the output .

5 is a flowchart illustrating a method for providing a user awareness and augmentation service in the UE or UE shown in FIG.

First, the user's augmentation apparatus senses a user's bio-signal and recognizes the user's emotional state (310). At this time, the biological signal includes at least one of a user's pulse wave (PPG), skin conductivity (GSR), and brain wave signal.

The recognition of the emotional state of the user is performed by extracting at least one emotional characteristic variable for recognizing the emotional state from the biological signal, performing a performance evaluation on the extracted emotional characteristic variable, And individual optimization is performed on the emotion feature variables by reducing the dimension of the emotion feature variables to the area including the extracted emotion feature variables. In addition, the emotion state of the user is recognized according to a preset method using the optimized emotion feature parameters. At this time, recognizing the user's emotional state may retrieve an emotional state corresponding to the optimized emotional feature variables in a previously stored emotional state database. Thus, the emotional characteristic variables can be adjusted so that the emotional state can be recognized in an optimum state for each individual through the biometric signals with a large individual variation.

In addition, the user's augmentation apparatus senses the external environment of the user to generate external environment information (320). The sensed external environment may include an actual external environment image viewed by a user through a camera, a sound of a real external environment heard by a user recorded through a microphone, a user's location, temperature, time, weather, The external environment information is generated through a predetermined processing on the environment.

In addition, the external environment information may include an object to which the user is focused, that is, an object of interest. Such an object of interest can be detected through the detected eye path by detecting the user's eye path on the sensed external environment, i.e., the actual external environment image. More specifically, the user's augmentation apparatus detects the user's gaze path to the external environment, detects the user's area of interest with respect to the external environment using the detected gaze path, and detects an object of interest on the area of interest. At this time, the ROI may be based on the ROI extracted from the detected ROI, and the ROI may be at least one of a time during which a line of sight in the same area remains and a line of sight in the same area have.

When the user's emotion state and external environment information are generated (310, 320), the user's augmentation device transmits the user's emotion state and external environment information to the server side (330), receives user-customized information from the server, and outputs it to the user (340). At this time, the user-customized information may be information matching the user emotion state and external environment information retrieved from the user-customized information database previously stored on the server side. On the other hand, outputting the user-customized information can be output in the form of at least one of visual image, sound, and vibration. Also, if the external environment information is matched to a specific external environment, it can be controlled to preferentially output the warning alarm before outputting the customized information.

Further, the user's augmentation apparatus detects the explicit / implied response from the user as a result of outputting the user-customized information or generates the feedback information based on the changed external environment information (350). That is, the user's augmentation apparatus re-senses the user's emotional state and external environment information while the user-customized information is output to the user and / or the user, and feedbacks based on the at least one of the emotional state and the external environment information Information. Accordingly, a self-evaluation can be performed as to whether the user-customized information is suitable for the user's situation, and the user-customized information database on the server side can be updated to grow the user-customized information database in real time.

FIG. 6 is a control block diagram of the server shown in FIG. 1. FIG.

6, the server 200 according to the present embodiment includes a communication unit 210, a memory unit 220, and a control unit 230. [ Figure 6 shows a server with various components. However, not all illustrated components are required. The server may be implemented by more components than the components shown, and the server may be implemented by fewer components. Hereinafter, the above-mentioned components will be described in detail.

The communication unit 210 may be a wireless Internet module, for example, for performing wireless communication with the user authentication / enhancement apparatus. The communication unit 210 receives the user emotion state and external environment information, which are user situation information, from the user awareness enhancing apparatus, and transmits user-customized information corresponding to the user situation information to the user's augmentation apparatus. Also, the communication unit 210 receives feedback information that is a result of transmitting the user-customized information.

The memory unit 220 may store a program for processing and controlling the controller 230, and may perform a function for temporary storage of input / output data.

The memory unit 220 stores a user-customized information database. The user-customized information database is a database that is learned by modeling the user's emotional state, external environment information, and mutual relation among them, and is a database storing a list of user-customized information corresponding to the current user situation.

The control unit 230 controls the overall operation of the server 200. The control unit 230 includes a user-customized information searching unit 231 and a learning unit 232.

The user-customized information searching unit 231 searches for user-customized information based on the user's emotion state and external environment information received from the communication unit 210, and controls the transmitted user-customized information through the communication unit 210. The user-customized information retrieving unit 231 retrieves user-customized information that best matches the current user situation from the user-customized information database that has been learned through the learning unit 232. [ Accordingly, the user-customized information searching unit 231 can provide an optimal cognitive enhancement service in a situation where the user is present, such as a user's emotional state, perceived state, and surrounding environment.

The learning unit 232 updates the user-customized information database of the memory unit 220 by reflecting the feedback information when the feedback information is received from the communication unit 210. [ The learning unit 232 models an existing user-customized information database by modeling the changed user emotional state, external environment information, and mutual relationship of the user customized information output from the user or the enhancer. That is, the learning unit 232 determines the fitness for the user's situation of the user-customized information transmitted based on the user's situation change according to the transmitted user-customized information, To learn progressively.

The learning unit 232 performs incremental / adaptive learning to efficiently update an existing learning model, that is, a user-customized information database, using only new data, that is, feedback information.

In addition, the learning unit 232 may include an Incremental Subspace (IMA) model that temporally models changes in features such as Incremental Principal Component Analysis (IPCA), Incremental Linear Discriminant Analysis (ILDA), and Incremental Non-Negative Matrix Factorization ) Algorithm, an ISVM (Incremental Support Vector Machine), or an INN (Incremental Neural Network) may be applied to update the user-customized information database.

7 is a diagram illustrating a user-customized information providing system according to another embodiment of the present invention.

Referring to FIG. 7, the user-customized information providing system 1000 according to the present embodiment includes a plurality of user awareness enhancing apparatuses 100-1, 100-2,... 100-N and a server 200.

Each user perception enhancing device is identical in function and configuration to the user enhancing device shown in FIG. 1, and a description thereof will be omitted.

The server 200 communicates with each of the plurality of user awareness enhancing apparatuses 100-1, 100-2, ..., 100-N. The server 200 has the same basic functions and configuration as the servers shown in FIGS. 1 and 6, but transmits the user-customized information to the plurality of user awareness enhancing apparatuses 100-1, 100-2, ..., 100-N 100-1, 100-2,..., 100-N, receives feedback information from each of the user aware augmentors 100-1, 100-2, ... 100- In the point of clustering users of the system. A description thereof will be described later in detail with reference to FIGS. 8 and 9. FIG.

8 is a control block diagram of the server shown in FIG.

Referring to FIG. 8, the server 200 according to the present embodiment includes a communication unit 210, a memory unit 220, and a control unit 230. Figure 8 shows a server with various components. However, not all illustrated components are required. The server may be implemented by more components than the components shown, and the server may be implemented by fewer components. Hereinafter, the above-mentioned components will be described in detail.

The communication unit 210 communicates with a plurality of user perception enhancing devices, respectively. The communication unit 210 receives the user's emotional state and external environment information from each user's perception device, transmits user-customized information corresponding thereto, and receives feedback information resulting from outputting customized information.

The memory unit 220 stores a user-customized information database corresponding to each user's perception enhancer. That is, the memory unit 220 may assign an ID to each of a plurality of user identification devices, and may store a user-customized information database for each ID.

Meanwhile, the user-customized information database according to the present embodiment includes a similar tendency user cluster in which a plurality of users are clustered according to the degree of similarity between users measured by the controller 230. [ For example, the user-customized information database may include a pseudo-evolving user cluster grouped with the identity of the user or the augmentor of similarity-oriented users.

Other configuration and features of the user-customized information database are the same as in FIG. 6, and a description thereof will be omitted.

The control unit 230 according to the present embodiment includes a similar tendency user cluster generating unit 231, a user-customized information searching unit 232, and a learning unit 233.

The similarity user cluster generating unit 231 calculates similarities between users of a plurality of user or augmentation apparatuses, and generates similarity-oriented user clusters based on the calculated similarities.

The similarity user cluster generation unit 231 can calculate the similarity based on the received feedback information for the same user-customized information and calculate the similarity based on the pattern of the user-customized information database of each user have.

The similarity user cluster generation unit 231 calculates the similarity of users' tendencies through various methods such as Pearson correlation score, Euclidean distance, Jacquard coefficient, Manhattan distance, and the like.

For example, the similarity tendency user cluster generation unit 231 of this embodiment can calculate the similarity degree through the Pearson correlation score. The Pearson correlation score is a measure of how well two datasets are represented in a straight line. The similarity user cluster generation unit 231 may calculate Pearson correlation scores among users by comparing feedback information of each user after providing the same user-customized information. At this time, the Pearson correlation score can be calculated as a value within a range of -1 to 1, which indicates that feedback information of the same tendency has been received after providing the same user-customized information as the number becomes 1.

When the degree of similarity among the plurality of users is calculated as described above, the pseudo-tendency user cluster generating unit 231 creates the clusters so that the users having similarities higher than the set value are managed in the same cluster. The similarity user cluster generating unit 231 can generate a cluster composed of IDs of the user-augmentation apparatuses of the similarity users.

On the other hand, the pseudo tendency user cluster generating section 231 can update the cluster.

The similarity-based user cluster generating unit 231 updates the user-augmented device of the new user to the cluster when the augmenting device and the server communicate with each other whether the user is a new user who is not registered in the cluster. Specifically, the similarity-based user cluster generating unit 231 measures the similarity between the new user and the existing user registered in the cluster, and notifies the clusters belonging to the existing user having the similarity degree equal to or higher than the set value to the user- Lt; / RTI >

In addition, the pseudo-tendency user cluster generating unit 231 may measure the similarity between users at regular intervals in consideration of the fact that the tendency between users may vary, and update the pseudo tendency user cluster according to the measured similarity.

The user-customized information searching unit 232 searches for user-customized information corresponding to the user's emotion state and external environment information of the user's augmentation apparatus received from the communication unit 210. [ The user-customized information searching unit 232 searches for the ID of the user-ID apparatus that requests the user-customized information, and retrieves user-customized information corresponding to the user's emotion state and external environment information from the user- do.

The learning unit 233 learns a user-customized information database in real time or periodically.

Specifically, when the feedback information is received from the user recognition / augmentation apparatus, the learning unit 233 reflects the feedback information to the user-customized information database corresponding to the user or the augmentation apparatus to perform real-time / incremental learning. Incremental subspace algorithm that temporally models changes in features such as Incremental Principal Component Analysis (IPCA), Incremental Linear Discriminant Analysis (ILDA), and Incremental Non-negative Matrix Factorization (INMF) Can be performed by applying classifiers that temporally model changes in feature distribution such as Incremental Support Vector Machine (ISVM) or Incremental Neural Network (INN).

In addition, the learning unit 233 periodically updates the user-customized information database using the similar-tendency user cluster.

More specifically, the learning unit 233 may search for a group of similarity-oriented users in the similarity-oriented user cluster, and may update each user-customized information database by reflecting the user-customized information databases of the group of similarity-oriented users retrieved . Thereby, not only feedback information based on personal experience but also a user-customized information database reflecting collective intelligence can be provided.

FIG. 9 is a flowchart showing a method of providing user-customized information in the server shown in FIG.

First, the server receives the user's emotional state and external environment information, which is the current status information of the user, from the user's augmentation device requesting the user-customized information (410).

In addition, the server verifies 420 whether the user of the user augmentation apparatus requesting the user-customized information is a new user. More specifically, the server searches based on the pseudo tendency user cluster whether the user is a new user registered in the pseudo tendency user cluster or an existing registered user. At this time, the server can search for the existence of the ID of the UE or the UE requesting the user-customized information in the similarity-oriented user cluster.

If it is determined that the new user is a new user (420), the server measures the similarity based on the user-customized information database of the new user and the existing user (430) Update the new user (440).

In addition, the user-customized information database corresponding to the received user's current situation information is retrieved and transmitted from the user's customized information database 450, and the user- The information database is updated (470).

Meanwhile, in the present embodiment, it has been confirmed that the user of the user or the augmenting device requesting the user-customized information is included in the similar tendency user cluster and the step of updating the similar tendency user cluster when the user is a new user. However, Element, and can be implemented only by a series of steps of retrieving and transmitting the user-customized information without receiving the above-described steps, receiving the feedback signal, and updating the user-customized information database accordingly.

According to one aspect of the present invention, it is possible to provide important information that the user is not aware of, or user-customized information corresponding to the user's situation, using the existing eyeglass platform, The intelligent cognitive enhancing service capable of real-time growth can be provided by progressively learning the user-customized information database through the user.

In addition, it is possible to provide a user-customized information providing system using collective intelligence by reflecting the provision of user-customized information according to the change of the user situation of the similar user group.

Such a technique for providing user-customized information may be implemented in an application or may be implemented in the form of program instructions that can be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, and the like, alone or in combination.

The program instructions recorded on the computer-readable recording medium may be ones that are specially designed and configured for the present invention and are known and available to those skilled in the art of computer software.

Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like.

Examples of program instructions include machine language code such as those generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules for performing the processing according to the present invention, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it will be understood by those skilled in the art that various changes and modifications may be made therein without departing from the spirit and scope of the invention as defined in the appended claims. It will be possible.

100: user perception enhancement device
200: Server

Claims (21)

CLAIMS 1. A method of providing user-customized information in a user-
Recognizes the user's emotional state by sensing the user's biometric signal,
Sensing external environment of the user to generate external environment information,
Transmitting the user emotion state and external environment information to the server,
Receiving user-customized information retrieved from the user-customized information database of the server based on the user emotion state and external environment information, and outputting the user-
And generating feedback information on the output user-customized information to update the user-customized information database to transmit the customized information to the server. The method according to claim 1,
The user-
The feedback information is incrementally incremented by Incremental Principal Component Analysis (IPCA), Incremental Linear Discriminant Analysis (ILDA), Incremental Non-Negative Matrix Factorization (INMF), Incremental Support Vector Machine (ISVM), or Incremental Neural Network (INN) A method for providing user-tailored information to be learned. The method according to claim 1,
Wherein the feedback information includes:
Wherein the user-customized information is generated based on a variation of at least one of the user emotion state and external environment information as a result of outputting the user-customized information to the user. The method according to claim 1,
Recognizing the user emotional state comprises:
Extracting at least one emotion feature parameter from the bio-signal,
Performing a performance evaluation on the extracted emotional characteristic variable,
The method according to any one of claims 1 to 3, further comprising the steps of: individually reducing the dimension of the emotion feature variable to an area including the emotion feature variable having the evaluation performance of the threshold value or more,
Wherein the user is aware of the emotional state of the user with the optimized emotional feature variable. 5. The method of claim 4,
The bio-
Wherein at least one of the user's pulse wave (PPG), skin conductivity (GSR), and brain wave signal is provided. The method according to claim 1,
Generating the external environment information includes:
And detecting the user's gaze path to the external environment to detect the user's point of interest in the external environment. The method according to claim 6,
Detecting the user's object of interest comprises:
Detecting a user's gaze path to the external environment,
Detecting a region of interest of the user with respect to the external environment using the detected sight path,
And detecting an object of interest on the region of interest. 8. The method of claim 7,
Detecting the region of interest comprises:
Extracting gaze feature information from the detected gaze path of the user, and detecting the gaze region of the user with the gaze feature information. 9. The method of claim 8,
Wherein the gaze characteristic information is at least one of a time at which a line of sight in the same area remains and a number of times a line of sight in the same area stays. The method according to claim 1,
The user-
A method of providing user-tailored information comprising a similar-trending user cluster clustering a plurality of users based on measured similarity after measuring a similarity between a plurality of users. 11. The method of claim 10,
The user-
Wherein the user-customized information is periodically updated based on user-customized information databases of the group of similar-oriented users retrieved from the similarity-oriented user cluster. The method according to claim 1,
Wherein the external environment information comprises:
Wherein the user-customized information includes at least one of an actual external environment image the user sees, a sound of the actual external environment the user hears, a location of the user, and a temperature. 13. The method of claim 12,
Outputting the user-customized information to the user,
And outputting the user-customized information in a form of at least one of visual image, sound, and vibration based on the external environment information. The method according to claim 1,
And if the external environment information matches a specific external environment, outputting a warning alarm preferentially. 15. A computer-readable recording medium on which a computer program is recorded for providing user-customized information according to any one of claims 1 to 14. Sensing user's emotional state by sensing a user's biological signal, sensing external environment of the user to generate external environment information, outputting user-customized information corresponding to the user emotion state and external environment information to the user, A user awareness enhancer for generating and transmitting feedback information on the user-customized information; And
Customized information corresponding to the user's emotion state and external environment information in the user-customized information database that has been learned, and transmits the user-customized information to the user's augmentation apparatus, and based on the feedback information, And a server for updating the user-customized information. 17. The method of claim 16,
The server comprises:
The feedback information is added to the user-customized information database by an Incremental Principal Component Analysis (IPCA), an Incremental Linear Discriminant Analysis (ILDA), an Incremental Non-Negative Matrix Factorization (INMF), an Incremental Support Vector Machine (ISVM) A user-tailored information provision system that progressively learns information. 17. The method of claim 16,
The user recognition enhancer is provided in plurality,
Wherein the server generates a similar tendency user cluster in which users of a plurality of users or augmentors are clustered based on a predetermined criterion. 19. The method of claim 18,
The server comprises:
And generating a similar tendency user cluster by clustering the plurality of users whose calculated similarity is equal to or greater than a set value. 20. The method of claim 19,
The server comprises:
Confirms whether the user of the user awareness enhancer requesting the user-customized information is a new user not registered in the pseudo-tendency user cluster,
And if the new user is a new user, calculates a similarity with an existing user registered in the similar tendency user cluster and updates the new user to a cluster of the existing user having a degree of similarity higher than a set value. 20. The method of claim 19,
The server comprises:
Wherein the user-customized information database periodically updates the user-customized information databases of the similarity-oriented users group retrieved from the similarity-oriented user cluster.

Images