WO2022183424A1

WO2022183424A1 - Emotion recognition-based online social method and apparatus, and storage medium

Info

Publication number: WO2022183424A1
Application number: PCT/CN2021/079035
Authority: WO
Inventors: 王化
Original assignee: 深圳技术大学
Priority date: 2021-03-04
Filing date: 2021-03-04
Publication date: 2022-09-09

Abstract

Disclosed are an emotion recognition-based online social method and apparatus, and a storage medium. The emotion of a local chat user is recognized on the basis of physiological characteristic data of the user, and emotion expression content such as an emotion expression animation and emotion expression dynamic text corresponding to the emotion of the user is sent in real time to a chat client used by a peer chat user for display; the emotion expression animation and the emotion expression dynamic text can be displayed in corresponding dynamic modes to reflect different emotions and feelings, for example, when the chat user is in high spirits, the animation or the dynamic text can use faster actions, higher speeds, larger sizes, and brighter colors, and when the chat user is in low spirits, the animation or the dynamic text use slower actions, lower speeds, smaller sizes, and darker colors. The emotion of a user is visualized and reflected by means of an animation or dynamic text, thereby effectively ensuring that two chat parties know each other's emotions, and significantly improving effectiveness of chat interaction between the two parties.

Description

Online social networking method, device and storage medium based on emotion recognition

technical field

The present invention relates to the field of electronic technology, and in particular, to an online social networking method, device and storage medium based on emotion recognition.

Background technique

With the continuous development of Internet technology, online chat applications have become the main tool for users to conduct instant communication. However, online chat is much more primitive than face-to-face communication, and it is often difficult for online chat to convey many basic information, such as user emotions. In practical applications, if both parties in the chat cannot accurately know each other's emotions, the content of the chat cannot be accurately understood, and the effectiveness of the chat interaction is poor.

technical problem

The main purpose of the embodiments of the present invention is to provide an online social networking method, device and storage medium based on emotion recognition, which can at least solve the problem that the chatting parties in the related art cannot accurately know each other's emotions, resulting in that the chatting content cannot be accurately understood and chatted. The problem of less effective interaction.

technical solutions

To achieve the above object, a first aspect of the embodiments of the present invention provides an online social networking method based on emotion recognition, the method comprising:

Collect the physiological characteristic data of local chat users through sensors;

Determine a corresponding user emotion recognition result based on the physiological characteristic data;

Obtain corresponding emotional expression content according to the user emotion recognition result; wherein, the emotional expression content includes at least one of emotional expression animation and emotional expression dynamic text;

The emotional expression content is sent to the chat client used by the peer chat user in real time.

To achieve the above object, a second aspect of the embodiments of the present invention provides an online social networking device based on emotion recognition, the device comprising:

The acquisition module is used to collect the physiological characteristic data of the local chat user through the sensor;

a determining module, for determining a corresponding user emotion recognition result based on the physiological characteristic data;

an acquisition module, configured to acquire corresponding emotional expression content according to the user emotion recognition result; wherein, the emotional expression content includes at least one of emotional expression animation and emotional expression dynamic text;

A sending module, configured to send the emotional expression content to the chat client used by the opposite chat user in real time.

To achieve the above object, a third aspect of the embodiments of the present invention provides an electronic device, the electronic device includes: a processor, a memory, and a communication bus;

The communication bus is used to realize the connection communication between the processor and the memory;

The processor is configured to execute one or more programs stored in the memory to implement the steps of any one of the above-mentioned online social networking methods.

To achieve the above object, a fourth aspect of the embodiments of the present invention provides a computer-readable storage medium, where the computer-readable storage medium stores one or more programs, and the one or more programs can be processed by one or more The server executes the steps to implement any one of the above-mentioned online social networking methods.

beneficial effect

According to the online social networking method, device and storage medium based on emotion recognition provided by the embodiment of the present invention, the physiological characteristic data of the chatting user at the local end is collected by sensors; the corresponding user emotion recognition result is determined based on the physiological characteristic data; and the user emotion recognition result is obtained according to the user emotion recognition result The corresponding emotional expression content; the emotional expression content is sent to the chat client used by the peer chat user in real time for display. Through the implementation of the present invention, the emotion of the chat user at the local end is identified based on the user's physiological characteristic data, and emotion expression content such as emotion expression animation and emotion expression dynamic text corresponding to the user's emotion are sent to the chat client used by the opposite end chat user in real time. The display realizes that the emotional information of chatting users is conveyed through animation, dynamic text, etc., which effectively ensures that both parties in the chat know each other's emotions, which can significantly improve the effectiveness of the chat interaction between the two parties.

Other features of the present invention and corresponding effects are set forth in later parts of the specification, and it should be understood that at least some of the effects will become apparent from the description of the present specification.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those skilled in the art, other drawings can also be obtained according to these drawings without creative efforts.

FIG. 1 is a schematic diagram of a basic flow of an online social networking method provided by a first embodiment of the present invention;

Fig. 2 is a kind of emotion recognition map provided by the first embodiment of the present invention;

3 is a schematic diagram of an emotion expression animation provided by the first embodiment of the present invention;

4 is a schematic diagram of a GSR data change provided by the first embodiment of the present invention;

5 is a schematic diagram of a program module of an online social networking device provided by a second embodiment of the present invention;

FIG. 6 is a schematic structural diagram of an electronic device according to a third embodiment of the present invention.

Embodiments of the present invention

In order to make the purpose, features and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. The embodiments described above are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts shall fall within the protection scope of the present invention.

First embodiment:

In order to solve the problem in the related art that the two chatting parties cannot accurately know each other's emotions, the chat content cannot be accurately understood, and the effectiveness of the chat interaction is poor, this embodiment proposes an online social networking method based on emotion recognition, such as FIG. 1 is a schematic diagram of the basic flow of the online socializing method provided by this embodiment. The online socializing method provided by this embodiment includes the following steps:

Step 101 , collecting the physiological characteristic data of the chat user at the local end through the sensor.

Specifically, linguist studies have shown that only 20% to 30% of the information conveyed in a conversation can be conveyed through text chat, thus proving the importance of non-verbal information, such as emotions, which is necessary for human cognition , and affect different aspects of people's lives. For example, when we are excited, our perceptions tend to select happy events, while negative emotions do the opposite. This suggests that online chat would benefit from understanding the emotional state of others. Armed with information about the user's emotional state, the system can interact with the user more effectively.

In practical applications, users connect specific sensors during chat, the sensors are digitally sampled, and data capture can be achieved through modules written in Visual C++.

In an optional implementation of this embodiment, the physiological characteristic data may be galvanic skin response data and electrical muscle response data; wherein, the galvanic skin response data is analyzed by a galvanic skin response meter (GSR, Galvanic Skin Response apparatus). Acquisition, the galvanic skin response instrument is an instrument that records the changes of the galvanic skin response in the form of curve ripples, and the electromyography can be used to collect the data of the electrical muscle response and record the muscle bioelectricity graph.

In another optional implementation of this embodiment, the physiological feature data may also be gaze tracking data, and the gaze tracking data may include pupil dilation data, eye movement data, etc. Eye images are collected, and then multiple eye images are recognized according to the image recognition algorithm to obtain gaze tracking data.

Step 102: Determine a corresponding user emotion recognition result based on the physiological characteristic data.

Specifically, in practical applications, when the user's emotions change, the physiological characteristics usually change adaptively. Therefore, in this embodiment, the user's emotions can be adaptively sensed through the physiological characteristics data.

In this embodiment, specific implementations for determining the corresponding user emotion recognition result based on the physiological feature data include but are not limited to the following two:

Mode 1: Determine the corresponding emotional arousal data based on the galvanic skin response data, and determine the corresponding emotional valence data based on the muscle electrical response data; combine the emotional arousal data and the emotional valence data to determine the corresponding user emotion recognition result.

Specifically, since GSR data has less noise signal than BVP data, this embodiment uses a GSR sensor to detect emotional arousal data, and the GSR sensor for measuring skin conductivity (SC) is connected to the middle finger and index finger of the user's non-dominant hand , and the signal can be recorded with the ProComp+ device. SC varied linearly with overall arousal levels and increased with anxiety and stress. In an embodiment, the emotional arousal data can be detected and obtained by analyzing the peaks and troughs of the GSR data.

Figure 2 shows an emotion recognition map provided by the present embodiment, wherein the horizontal axis represents emotional valence data, the vertical axis represents emotional arousal data, and the elements in each quadrant are different types of emotion recognition results, including excitement , relaxation, sadness, fear, etc. In this embodiment, the emotional valence data represents the user's emotional type, the negative valence emotions include fear, sadness, etc., the positive valence emotions include happiness, excitement, etc., and the emotional arousal data indicates that the user is in the corresponding The intensity of the emotion under the emotion type.

In the second method, the corresponding attention representation data and interest representation data are determined based on the gaze tracking data; the corresponding user emotion recognition result is determined by combining the attention representation data and the interest representation data.

Specifically, this embodiment makes real-time response to the user's sight line information, and records eye gaze and pupil dilation data during the online chat. Eye movements can reveal the user's interest and attention, enrich non-verbal information by detecting attention, interest, etc. from the user's real-time eye movement data, and use gaze and eye movement information to facilitate emotion-related reasoning.

Step 103: Acquire corresponding emotion expression content according to the user emotion recognition result.

In this embodiment, the emotion expression content includes at least one of emotion expression animation and emotion expression dynamic text. FIG. 3 is a schematic diagram of an emotion expression animation provided in this embodiment. The emotion expression content of this embodiment is adapted to the current emotional state of the user, and is different from the plain text information, and can be used to express more subtle information including the user's emotion.

In an optional implementation of this embodiment, the above-mentioned step of obtaining corresponding emotional expression content according to the user emotion recognition result specifically includes: determining the corresponding content type and content display attribute according to the user emotion recognition result; and content display attributes to obtain the corresponding emotional expression content.

Specifically, the content type of the emotional expression content may include: joy, anger, sadness, joy, etc., and the content display attributes may include: change speed, size, color, and the like. In this embodiment, the display attributes of the emotional expression animation and the emotional expression dynamic text can be determined according to the emotional awakening data, and the specific types of the emotional expression animation and the emotional expression dynamic text can be determined according to the emotional valence data. In practical applications, for example, when the GSR data suddenly increases and peaks, the corresponding generated animation or dynamic text can have higher speed, larger size and brighter color; when the GSR data decreases, all Correspondingly, the change speed of the generated emotional expression content becomes slower, and the display color becomes darker.

Step 104: Send the emotional expression content to the chat client used by the opposite chat user in real time.

Specifically, this embodiment uses emotional expression content to describe and express user emotions to both chatting parties, so as to convey emotions and subtle information during online chatting. Since both chatting parties can effectively perceive each other's emotions, it is possible to increase each other's presence in chatting. engagement and more effective chat interaction. In practical applications, users can formulate expression animations or emotional expression dynamic text through buttons, shortcut keys or markers embedded in text messages, and embed emotional expression animations in specific positions (such as the front) of text messages to accurately convey chatting parties. For example, "<Happy> I'm happy!", where <Happy> is the emotion expression animation shown in the first row and first column of Figure 3, so as to fully convey the user's joyful emotion through this animation. In practical applications, different types of emotion expression animations can be flexibly used according to different application scenarios. For example, when the user's emotions are sad, the <Sad> animation shown in the second row and third column in Figure 3 can be used. .

It should also be noted that this embodiment also conducts usability research on the chat client implemented by the online social networking method, and the research objects are users who use the chat client of this embodiment and the traditional chat client at the same time. Subjects were randomly assigned to use both chat systems, they were in the same building but were not supposed to meet. Divide them into three pairs and talk to their partners individually, each lasting more than an hour, and they talk about school lessons, etc. After the conversation, subjects answered questionnaires and rated the chat system. First, this example examines GSR data and responses from subjects who were asked when they felt they enjoyed chatting the most, and this example compared their responses with the GSR results. There is a good correlation between GSR data and user-reported tension. The subjects indicated that they gradually focused their attention on the dialogue, and the GSR also showed similar changes. Figure 4 shows the schematic diagram of the changes in the GSR data provided in this example. As the subjects focused more on the dialogue, the GSR increased. . (The graph records the user's GSR every minute). This suggests that GSR can be used to judge changes in emotions in online conversations in real time. The findings also suggest that emotional information may increase subjects' engagement in the conversation. Emotional messages from chat partners give users the feeling that they are not only exchanging text messages but also communicating their feelings with each other, which enables them to be more involved in the conversation.

According to the online social networking method based on emotion recognition provided by the embodiment of the present invention, the physiological feature data of the chatting user at the local end is collected by sensors; the corresponding user emotion recognition result is determined based on the physiological feature data; the corresponding emotion expression content is obtained according to the user emotion recognition result ; Send the emotional expression content to the chat client used by the peer chat user for display. Through the implementation of the present invention, the emotion of the chat user at the local end is identified based on the user's physiological characteristic data, and emotion expression content such as emotion expression animation and emotion expression dynamic text corresponding to the user's emotion are sent to the chat client used by the opposite end chat user in real time. The display realizes that the emotional information of chatting users is conveyed through animation, dynamic text, etc., which effectively ensures that both parties in the chat know each other's emotions, which can significantly improve the effectiveness of the chat interaction between the two parties.

Second embodiment:

In order to solve the problems in the related art that the two chatting parties cannot accurately know each other's emotions, the chat content cannot be accurately understood, and the effectiveness of the chat interaction is poor, this embodiment shows an online social device based on emotion recognition, specifically Referring to FIG. 5 , the online social networking device in this embodiment includes:

The collection module 501 is used for collecting the physiological characteristic data of the chat user at the local end through the sensor;

A determination module 502, configured to determine a corresponding user emotion recognition result based on the physiological characteristic data;

The obtaining module 503 is configured to obtain the corresponding emotion expression animation according to the user emotion recognition result; wherein, the emotion expression content includes at least one of emotion expression animation and emotion expression dynamic text;

The sending module 504 is configured to send the emotional expression content to the chat client used by the opposite chat user in real time.

In some implementations of this embodiment, the physiological characteristic data are galvanic skin response data and electrical muscle response data. Correspondingly, the determining module 502 is specifically configured to: determine the corresponding emotional arousal data based on the galvanic skin response data, and determine the corresponding emotional valence data based on the electrical muscle response data; and determine the corresponding user based on the emotional arousal data and the emotional valence data. Emotion recognition results.

In other implementations of this embodiment, the physiological feature data is gaze tracking data. Correspondingly, the determining module 502 is specifically configured to: determine the corresponding attention representation data and interest representation data based on the gaze tracking data; and determine the corresponding user emotion recognition result in combination with the attention representation data and the interest representation data.

In some implementations of this embodiment, the obtaining module 503 is specifically configured to: obtain the corresponding content type and content display attribute according to the user emotion recognition result; obtain the corresponding emotion expression content according to the content type and content display attribute.

It should be noted that, the online socializing methods based on emotion recognition in the foregoing embodiments can all be implemented based on the online socializing device based on emotion recognition provided in this embodiment, and those of ordinary skill in the art can clearly understand that the descriptions For convenience and simplicity, for the specific working process of the online social networking device described in this embodiment, reference may be made to the corresponding process in the foregoing method embodiments, which will not be repeated here.

Using the online social device based on emotion recognition provided by this embodiment, the physiological characteristic data of the chatting user at the local end is collected through the sensor; the corresponding user emotion recognition result is determined based on the physiological characteristic data; the corresponding emotion expression content is obtained according to the user emotion recognition result; Send the emotional expression content to the chat client used by the peer chat user for display. Through the implementation of the present invention, the emotion of the chat user at the local end is identified based on the user's physiological characteristic data, and emotion expression content such as emotion expression animation and emotion expression dynamic text corresponding to the user's emotion are sent to the chat client used by the opposite end chat user in real time. The display realizes that the emotional information of chatting users is conveyed through animation, dynamic text, etc., which effectively ensures that both parties in the chat know each other's emotions, which can significantly improve the effectiveness of the chat interaction between the two parties.

Third embodiment:

This embodiment provides an electronic device, as shown in FIG. 6 , which includes a processor 601, a memory 602, and a communication bus 603, wherein: the communication bus 603 is used to implement connection and communication between the processor 601 and the memory 602; processing The device 601 is configured to execute one or more computer programs stored in the memory 602 to implement at least one step of the online social networking method in the first embodiment.

The present embodiments also provide a computer-readable storage medium embodied in any method or technology for storing information, such as computer-readable instructions, data structures, computer program modules, or other data volatile or nonvolatile, removable or non-removable media. Computer-readable storage media include but are not limited to RAM (Random Access Memory, random access memory), ROM (Read-Only Memory, read-only memory), EEPROM (Electrically Erasable Programmable read only memory, electrically erasable programmable read only memory), flash memory or other memory technology, CD-ROM (Compact Disc Read-Only Memory), Digital Versatile Disc (DVD) or other optical disk storage, magnetic cartridge, magnetic tape, magnetic disk storage or other magnetic storage device, or may be used to store desired information and be accessible by a computer any other medium.

The computer-readable storage medium in this embodiment may be used to store one or more computer programs, and the stored one or more computer programs may be executed by a processor to implement at least one step of the method in the first embodiment.

This embodiment also provides a computer program, which can be distributed on a computer-readable medium and executed by a computer-readable device to implement at least one step of the method in the above-mentioned first embodiment; and in some cases , at least one of the steps shown or described may be performed in an order different from that described in the above embodiments.

This embodiment also provides a computer program product, including a computer-readable device, where the computer program as shown above is stored on the computer-readable device. In this embodiment, the computer-readable device may include the computer-readable storage medium as described above.

It can be seen that those skilled in the art should understand that all or some of the steps in the methods disclosed above, the functional modules/units in the system and the device can be implemented as software (which can be implemented by computer program codes executable by a computing device) ), firmware, hardware, and their appropriate combination. In a hardware implementation, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be composed of several physical components Components execute cooperatively. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit .

In addition, communication media typically embodies computer readable instructions, data structures, computer program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and can include any information delivery, as is well known to those of ordinary skill in the art medium. Therefore, the present invention is not limited to any particular combination of hardware and software.

The above content is a further detailed description of the embodiments of the present invention in combination with specific embodiments, and it cannot be considered that the specific implementation of the present invention is limited to these descriptions. For those of ordinary skill in the technical field of the present invention, without departing from the concept of the present invention, some simple deductions or substitutions can be made, which should be regarded as belonging to the protection scope of the present invention.

Claims

An online social networking method based on emotion recognition, characterized in that it includes:

Collect the physiological characteristic data of local chat users through sensors;

Determine a corresponding user emotion recognition result based on the physiological characteristic data;

Obtain corresponding emotional expression content according to the user emotion recognition result; wherein, the emotional expression content includes at least one of emotional expression animation and emotional expression dynamic text;

The emotional expression content is sent to the chat client used by the peer chat user in real time.
The online social networking method according to claim 1, wherein the physiological characteristic data is galvanic skin response data and electrical muscle response data;

The determining a corresponding user emotion recognition result based on the physiological characteristic data includes:

Determine corresponding emotional arousal data based on the galvanic skin response data, and determine corresponding emotional valence data based on the electrical muscle response data;

Combining the emotional arousal data and the emotional valence data, a corresponding user emotion recognition result is determined.
The online social networking method according to claim 1, wherein the physiological characteristic data is gaze tracking data;

The determining a corresponding user emotion recognition result based on the physiological characteristic data includes:

Determine corresponding attention representation data and interest representation data based on the gaze tracking data;

Combining the attention representation data and the interest representation data, a corresponding user emotion recognition result is determined.
The online social networking method according to any one of claims 1 to 3, wherein the obtaining corresponding emotional expression content according to the user's emotion recognition result comprises:

Determine the corresponding content type and content display attribute according to the user emotion recognition result;

According to the content type and the content display attribute, the corresponding emotional expression content is acquired.
An online social device based on emotion recognition, comprising:

The acquisition module is used to collect the physiological characteristic data of the local chat user through the sensor;

a determining module, for determining a corresponding user emotion recognition result based on the physiological characteristic data;

an acquisition module, configured to acquire corresponding emotional expression content according to the user emotion recognition result; wherein, the emotional expression content includes at least one of emotional expression animation and emotional expression dynamic text;

A sending module, configured to send the emotional expression content to the chat client used by the opposite chat user in real time.
The online social networking device according to claim 5, wherein the physiological characteristic data is galvanic skin response data and electrical muscle response data;

The determining module is specifically configured to: determine corresponding emotional arousal data based on the galvanic skin response data, and determine corresponding emotional valence data based on the electrical muscle response data; combine the emotional arousal data and the emotional valence data to determine the corresponding user emotion recognition results.
The online social networking device according to claim 5, wherein the physiological characteristic data is gaze tracking data;

The determining module is specifically configured to: determine corresponding attention representation data and interest representation data based on the gaze tracking data; and determine a corresponding user emotion recognition result in combination with the attention representation data and the interest representation data.
The online social networking device according to any one of claims 5 to 7, wherein the obtaining module is specifically configured to: obtain the corresponding content type and content display attribute according to the user emotion recognition result; type and the content display attribute to obtain the corresponding emotional expression content.
An electronic device, comprising: a processor, a memory and a communication bus;

The communication bus is used to realize the connection communication between the processor and the memory;

The processor is configured to execute one or more programs stored in the memory to implement the steps of the online social networking method according to any one of claims 1 to 4.
A computer-readable storage medium, characterized in that the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors, so as to realize the invention as claimed in claim 1 Steps of the online social networking method described in any one of to 4.