KR102573465B1 - Method and system for providing emotion correction during video chat - Google Patents

Abstract

A method and subsystem for altering the video and audio signals of a person participating in a video chat to yield a modified video-audio signal having an increase in entrainment between one or more of the person's perceived emotions and the person's semantic emotion are described herein. do. Such a method includes obtaining a video signal and an audio signal of a first person engaging in a video chat with a second person, determining one or more types of perceived emotion of the first person based on the video signal, and including the audio signal. and determining the semantic emotion of the first person based on The method also includes altering the video signal to increase the entrainment between at least one of the one or more types of perceived emotion of the first person and the semantic emotion of the first person.

Description

Method and system for providing emotion correction during video chat

The present disclosure generally relates to methods and systems for use during video chat, and in particular embodiments, modifies the video and audio signals of a person participating in a video chat to change the person's perceived emotions. A method and system for producing modified video and audio signals with an increase in alignment between one or more of the perceived emotions and a semantic emotion of a person.

Drivers of cars and other types of vehicles often use their smartphones or other mobile computing devices to chat with others while driving. Such chats may be voice chats or video chats. For the purposes of this discussion, voice chat refers to communication that is audio-only: two people participating in a voice chat can hear each other, but cannot see each other. In contrast, video chat refers to communication involving both audio and video between two participants in a video chat, whereby the two participants can hear and see each other. Videotelephony technology, which enables the reception and transmission of audio and video signals, can be used to conduct video chatting. Exemplary video calling products include FaceTime, available from Apple Inc., Google Duo, both available from Google LLC, and Google Hangouts, to name just a few. Hangouts), Skype available from Microsoft Corp. and WeChat available from Tencent Corp.. Indeed, a study found that 10% of drivers indicated that they used their smartphones to video chat while driving. That percentage is likely to increase in the future, especially as semi-autonomous and fully-autonomous vehicles become more common.

Road rage, the aggressive or angry behavior exhibited by the driver of a vehicle, is very common. Indeed, the survey found that the majority of drivers expressed significant anger while driving in the past year. Road rage can lead to numerous types of direct adverse effects. For example, for drivers and their occupants of vehicles, road rage can lead to altercations, assaults and collisions resulting in serious physical injury or even death. Road rage can also lead to certain non-direct adverse effects. For example, if a first person driving a first vehicle is engaged in a video chat with a second person driving a second vehicle, during which the first driver has a road rage, the first person is angry. may be transmitted to and/or otherwise distract the second person, which may increase the likelihood that the second person is involved in a conflict. As another example, if the first person driving the first vehicle is engaged in a business-related video chat with one or more other people during which the first driver has a road rage, the first person and the first person Business relationships between 100 or more other people could be spoiled or otherwise adversely affected.

According to one aspect of the present disclosure, a method includes obtaining a video signal and an audio signal of a first person engaging in a video chat with a second person, and based on the video signal, recognizing one or more types of the first person. and determining the semantic emotion of the first person based on the audio signal. The method also includes altering the video signal to increase the entrainment between at least one of the one or more types of perceived emotion of the first person and the semantic emotion of the first person.

Optionally, in any of the preceding aspects, determining one or more types of perceived emotion of the first person based on the video signal comprises determining a facial expression or body pose of the first person based on the video signal. Detecting at least one of body pose, and facial expression perceived emotion or body pose perceived emotion of the first person based on at least one of facial expression or body pose of the first person pose perceived emotion).

Optionally, in any of the preceding aspects, determining the perceived emotion of the one or more types of the first person is also based on the audio signal, a pitch of the first person's speech, a vibrato ) or inflection, and determining speech perceived emotion of the first person based on a result of the audio signal processing of the audio signal. includes doing Such a method may further include altering the audio signal to increase the entrainment between the first person's spoken perceived emotion and the first person's semantic emotion.

Optionally, in any of the preceding aspects, modifying the video signal to produce a modified video signal comprises modifying image data of the video signal corresponding to at least one of a facial expression or a body pose, and wherein the modified audio signal Modifying the audio signal to produce ? includes modifying audio data of the video signal corresponding to at least one of pitch, vibrato or tone.

Optionally, in any of the preceding aspects, the method provides the modified video signal and the modified audio signal to a subsystem associated with the second person participating in the video chat, thereby causing the second person to interact with the first person. Further comprising enabling viewing and hearing of images and audio of the first person with increased entrainment between at least one of the one or more types of perceived emotion and the semantic emotion of the first person.

Optionally, in any of the preceding aspects, determining the semantic sentiment of the first person based on the audio signal is a combination of performing natural language processing of the audio signal and natural language processing of the audio signal. and determining the semantic emotion of the first person based on the result.

Optionally, in any of the preceding aspects, determining the perceived emotion of one or more types of the first person based on the video signal comprises determining the positiveness of a facial expression of the first person based on the video signal and Using a facial circumplex model to quantify activeness or a pose circumplex model (to quantify the positivity and activeness of the body pose of the first person based on the video signal) pose circumplex model). In addition, determining the semantic emotion of the first person based on the audio signal is using a language circumplex model to quantify the positivity and aggressiveness of the first person's language based on the audio signal. include Additionally, modifying the video signal to produce a modified video signal modifies image data of the video signal to reduce the distance between the positivity and assertiveness of the facial expression of the first person and the positivity and assertiveness of the first person's language. or changing image data of the video signal to reduce a distance between positiveness and assertiveness of the body pose of the first person and positiveness and assertiveness of language of the first person.

Optionally, in any of the preceding aspects, determining the perceived emotion of the one or more types of the first person is also based on the audio signal, and quantifies positivity and assertiveness of the first person's utterance based on the audio signal. This includes using a speech circumplex model to The method may further include modifying audio data of the audio signal to produce a modified audio signal to reduce a distance between positiveness and assertiveness of the first person's utterance and positiveness and assertiveness of the first person's speech. there is.

Optionally, in any of the preceding aspects, the method provides the modified video signal and the modified audio signal to a subsystem associated with a second person engaged in a video chat, thereby causing the second person to perform one or more types of the first person. and enabling viewing and hearing of images and audio of the first person with increased entrainment between at least one of the perceived emotions of the first person and the semantic emotion of the first person.

According to one other aspect of the present disclosure, a subsystem includes one or more interfaces and one or more processors. The one or more interfaces are configured to receive video signals and audio signals of a first person participating in a video chat with a second person. The one or more processors are communicatively coupled to the one or more interfaces, determine the perceived emotion of one or more types of the first person based on the video signal, and determine the semantic emotion of the first person based on the audio signal. It is configured to determine emotions. The one or more processors are also configured to alter the video signal to increase the entrainment between at least one of the one or more types of perceived emotion of the first person and the semantic emotion of the first person. The subsystem may also include one or more cameras configured to obtain video signals and one or more microphones configured to obtain audio signals.

Optionally, in any of the preceding aspects, the one or more processors comprise one or more neural networks configured to determine a perceived emotion of the first person based on the video signal and to determine a semantic emotion of the first person based on the audio signal ( implement a neural network.

Optionally, in any of the preceding aspects, the one or more processors are configured to alter the video signal to increase the entrainment between at least one of the one or more types of perceived emotion of the first person and the semantic emotion of the first person. Implement the above neural network.

Optionally, in any of the preceding aspects, to determine one or more types of perceived emotion of the first person based on the video signal, the one or more processors may include a facial expression or body pose of the first person based on the video signal. and determine at least one of facial expression perceived emotion or body pose perceived emotion of the first person based on at least one of the facial expression or body pose of the first person.

Optionally, in any of the preceding aspects, the one or more processors perform audio signal processing of the audio signal to determine at least one of pitch, vibrato or tone of the first person's speech, based on a result of the audio signal processing. to determine the first person's utterance perceived emotion, and modify the audio signal to increase the entrainment between the first person's utterance perceived emotion and the first person's semantic emotion.

Optionally, in any of the preceding aspects, the one or more processors modify the video signal to modify image data of the video signal corresponding to at least one of a facial expression or a body pose, thereby yielding a modified video signal; , vibrato or tone, thereby modifying the audio signal to produce a modified audio signal.

Optionally, in any of the preceding aspects, the one or more processors are configured to perform natural language processing of the audio signal and determine a semantic sentiment of the first person based on a result of the natural language processing of the audio signal.

Optionally, in any of the preceding aspects, the one or more processors use the facial complex circulatory model to quantify positivity and aggressiveness of a facial expression of the first person based on the video signal; and use the pose complex cyclic model to quantify the positivity and aggressiveness of the body pose of the person, and use the language complex cyclic model to quantify the positivity and aggressiveness of the language of the first person based on the audio signal. Additionally, the one or more processors reduce a distance between the positivity and assertiveness of the first person's facial expression and the positivity and assertiveness of the first person's language, and reduce the distance between the positivity and assertiveness of the first person's body pose and the first person's positiveness and assertiveness. and change image data of the video signal to reduce the distance between positivity and assertiveness of the language of .

Optionally, in any of the preceding aspects, the one or more processors use an utterance complex cycle model to quantify positiveness and aggressiveness of utterances of the first person based on the audio signal, and and change the audio data of the audio signal to reduce the distance between the assertiveness and the positiveness and aggressiveness of the language of the first person.

Optionally, in any of the foregoing aspects, the subsystem transmits modified video signals and modified audio signals to a subsystem associated with a second person engaged in a video chat, thereby causing the second person to perform one or more actions of the first person. and a transmitter configured to enable viewing and hearing video and audio of the first person with increased entrainment between at least one of the perceived emotions of the type and the semantic emotion of the first person.

According to one other aspect of the present disclosure, a non-transitory computer readable medium stores computer instructions that, when executed by one or more processors, cause the one or more processors to perform the following steps: acquiring a video signal and an audio signal of a first person participating in a video chat with the person; determining one or more types of perceived emotion of the first person based on the video signal; determining the semantic emotion of the first person based on the audio signal; and altering the video signal to increase the entrainment between at least one of the one or more types of perceived emotion of the first person and the semantic emotion of the first person. The non-transitory computer readable medium may also store computer instructions that, when executed by the one or more processors, cause the one or more processors to perform additional steps of the methods summarized above and described in further detail below.

This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. The claimed subject matter is not limited to implementations that solve any or all of the deficiencies noted in the background.

Aspects of the present disclosure are shown by way of example and are not limited by the accompanying drawings in which like reference numerals indicate like elements.
1 shows an example system that allows a first person and a second person to engage in a video chat.
2, 3 and 4 allow first and second persons to participate in a video chat, wherein the audio and video of the first person heard and seen by the second person differs from the actual audio and video of the first person. A system is shown, according to various embodiments of the present technology, that also modifies audio and video of at least a first person to
5 shows a modification subsystem according to an embodiment of the present technology that may be used to modify audio and video signals of persons participating in a video chat.
FIG. 6 shows additional details of the emotion detector and emotion modifier of the modification subsystem introduced in FIG. 5 .
Figure 7a shows a general complex circulation model.
7b shows a facial complex circulation model.
7c shows a pose complex circulation model.
7d shows a firing complex cycle model.
8 shows how different types of perceived emotion and semantic emotion can be mapped to a complex circular model, how a distance between perceived emotion and semantic emotion can be determined, and different types of perceived emotion and how such distances can be reduced to increase entrainment between semantic emotions.
9 is a high-level illustration of how the distance between perceived emotion and semantic emotion can be used to determine whether to modify certain characteristics of video and audio signals to increase the entrainment between perceived emotion and semantic emotion. (high level) Shows a flow chart.
10 shows a high-level flow diagram used to summarize a method according to certain embodiments of the present technology.
11 shows example components of an example mobile computing device, in which embodiments of the present technology may be used.

Certain embodiments of the present technology modify the video and audio signals of a first person participating in a video chat with a second person so that, when the modified signals are played for the second person, the second person What you see and hear is different from the originally captured video and audio signals. If a first person and a second person are participating in a video chat while both are driving a vehicle, such an embodiment of the present technology is such that if the first person experiences road rage while participating in the video chat, It is possible to prevent the first person's anger from being transmitted to the second person. If a first person driving a vehicle is engaging in a business-related video chat with one or more other persons, such embodiments of the present technology may prevent the first person's upset from being witnessed by the other person(s). may be prevented from spoiling or otherwise adversely affecting the business relationship between the first person and the one or more other persons. According to certain embodiments, described in more detail below, the perceived emotion of one or more types of the first person may be determined based on the video signal (and potentially also the audio signal) of the first person; The semantic emotion of the person may be determined based on the audio signal of the first person. The video and audio signals of the first person may be subsequently modified (or altered) such that the resulting modified video and audio of the first person is more in tune with the first person's semantic emotion than the first person's perceived emotion(s). referred to as being) can be. More specifically, the video and audio signals are modified to reduce the difference between the perceived emotion of one or more types of the person and the semantic emotion of the person.

Perceived emotion, as the term is used in this document, is generally the emotional state of a first person that a second person becomes aware of using the second person's senses, eg, using the second person's sight and hearing. related to In contrast, semantic sentiment, as the term is used in this document, is generally the verbal language (also spoken language, or more succinctly) of a second person being spoken by a first person. It relates to the emotional state of a first person, which is known using the second person's understanding of (referred to as language). There are several instances where perceived emotions can actually be aligned with semantic emotions, such as if a first person is smiling and using positive body language during a conversation with a second person while telling them that they are having a great day. is the case with However, there are other cases where the perceived emotion and semantic emotion are significantly out of tune, such as if a first person says that he is having a wonderful day during a conversation with a second person, while frowning and using negative body language (e.g., looking down and seeing his arms folded). According to certain embodiments of the present technology, if a first person is frowning and has negative body language (e.g., looking down and crossing his arms) while saying that he is having a great day during a video chat with a second person, the first person The person's video shows that when the video is played for the second person, the first person's body language has changed from negative body language to positive body language, so the first person's body language is the positive spoken language he used. To be more in tune with, it is changed. Additionally, the first person's audio may also be altered, such as to change the pitch, vibrato and/or tone of the first person's voice to be more in tune with the positively spoken language he is using.

1 shows an example system that allows first and second persons to engage in a video chat. In FIG. 1 , blocks 110A and 110B use respective client computing devices (also referred to herein more generally as audio-video (A-V) subsystems 120A and 120B) Indicates first and second persons participating in the video chat. A-V subsystems 120A and 120B may collectively be referred to as A-V subsystem 120 or individually as A-V subsystem 120 . First and second persons 110A and 110B may collectively be referred to as person 110 or individually as person 110 . The A-V subsystem 120A is capable of acquiring the video signal and audio signal of the first person 110A, and the A-V subsystem 120B is capable of obtaining the video signal and audio signal of the second person 110B. do. Accordingly, each of the A-V subsystems 120 may include at least one microphone used to acquire audio signals and at least one camera used to acquire video signals. The at least one camera includes an image sensor (eg, a CMOS image sensor) that can be used to capture several two-dimensional RGB/NIR (Near Infrared) images per second (eg, 30 images per second). It may be a NIR camera. The at least one additional camera may use a structured light and/or time-of-flight (TOF) sensor, for example to recreate a 3D structure on a point cloud or the like, RGB/ It may be a depth camera that produces a depth image rather than a NIR image.

Further, the A-V subsystem 120A may be capable of reproducing video and audio of a second person (eg, 120B) for the first person 110A, and the A-V subsystem 120B may be capable of reproducing the video and audio of the second person 110B. For this purpose, it may be possible to reproduce video and audio of the first person (eg, 120A). Accordingly, each of the A-V subsystems 120 may include at least one audio-speaker used to output audible sound and at least one display used to display video images. One or both of the A-V subsystems 120A and 120B may be used in an in-cabin computer system or mobile, such as, but not limited to, a smartphone, tablet computer, notebook computer, laptop computer, or the like. ) may be a computing device. One or both, or portions of audio-video subsystems 120A and 120B may include microphones, cameras, audio speakers, and/or displays built into the vehicle, such as as part of a vehicle entertainment system. Also possible.

When first and second persons 110A and 110B are participating in a video chat using their respective A-V subsystems 120A and 120B, at least one microphone of A-V subsystem 120A is connected to the first person. 110A, and at least one camera of the A-V subsystem 120A acquires a video signal of the first person 110A. Similarly, at least one microphone of A-V subsystem 120B obtains an audio signal of second person 110B, and at least one camera of A-V subsystem 120B obtains a video signal of second person 110B. Acquire The audio and video signals of first person 110A, obtained by A-V subsystem 120A, are transmitted to A-V subsystem 120B via one or more communication networks 130 . Similarly, the audio and video signals of second person 110B, obtained by A-V subsystem 120B, are transmitted to A-V subsystem 120A via one or more communication networks 130.

Communications network(s) 130 may be any wired or wireless local area network (LAN) and/or wide area network (WAN), such as an intranet, extranet, or It may be the Internet, or a combination thereof, but is not limited thereto. The communication network 130 suffices to provide communication capability between the A-V subsystems 120, and there are other devices and systems that are optional. In some implementations, communication network(s) 130 may use the HyperText Transport Protocol (HTTP) to transport information using Transmission Control Protocol/Internet Protocol (TCP/IP). Use HTTP allows A-V subsystem 120 to access various resources available over communication network(s) 130 . However, the various implementations described herein are not limited to use of any particular protocol.

At least one audio-speaker of the A-V subsystem 120A is configured to output audible sound of a second person 110B (eg, words uttered by it), which is audible to the first person 110A. The audio signal of the person 110B is used. At least one display of A-V subsystem 120A uses the video signal of second person 110B to display a video image of second person 110B that first person 110A can view. Similarly, at least one audio-speaker of A-V subsystem 120B is configured to output audible sound of first person 110A (e.g., words uttered by it) that second person 110B can hear. For this purpose, the audio signal of the first person 110A is used. At least one display of A-V subsystem 120B uses the video signal of first person 110A to display a video image of first person 110A, which is viewable by second person 110B.

Conventionally, an unmodified version of the audio and video signals of a first person 110A (obtained by the A-V subsystem 120A) is transmitted to the A-V subsystem 120B (which is proximate to a second person 110B). used) to output and display the audio and video of the first person 110A to the second person 110B. Therefore, if a first person 110A engages in a video chat with a second person 110B, an angry facial expression (eg, furrowed forehead), an angry body pose (eg, clenched fists), and an angry voice With a (eg, high) tone, the second person 110B will see the angry facial expression and angry body pose of the first person 110A and will hear the angry tone of the first person 110A. Note that the term body pose, as used herein, also encompasses hand pose.

According to some embodiments of the present technology, first person 110A's audio and video signals are modified before being presented to A-V subsystem 120B, which is heard and viewed by second person 110B. ) results in a difference between what the first person 110A actually sees and hears. Such modifications to the audio and video signals of first person 110A may be performed by the same A-V subsystem that acquires the audio and video signals. More specifically, as shown in FIG. 2 , the A-V and correction subsystem 220A obtains audio and video signals of a first person and converts those signals to an A-V subsystem (proximate to a second person 110B) ( 120B) to modify the modified audio and video signals prior to providing them to the communications network(s) 130. Alternatively, such modifications to the audio and video signals of first person 110A may be performed by an additional subsystem separate from the A-V subsystem 120A that obtains the audio and video signals of first person 110A. can be performed For example, as shown in FIG. 3 , modification subsystem 320A may receive audio and video signals of a first person 110A, and modification subsystem 320A may transmit those signals to a second person. 110B may be modified prior to providing such signals to communication network(s) 130 that provide the modified audio and video signals to an A-V subsystem 120B proximate to 110B. Another option is for the A-V subsystem 120A (which acquires the audio and video signals of first person 110A) to modify the audio and video signals of first person 110A over one or more communication networks 130. 420A, which then, after modification subsystem 420 modifies those signals, allows modification subsystem 420 to send the modified audio and video signals of first person 110A to the communications network(s). Through 130, it can be provided to the A-V subsystem 120B that is close to the second person 110B. Other variations are also possible and within the scope of the embodiments described herein. Although not shown in FIGS. 1-4 , the video and audio signals of the second person 110B are also such that the second person's perceived emotions appear to be more in tune with the semantic emotions of the second person 110B. can be provided to a similar modification subsystem to correct .

of first person 110A, captured or otherwise obtained by AV subsystem 120A (in FIGS. 1, 3 and 4) or by A-V and correction subsystem 220A (in FIG. 2). The audio and video signals may also be referred to as the captured audio and video signals of the first person 110A. 5 shows a modification subsystem 520 that receives captured audio and video signals from AV subsystem 120A, or is part of AV and modification subsystem 220A. As shown in FIG. 5 , correction subsystem 520 includes emotion detection block 530 (which may also be referred to as emotion detector 530 ) and emotion correction block 540 (which may also be referred to as emotion corrector 540 ). )). The emotion detector 530 may detect, for example, negative, positive, and/or neutral emotions of the first person 110A. Exemplary negative emotions include, but are not limited to, anger, nervousness, distraction, and disappointment. Emotion modifier 540 may, for example, modify the audio and video signals so that the perceived emotions of one or more types of first person 110A within the modified audio and video signals are neutral or positive emotions. Exemplary neutral or positive emotions include, but are not limited to, happy, calm, alert, and joyous. Additional details of emotion detector 530 and emotion modifier 540, in accordance with specific embodiments of the present technology, are discussed below with reference to FIG. 6 .

Referring to FIG. 6 , emotion detector 530 includes face detection block 610 (also referred to as face detector 610 ) and facial expression recognition block 612 (also referred to as facial expression recognizer 612 ). are shown to contain. Emotion detector 530 is also shown as including a skeleton detection block 614 (also referred to as skeleton detector 614 ) and a pose recognition block 616 (also referred to as pose recognizer 616 ). As shown in FIG. 6 , face detector 610 and bone detector 614 are shown receiving video signal 602 . Video signal 602 may be, for example, a video signal of first person 110A captured by AV subsystem 120A, and more specifically, one or more cameras thereof. Still referring to FIG. 6 , emotion detector 530 also includes audio signal processing block 624 (also referred to as audio signal processor 624 or audio signal analyzer 624) and natural language processing block 626 (also referred to as audio signal processor 624 or audio signal analyzer 624). Referred to as natural language processor 626 or natural language analyzer 626). As shown in FIG. 6 , audio signal analyzer 624 and natural language analyzer 626 receive audio signal 622 . Audio signal 622 may be, for example, the audio signal of first person 110A captured by AV subsystem 120A, or more specifically, its microphone. Video signal 602 and audio signal 622 are assumed to be digital signals unless specifically stated otherwise. Interfaces 603 and 623 may receive video signals 602 and audio signals 622, such as from a camera and microphone, respectively, or from one or more other subsystems.

According to some embodiments, face detector 610 may detect a person's face within an image, and may also detect facial features within an image. Already developed (or future developed) computer vision techniques may be used by face detector 610 to detect such facial features. For example, a Hue-Saturation-Value (HSV) color model or some other computer vision technique may be used to detect faces in an image. A feature detection model or some other computer vision technique may be used to identify facial features, such as, but not limited to, eyes, nose, lips, chin, cheeks, eyebrows, forehead, and/or the like. Feature detection can also be used to detect wrinkles in specific facial regions, such as on the forehead, around the mouth, and/or around the eyes. In some embodiments, a person's face and its facial features may be identified using a bounding box. Some features to be identified, such as eyes on a user's face, may be included within other features, in which case the containing feature (eg, face) is first identified and the included feature (eg, each eye of a pair of eyes) ) can then be used to identify consecutive bounding boxes. In another embodiment, a single bounding box may be used to identify each distinct feature. In certain embodiments, one or more algorithm libraries are used to identify these facial features and create bounding boxes, such as the OpenCV (http://opencv.willowgarage.com/wiki/) computer vision library and/or the Dlib algorithm library ( http://dlib.net/) may be used. In certain embodiments, the bounding box need not be rectangular, but rather may be of another shape, such as but not limited to an oval. In certain embodiments, machine learning techniques, such as boosting, may be used to increase the confidence level in the detection of facial features (eg, eyes, nose, lips, etc.). More generally, the data set can be used to train a deep neural network (DNN) and/or other computer model to detect facial features from images, and the trained DNN and/or other computer model can do that. It can be used later for facial feature recognition.

Once facial features have been identified (also referred to as detected) by face detector 610, facial expression recognizer 612 can determine the facial expression of the person. Generally, the human face consists of different parts such as the chin, mouth, eyes and nose, as pointed out above. The shape, structure and size of such facial features may vary for different facial expressions. Additionally, along with a given facial expression, wrinkles at specific facial locations may change. For example, the shape of a person's eyes and mouth may be used to distinguish different facial expressions, as may wrinkles on a person's forehead and/or the like. Based at least in part on the detected facial expressions of the person, one or more types of perceived emotion of the person may be determined by the perceived emotion detector 632 in FIG. 6 . Exemplary perceived emotions that may be detected based at least in part on the detected facial expressions include, but are not limited to, happy, calm, alert, and joyous, as well as angry, nervous, distracted, and disappointed. Certain techniques for quantifying perceived emotion are described below.

Skeleton detector 614 may use a skeletal detection model or some other computer vision technique to identify human body parts and joints, such as, but not limited to, arms, hands, elbows, wrists, and/or the like. Pose analyzer 616 may be used to determine a particular pose, such as whether a person is holding the vehicle's steering wheel with both hands while driving a vehicle, or whether a character is lifting one of his arms with his hand clasped into a fist while driving a vehicle. You can detect if it is. The data set can be used to train a deep neural network (DNN) and/or other computer model to detect human poses from images, which the trained DNN and/or other computer model then recognizes the pose. can be used for

Once the human body part is detected within the image by bone detector 614, pose recognizer 616 can determine the pose of the person. Generally, the human body consists of different parts such as head, neck, torso, upper arm, elbow, forearm, wrist, hand and the like. With any pose, the overall and relative positions and orientations of those body parts may change. For example, while a person is driving a vehicle, the person will often place both of their hands on the vehicle's steering wheel, but the driver of another vehicle, for example, may cause the person to suddenly stop, swerve, and/or Made to do something similar, so the character can raise one of his arms and make a fist if he's angry. As can be seen from FIG. 6 , the detected pose can also be used to determine the perceived emotion of the person, as indicated by the line from the pose recognizer 616 to the perceived emotion detector 632 .

As pointed out above, in FIG. 6 audio signal analyzer 624 and natural language analyzer 626 receive audio signal 622 . Audio signal 622 may be, for example, the audio signal of first person 110A captured by AV subsystem 120A. The audio signal analyzer 624 may analyze the audio signal 622 to detect various characteristics of the audio signal 622 that may vary depending on the emotional state of the person. Examples of such audio characteristics include pitch, vibrato and tone. Pitch is related to the frequency of a signal, and thus can be quantified as a frequency. A change in the pitch of a person's voice is often correlated with the person's arousal state, or more generally the emotional state. For example, an increase in pitch often correlates with a state of high arousal such as anger, ecstasy or fear, while a decrease in pitch often correlates with a state of low arousal such as sadness or calm. Vibrato is a periodic modulation of the pitch (eg, fundamental frequency) of a person's voice that occurs with a given speed and depth. Vibrato is also related to jitter, which is often correlated with changes in mood. Increased fluctuations in pitch, and thus vibrato, may indicate, for example, increased happiness, anguish or fear. Tone is a rapid modification of pitch at the beginning of each utterance, which overshoots its target by several semitones but quickly decays to normal values. The use of tone leads to increased variation in pitch, which is associated with high emotional intensity and positive valence. As can be seen from FIG. 6, the result of the audio signal analysis performed by the audio signal analyzer 624, as indicated by the line from the audio signal analyzer 624 to the perceived emotion detector 632, is the It can also be used to determine perceived emotion. As can be seen from the discussion above, a given change in a particular audio characteristic can indicate either an increase in positive emotion (eg, happiness) or negative emotion (eg, anger). For example, an increase in happiness or fear can both cause an increase in pitch. However, by analyzing several voice characteristics alone or in combination with facial expressions and/or body poses, it is possible to determine a relatively accurate perceived emotion of a person.

The natural language analyzer 626 performs natural language processing (NLP) of the audio signal 622, as represented by the line from the natural language analyzer 626 to the semantic emotion detector 634: The result is used to determine the character's semantic emotion. The NLP performed by the natural language analyzer 626 may include utterance recognition to provide a textual representation of a person's utterances. There is rarely any pause between successive words in natural speech, so speech segmentation can be a subtask of speech recognition, in which a sound clip of a person is divided into several words. entails separating into Natural language analyzer 626 may be configured to recognize a single language, or several different languages, such as English, Chinese, Spanish, French, and German, to name only a few. In cases where natural language analyzer 626 is capable of performing NPL for several different languages, the output of natural language analyzer 626 may include an indication of the particular language the person is speaking.

Perceived emotion detector 632 includes one or more lookup tables to determine one or more types of perceived emotion associated with a person based on the outputs of facial expression analyzer 612, pose recognizer 616, and audio signal analyzer 624. (Look Up Table: LUT) can be used. The output of the facial expression analyzer 612 may specify one or more facial expression characteristics of the determined person based on the video signal 602 of the person, and the output of the pose recognizer 616 may specify based on the video signal 602 of the person. One or more body poses of the determined person may be designated by the above, and an output of the audio signal analyzer 624 may designate one or more audio characteristics determined based on the audio signal 622 . Instead of, or in addition to, using a LUT, the perceived emotion detector 632 may include facial expression training data, body pose training data, speech training data, and/or other perceived emotion training data. may be implemented by one or more DNNs and/or one or more other computer models trained based on emotional training data.

The semantic emotion detector 634 may use one or more look up tables (LUTs) to determine a perceived emotion associated with a person based on the output of the natural language analyzer 626 . The output of natural language analyzer 626 may designate words and sentences spoken by the person as determined based on audio signal 622 and may also indicate that language is being spoken. Instead of, or in addition to, using LUTs, semantic emotion detector 634 may be implemented by one or more DNNs and/or other computer models trained based on semantic emotion training data.

Still referring to FIG. 6 , the outputs of the perceived emotion detector 632 and the semantic emotion detector 634 are also provided to an emotion modification block 540 (which may also be referred to as an emotion modifier 540 ). is shown as Emotion modifier 540 is also shown receiving a captured video signal 602 and a captured audio signal 622 . Emotion modifier 540 is shown as including facial expression modification block 642, pose modification block 646 and audio modification block 648, which also include facial expression modifier 642, pose modifier ( 646) and audio modifier 648. As noted above, the perceived emotion detector 632 includes detected facial expressions, detected body poses (determined based on video signal 602), and detected audio features (e.g., pitch, vibrato, and tone) ( based on the audio signal 622) to determine one or more types of perceived emotion of the person. Also as pointed out above, semantic emotion detector 634 determines a person's semantic emotion based on their uttered language using NLP.

In accordance with certain embodiments of the present technology, facial expression modifier 642 determines the person's facial expression perceived emotion (as determined by perceived emotion detector 632) and semantic emotion detector 634. Modify the facial expression image data of the video signal 602 to increase the entrainment between the semantic emotion of the person (as determined by According to certain embodiments of the present technology, the pose modifier 646 is configured by the character's body pose (as determined by the perceived emotion detector 632) and the perceived emotion (as determined by the semantic emotion detector 634). The image data of the video signal 602 is modified to increase the entrainment between the semantic emotion of the person (as determined). According to certain embodiments of the present technology, the audio modifier 648 includes the person's utterances (as determined by the perceived emotion detector 632) and the perceived emotion (as determined by the semantic emotion detector 634). Modify the audio data of the audio signal 622 to increase the entrainment between the semantic emotions of the person (as described above). Emotion modifier 540 is shown outputting a modified video signal 652 and a modified audio signal 662 .

Certain embodiments of the present technology allow a character's emotions in response to and/or caused by environmental factors to be different from a character's emotions in response to and/or caused by the context of a conversation in a certain feature space. relies on the assumption that differences between emotions that respond to and/or are caused by environmental factors and emotions that respond to and/or are caused by the context of a conversation can be quantified. According to certain embodiments, the feature space used to quantify the difference between perceived and semantic emotion was originally developed by James Russell and described in Journal of Personality and Social Psychology, Vol. 39(6), Dec. It is a feature space defined by the arousal/valance circumplex model published in an article titled "A circumplex model of affect" in 1980, pages 1161-1178. The arousal/significance complex cyclical model (which may also be more concisely referred to as the complex cyclical model) proposes that emotions are distributed in a two-dimensional circular space containing the arousal and significance dimensions. Arousal corresponds to the vertical axis and significance corresponds to the horizontal axis, while the center of the circle corresponds to neutral significance and intermediate levels of arousal. In this model, emotional states can be represented at any level of significance and arousal, or at a neutral level of one or both of these factors. James Russell and Lisa Feldman Barrett later developed a modified arousal/significance complex cycle model, which they published in Journal of Personality and Social Psychology, Vol. 76(5), May 1999, pages 805-819 in an article entitled "Core affect, prototypical emotional episodes, and other things called emotion: dissecting the elephant".

According to some embodiments of the present invention, the perceived emotion detector 632 uses one or more arousal/significance complex cyclic models to determine three types of perceived emotion, respectively, based on facial expressions, body poses, and utterances. use. More specifically, in certain embodiments, a facial complex circulatory model is used to determine arousal and significance associated with a person's facial expression; A pose complex cycle model is used to determine the arousal and significance associated with a person's body pose; The utterance complex cycle model is used to define the arousal and significance associated with the utterance of a person. The significance dimension is shown on the horizontal axis and spans between positive and negative significance. Positive and negative significance (along the horizontal axis) are also known as pleasant emotion and unpleasant emotion, respectively, or more generally as positivity. The arousal dimension is shown on the vertical axis intersecting the horizontal “significance” axis and spans between activated and deactivated. Active arousal and inactive arousal (along the vertical axis) are also known as intense arousal and inactive arousal, respectively, or more generally as aggression. A general complex circulation model is illustrated in FIG. 7A, a facial complex circulation model is illustrated in FIG. 7B, a pose complex circulation model is illustrated in FIG. 7C, and a speech complex circulation model is illustrated in FIG. 7D.

According to certain embodiments of the present technology, feature vectors generated from facial expression detection, pose detection and utterance detection algorithms are input into a DNN. Facial expression detection may be performed by face detector 610 and facial expression analyzer 612, discussed above with reference to FIG. 6 . The result of facial expression detection may be one or more facial feature vectors. Pose detection may be performed by bone detector 614 and pose analyzer 616 . The result of pose detection may be one or more pose feature vectors. Speech detection may be performed by the audio signal analyzer 624. The result of utterance detection may be one or more utterance feature vectors. According to some embodiments, the feature vectors described above are concatenated together and fed into a DNN. Such a DNN can be used to implement the perceived emotion detector 632 in FIG. 6 .

According to certain embodiments of the present technology, the output of the DNN implementing the perceived emotion detector 632 is six values denoted { aro _f , val _f , aro _p , val _p , aro _s , val _s } " aro " refers to arousal and " val " refers to significance, and the subscripts f , p and s refer to face, pose and utterance, respectively. Accordingly, there are an arousal value and a significance value indicating a facial expression of a person, an arousal value and a significance value indicating a body pose of a person, and an arousal value and a significance value indicating a speech of a person. According to certain embodiments, as described in further detail below, these values are used to modify a person's facial expression, body pose, and/or speech. The terms modification and change are used interchangeably in this document.

According to certain embodiments of the present technology, in order to quantify a person's semantic emotion, a natural language processing (NLP) algorithm based on deep learning is applied. The main idea is to determine the context-dependence of emotion as a recognized utterance. In natural language processing, textual instances are often represented as vectors in a feature space. The number of features can often be as large as hundreds of thousands, and traditionally, these features have a known meaning. For example, whether an instance has a particular word previously observed in the training data, whether the word is listed as a positive/negative term in a sentiment lexicon, and so forth. By using NLP algorithms, a person's semantic emotion can be estimated. According to some embodiments, the output of the DNN implementing the semantic emotion detector 634 is two values denoted { aro _sem , val _sem }, which can be collectively represented as Emo _sem . According to certain embodiments, as described in further detail below, the semantic emotion Emo _sem of a person is used to modify image data and audio data corresponding to the person's facial expressions, body poses and/or utterances. do.

Each perceived emotion Emo ⁱ _perc denotes a point on the complex cycle model, where i = { face, pose, speech }. This allows mapping of different types of perceived emotion onto a complex circulatory model, as shown in FIG. 8 . The semantic emotion Emo _sem also marks a point on the complex cycle model, which can be mapped to the same complex cycle model as also shown in FIG. 8 . Referring to FIG. 8 , "X" labeled 802 corresponds to the perceived facial emotion of the person, "X" labeled 804 corresponds to the perceived pose emotion of the person, and "X" labeled 806 corresponds to the perceived facial emotion of the person. X" corresponds to the person's perceived utterance emotion. The location of X 802, 804 and 806 is defined by the six values discussed above, denoted { aro _f , val _f , aro _p , val _p , aro _s , val _s }. More specifically, the location of "X" labeled 802 is defined by the values aro _f and val _f , the location of "X" labeled 804 is defined by the values aro _p and val _p , and labeled 806 The position of "X" is defined by the values aro _s and val _s . Still referring to FIG. 8 , the dot labeled 808 corresponds to the semantic emotion Emo _sem of the person. The location of the point labeled 808 is the value aro _sem and val _sem .

As pointed out above, assertiveness is a measure of arousal, and positivity is a measure of significance. Any of the perceived emotions and semantic sentiment distance between can be calculated using the formula below:

perceived emotion and semantic sentiment The distance between perceived and semantic emotions indicates how closely they are tuned. For example, if the distance between a particular perceived emotion (eg, body pose) and a semantic emotion is relatively small, it indicates that the perceived emotion is substantially aligned with the semantic emotion. Conversely, if the distance between a particular perceived emotion (eg, body pose) and semantic emotion is relatively large, it indicates that the perceived emotion is not substantially aligned with the semantic emotion. According to certain embodiments, for each type of determined perceived emotion, a distance between the perceived emotion and the semantic emotion is determined. This will result in three distance values being determined, one for each of the facial expression, body pose and utterance. If the determined distance exceeds a specified distance threshold, it will be determined that the perceived emotion is substantially out of sync with the semantic emotion, and in response to that determination, the individual's characteristics (e.g., facial expression, body pose or utterance) is modified to increase the alignment between perceived emotion and semantic emotion. By way of further example, the determined distance between a facial perceived emotion (represented by “X” labeled 802 in FIG. 8 ) and a semantic emotion (represented by a dot labeled 808 in FIG. 8 ) is a specified distance If greater than the threshold, the face image data of the video signal is modified to yield a modified video signal in which the face perceived emotion is more in tune with the semantic emotion. Conversely, if the determined distance between the face perceived emotion and the semantic emotion is smaller than a specified distance threshold (also referred to as within the distance threshold), the face image data of the video signal is not modified. This determining the respective distance and comparing the determined distance to the distance threshold is also determined for the body pose as well as the utterance. The result of such comparison is used to determine whether to modify the body pose data of the video signal and/or speech data of the audio signal.

The distance determination and comparison described above is summarized in the flowchart of FIG. 9 . Referring to FIG. 9 , in step 902, the distance between one of the perceived emotions (face, pose, utterance) and the semantic emotion is determined, and more specifically calculated, eg, using the equations discussed above . In step 904, the calculated distance is compared to a distance threshold, and in step 906, a determination is made whether the calculated distance is within (i.e., less than) the distance threshold. If the calculated distance is not within the distance threshold (i.e., the answer to the determination at step 906 is No), flow goes to step 908 and flow proceeds to step 910. Before doing so, the relevant signal or portion thereof is modified in step 908 . If the calculated distance is within the distance threshold (i.e., the answer to the determination at step 906 is Yes), the flow proceeds to step 910 without any modification to the associated signal or portion thereof. Goes. The steps outlined above can be performed for each of the different types of perceived emotion including face, pose and speech.

According to certain embodiments of the present technology, the first person's video and audio are modified by creating a composite image/audio to replace its original version. More specifically, the first person's original acquired video and audio signals, when viewed or heard by a second person (or several other persons), have a modified video that has a perceived emotion that is more in tune with the first person's semantic emotion. and modified to yield an audio signal. According to certain embodiments, the perceived emotion of the generated image/audio should approximate semantic emotion as closely as possible.

Referring again to FIG. 6 , emotion modifier 540, which may also be more specifically referred to as perceived emotion modifier 540, includes facial expression modifier 642, pose modifier 646, and audio modifier ( 648). Modifiers 642, 646 and 648, which may also be referred to as modules, each use an algorithm to create a composite image or composite audio by modifying certain data within the captured audio and video signals. As a simple example, suppose that it is determined that the first person's semantic emotion is happy, her face-recognized emotion is nervous, her pose-recognized emotion is puzzled, and her utterance-recognized emotion is nervous. Using an embodiment of the present technology, facial image data is modified such that the person's facial expression is happy (rather than nervous), pose image data is modified such that the person's body pose is happy (rather than embarrassed), and audio data is modified The character's speech is modified to be happy (rather than nervous). Such modifications should be done in real-time or near real-time so that there is no appreciable lag in the video chat.

As pointed out above, one or more DNNs and/or other computer models may be used to modify the captured video and audio signals to produce modified video and audio signals. According to a particular embodiment, a Generative Adversarial Network (GAN) is used to perform such modifications. A GAN is a deep neural network architecture that includes two neural networks that pit one against the other in competition (hence the use of the term “adversarial”): a generative neural network and a discriminative neural network. Thus, generative and discriminant neural networks can be considered as subnetworks of GAN neural networks. A generative neural network generates candidates while a discriminant neural network evaluates them. Competition works in terms of data distribution. A generative neural network can learn to map from a latent space to a data distribution of interest, whereas a discriminant neural network can distinguish candidates produced by a generative neural network from true data distributions. The training goal of a generative network is to increase the error rate of the discriminant network (i.e. to "deceive" the discriminator network by yielding new candidates that the discriminator thinks are not synthesized (part of the true data distribution). " thing) can be. A known dataset serves as the initial training data for the discriminant neural network. Training a discriminant neural network may involve presenting it with samples from a training dataset until it achieves acceptable accuracy. A generative neural network can be trained based on whether it succeeds in fooling a discriminant neural network. A generative neural network may be seeded with randomized inputs sampled from a predefined latent space (eg, a multivariate normal distribution). Candidates synthesized by the generative neural network can then be evaluated by the discriminant neural network. Backpropagation can be applied in both networks so that the generative network yields better images, while the discriminant network becomes more adept at flagging synthetic images. A generative neural network can be, for example, a deconvolutional neural network, and a discriminant neural network can be, for example, a convolutional neural network. A GAN must be trained before it can be used to modify signals during video chat.

Referring back to FIG. 6 , the facial expression modifier 642 may be implemented by a GAN. More specifically, GANs generate modified video signals that can be used to display realistic images of a person, where the image has been modified to make the person's face and pose perceived emotions more in sync with the person's semantic emotions. It can be used to modify image data within a video signal to produce GANs can also be used to modify audio signals so that the perceived emotion of a person's speech is more in tune with the semantic emotion of the person. In certain embodiments, StarGAN may be used to perform image and/or audio modifications. An article titled "StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation" by Y. Choi et al, CVPR, 2018 describes how StarGAN can be used to modify people's facial expressions in a realistic way. discuss what happened The use of additional and/or alternative types of neural networks and/or other types of computer models is also within the scope of the embodiments described herein.

Still referring to FIG. 6 , a GAN may also be used to implement pose modifier 646 . Alternatively, a pretrained visual generator model may be used to implement pose modifier 646. As shown in FIG. 6 , the original video signal 602 is provided to a skeleton detector 614 . Original video signal 602 may also be referred to as original image stream 602 . Skeleton detector 614 extracts bone information from the original image stream. Bone information can be represented as a vector X, which stores all joint positions within the bone. According to an embodiment, vector X is combined with a semantic emotion signal represented by vector e. These two vectors can be concatenated to be vector X and used as input for a pretrained visual generator model. Pretrained visual generator models can be implemented as, but not limited to, convolutional layers, maxpooling layers, deconvolutional layers, and batch normalization layers, for example. The output of the pretrained visual generator model can be used to generate a modified video signal 652 that includes a modified body pose that is more in tune with the semantic emotion.

Still referring to FIG. 6 , as already pointed out above, the original audio signal 622 is shown as being provided to an audio signal analyzer 624 . Characteristics of the audio signal that may be modified include, but are not limited to, pitch, vibrato, and tone, to bring the perceived emotion corresponding to a person's utterance to more in sync with their semantic emotion. More specifically, the pitch can be shifted, which means multiplying the pitch of the original speech signal by a factor α. Increased pitch (α > 1) often correlates with highly aroused states such as happiness, whereas decreased pitch (α < 1) correlates with low significance such as sadness. Vibrato is a periodic modulation of the pitch (fundamental frequency) of the voice that occurs with a given rate and depth. Vibrato, also related to hunting, is frequently reported to be correlated with high arousal and is an important marker of emotion even as a single vowel. Vibrato can be modified to change the perceived emotion corresponding to the utterance. Tone is a rapid modification (e.g., ˜500 ms) of pitch at the beginning of each utterance that misses its target by a few semitones but quickly decays to normal values. The use of tone leads to increased variation in pitch, which is associated with high emotional intensity and positive significance. Tone can also be modified to change the perceived emotion corresponding to the utterance. The audio signal may also be filtered to alter the perceived emotion corresponding to the utterance, filtering being the process of emphasizing or attenuating the energy contribution of certain regions of the frequency spectrum. For example, high arousal emotions tend to be associated with increased high-frequency energy, making speech sound sharper and clearer. In cases where a person's semantic emotion corresponds to less activated arousal than the person's perceived emotion corresponding to the utterance, the high-frequency energy in the audio signal is filtered using filtering to make the perceived emotion more in tune with the semantic emotion. may be weakened. The emotional tone of the modified audio signal should be recognizable, and the voice should sound natural and should not be perceived as synthetic. As pointed out above, the terms modification and change are used interchangeably in this document.

One or more processors may be used to implement the neural network described above. Where multiple processors are used, they may be collocated, widely distributed, or a combination thereof.

The high-level flow diagram of FIG. 10 will now be used to summarize a method according to certain embodiments of the present technology. Referring to FIG. 10 , step 1002 involves obtaining video and audio signals of a first person participating in a video chat with a second person. Referring again to FIGS. 1-4 , step 1002 is performed by an A-V subsystem (e.g., 120A), or more specifically, one or more cameras and one or more microphones of the A-V subsystem, or any other subsystem or system. can be performed

Referring again to FIG. 10 , step 1004 involves determining one or more types of perceived emotion of the first person based on the video signal. Step 1006 involves determining the semantic sentiment of the first person based on the audio signal. Types of perceived emotion that may be determined in step 1004 include facial expression perceived emotion, body pose perceived emotion, and utterance perceived emotion, as described above.

Referring also briefly to FIGS. 5 and 6 , various types of perceived emotions can be determined, such as by emotion detector 530 , or more specifically by its perceived emotion detector 632 . More specifically, based on the video signal obtained in step 1002, the facial expression and body pose of the first person may be determined, and based on this, the perceived emotion of the facial expression and the perceived body pose of the first person may be determined. can be judged. Additionally, audio signal processing of the audio signal obtained in step 1002 may be performed to determine at least one of pitch, vibrato or tone of speech of the first person, and based on a result of the audio signal processing, the first person The utterance of the perceived emotion can be determined. Additional and/or alternative modifications are also possible and within the scope of the embodiments described herein.

According to a particular embodiment, in step 1004, the facial complex circulation model is used to quantify the positivity and aggressiveness of the facial expression of the first person based on the video signal, and the body pose of the first person based on the video signal. A pose complex cyclic model is used to quantify the positivity and aggressiveness of , and a utterance complex cyclic model is used to quantify the positivity and aggressiveness of the first person's utterance based on the audio signal.

The semantic emotion determined in step 1006 may be determined, for example, by emotion detector 530 or, more specifically, by its semantic emotion detector 634 . As described in further detail above, step 1006 may involve performing natural language processing of the audio signal and determining the semantic sentiment of the first person based on a result of the natural language processing of the audio signal. there is. According to a particular embodiment, in step 1006, the first person's semantic emotion is determined based on the audio signal using a language complex circular model to quantify the positivity and assertiveness of the first person's language based on the audio signal. is judged as

Referring back to FIG. 10 , step 1008 involves altering the video and audio signals to increase the entrainment between at least one of the one or more types of perceived emotion of the first person and the semantic emotion of the first person. do. As described in detail above, one or more computer implemented neural networks may be used to perform step 1008 . Alternatively or additionally, other types of computer implemented models may be used to perform step 1008 . Step 1008 may involve modifying facial expression and body pose of image data included in the video signal, and also modifying at least one of pitch, vibrato or tone of audio data included in the audio signal.

According to a particular embodiment, step 1008 involves altering image data included in the video signal to reduce the distance between the positivity and assertiveness of the first person's facial expression and the positivity and assertiveness of the first person's language. accompanies Step 1008 may also involve altering image data included in the video signal to reduce the distance between the positivity and assertiveness of the first person's body pose and the positivity and assertiveness of the first person's speech. Further, step 1008 may also involve modifying audio data included in the audio signal to reduce the distance between the positivity and assertiveness of the first person's utterance and the positivity and assertiveness of the first person's speech. .

Still referring to FIG. 10 , step 1010 involves providing (eg, a device) the modified video signal and the modified audio signal to a subsystem (eg, device) associated with (eg, in close proximity to) a second person participating in a video chat. . Allows you to see and hear images and audio.

For example, with reference to FIGS. 9 and 10 , the method described above can be performed at least in part on a mobile computing or in-cabin computer system, such as, but not limited to, a smartphone, tablet computer, notebook computer, laptop computer, or the like. can be performed by the device. The steps of such a method may be performed solely by a mobile computing device, or by a mobile computing device communicating over a network with one or more servers through one or more communication networks. 11 shows example components of an example mobile computing device, in which embodiments of the present technology may be used. Such a mobile computing device may be used, for example, but not limited to, to implement an A-V subsystem (eg, 120A or 220A in FIGS. 1-4).

11 shows an exemplary mobile computing device 1102 whereby embodiments of the technology described herein may be used. The mobile computing device 1102 may be a smartphone such as, but not limited to, an iPhone(TM), Blackberry(TM), Android(TM) based or Windows(TM) based smartphone. The mobile computing device 1102 may alternatively be a tablet computing device such as, but not limited to, an iPad(TM), an Android(TM) based or a Windows(TM) based tablet. For another example, mobile computing device 1102 may be an iPod Touch(TM) or the like.

Referring to the block diagram of FIG. 11 , mobile computing device 1102 includes camera 1104, accelerometer 1106, magnetometer 1108, gyroscope 1110, microphone 1112, (which may or may not be a touch screen display) display 1114 (which may not be present), processor 1116, memory 1118, transceiver 1120, speaker 1122 and drive unit 1124. Each of these elements is shown connected to a bus 1128, which allows the various components to communicate with each other and transfer data from one element to another. It is also possible that some of the elements can communicate with each other without using the bus 1128.

Camera 1104 may be used to acquire a video signal comprising an image of a person using mobile computing device 1102 . The microphone 1112 may be used to produce an audio signal representative of what the person using the mobile computing device 1102 is saying.

Accelerometer 1106 can be used to measure linear acceleration relative to a frame of reference, and thus to detect the angle of mobile device 1102 relative to the horizontal or ground, as well as for mobile computing purposes. It can be used to detect movement of device 1102. Magnetometer 1108 can be used as a compass to determine the direction of magnetic north and bearing with respect to magnetic north. The gyroscope 1110 can be used to detect both vertical and horizontal orientations of the mobile computing device 1102 and, along with the accelerometer 1106 and magnetometer 1108, can be used to determine the orientation of the mobile computing device 1102. It can be used to obtain accurate information. It is also possible for the mobile computing device 1102 to include additional sensor elements, such as, but not limited to, an ambient light sensor and/or a proximity sensor.

Display 1114, which may or may not be a touch screen type display, will be used as a user interface to visually display items (eg, images, options, commands, etc.) to the user and to accept input from the user. can Display 1114 can also be used to allow a user of mobile computing device 1102 to engage in a video chat. Mobile computing device 1102 may also include additional elements, such as keys, buttons, track-pads, trackballs, or the like, that accept input from a user. .

Memory 1118 may be used, but is not limited to, to store images captured using camera 1104, as well as software and/or firmware controlling mobile computing device 1102. A variety of different types of memory may be included in the mobile computing device 1102, including non-volatile and volatile memory. Drive unit 1124 , such as but not limited to a hard drive, may also be used to store images captured using camera 1104 as well as software controlling mobile computing device 1102 . It can be used for, but is not limited to. Memory 1118 and disk unit 1124 may be a machine readable medium having stored thereon one or more sets of executable instructions (e.g., apps) that embody one or more of the methodologies and/or functions described herein. can include Instead of, or in addition to, the drive unit 1124, the mobile computing device includes a solid-state storage device, such as flash memory or any form of non-volatile memory. may include The term "machine-readable medium" as used herein shall be taken to include all forms of storage media, whether as a single medium or as multiple media, in any form: e.g., centralized or distributed databases and/or associated caches and servers; One or more storage devices, such as storage devices (including, for example, magnetic and optical drives and storage mechanisms), and memory devices or modules (whether main memory, cache storage (whether internal or external to the processor), or buffers) one or more instances. The term “machine readable medium” or “computer readable medium” refers to any type capable of storing or encoding a sequence of instructions for execution by a machine and causing the machine to perform any of the methodologies. will be taken to include tangible, non-transitory media. The term "non-transitory medium" includes all forms of storage drives (optical, magnetic, etc.) and all forms of memory devices (e.g., DRAM, Flash (of any storage design), SRAM, MRAM, phase change, etc.). Of course, it explicitly includes any other structure designed to store any type of information for later retrieval.

Transceiver 1120 coupled to antenna 1126 may be used to transmit and receive data wirelessly using, for example, Wi-Fi, cellular communications, or mobile satellite communications. The mobile computing device 1102 may also be capable of performing wireless communications using Bluetooth and/or other wireless technologies. It is also possible that the mobile computing device 1102 includes different types of transceivers and/or different types of antennas. The transceiver 1120 may include a transmitter and a receiver.

Speaker 1122 is used to enable mobile computing device 1102 to operate as a mobile phone, as well as to provide auditory commands, feedback and/or indicators to a user, and to play recordings (eg, music recordings). can be used The speaker 1122 can also be used to allow a user of the mobile computing device 1102 to engage in a video chat.

Processor 1116 may be used to control various other elements of mobile computing device 1102, such as under the control of software and/or firmware stored in memory 1118 and/or drive unit 1124. It is also possible to have multiple processors 1116, such as a Central Processing Unit (CPU) and a Graphics Processing Unit (GPU). Processor(s) 1116 may provide computer instructions (stored on non-transitory computer readable media) to cause the processor(s) to perform steps used to implement embodiments of the technology described herein. can run

Certain embodiments of the subject technology described herein may be implemented using hardware, software, or a combination of both hardware and software. The software used is stored on one or more of the processor readable storage devices described above to program one or more of the processors to perform the functions described herein. Processor readable storage devices may include computer readable media such as volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may include computer readable storage media and communication media. A computer readable storage medium may be implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer readable storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device, or any other medium that can be used to store desired information and can be accessed by a computer. The computer readable medium or media does not contain a propagated, modulated or transitory signal.

Communication media typically embodies computer readable instructions, data structures, program modules or other data in a propagated, modulated or transitory data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a way as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as RF and other wireless media. Combinations of any of the above are also included within the scope of computer readable media.

In alternative embodiments, some or all of the software may be replaced by dedicated hardware logic components. Exemplary types of hardware logic components that may be used include, for example, without limitation, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standards Application-Specific Standard Product (ASSP), System-On-a-Chip system (SOC), Complex Programmable Logic Device (CPLD), special purpose computer Include etc. In one embodiment, software (stored on a storage device) implementing one or more embodiments is used to program the one or more processors. One or more processors may be in communication with one or more computer readable media/storage devices, peripherals and/or communication interfaces.

It is understood that this subject matter may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this subject matter will be thorough and exhaustive, and will fully convey the disclosure to those skilled in the art. Indeed, this subject matter is intended to cover alternatives, modifications and equivalents of these embodiments, included within the scope and spirit of subject matter as defined by the appended claims. Further, in the following detailed description of the subject matter, numerous specific details are set forth in order to provide a thorough understanding of the subject matter. However, it will be clear to those skilled in the art that the subject matter may be practiced without such specific details.

Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be appreciated that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks within the flowchart illustrations and/or block diagrams, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing device to produce a machine, so that the processor of the computer or other programmable instruction execution device The executed instructions create a mechanism for implementing the function/behavior specified within the flow diagram and/or block diagram block or blocks.

The description of this disclosure has been presented for purposes of illustration and description, but is not intended to be exhaustive or to limit the disclosure to the form disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of this disclosure. Aspects of the disclosure in this document have been selected and described in order to best explain the principles and practical applications of the disclosure, and to enable others skilled in the art to understand the disclosure with various modifications suited to the particular use contemplated. It became.

The present disclosure has been described in conjunction with various embodiments. However, other variations and modifications to the disclosed embodiments may be understood and result from a study of the drawings, disclosure and appended claims, and such variations and modifications should be construed as being encompassed by the appended claims. In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “an” or “an” does not exclude a plurality.

For the purposes of this document, it should be noted that the scale of various features depicted in the drawings may not necessarily be drawn to scale.

For purposes of this document, references in the specification to “one embodiment,” “one embodiment,” “some embodiments,” or “another embodiment” may be used to describe a different embodiment or the same embodiment. can

For the purposes of this document, a linkage may be a direct linkage or an indirect linkage (eg, through one or more other parts). In some cases, when an element is referred to as being connected or coupled to another element, the element may be directly connected to the other element or indirectly connected to the other element through an intervening element. When an element is referred to as being directly connected to another element, there are no intervening elements between that element and the other element. Two devices are "in communication" if they are directly or indirectly connected such that electronic signals can be communicated between them.

For the purposes of this document, the term "based on" may be read as "based at least in part on."

For the purposes of this document, without further context, the use of numerical terms such as "first" object, "second" object, and "third" object may not imply an ordering of objects, but instead identify different objects. can be used for identification purposes.

The foregoing detailed description has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the claimed subject matter in this document to the precise form(s) disclosed. Many modifications and variations are possible in light of the above teaching. The described embodiments are intended to best explain the principles of the disclosed technology and its practical applications, thereby enabling others skilled in the art to best utilize the technology in various embodiments and with various modifications suited to the particular use contemplated. selected and described for It is intended that the scope be defined by the claims appended hereto.

Although subject matter has been described in terms specific to structural features and/or methodological acts, it is to be understood that subject matter defined in appended claims is not necessarily limited to the particular features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims (26)

obtaining a video signal and an audio signal of a first person participating in a video chat with a second person;
determining one or more types of perceived emotion of the first person based on the video signal;
determining a semantic emotion of the first person based on the audio signal;
modifying the video signal to increase alignment between at least one of the one or more types of perceived emotion of the first person and the semantic emotion of the first person.
method. According to claim 1,
Determining the one or more types of perceived emotion of the first person based on the video signal comprises:
detecting a facial expression of the first person based on the video signal;
determining a facial expression perceived emotion of the first person based on the facial expression of the first person; or
detecting a body pose of the first person based on the video signal;
determining a body pose perceived emotion of the first person based on the body pose of the first person; or
detecting a facial expression and body pose of the first person based on the video signal;
Determining a facial expression-recognized emotion and a body pose-recognized emotion of the first person based on the facial expression and the body pose of the first person,
method. According to claim 2,
Determining the one or more types of perceived emotion of the first person is also based on the audio signal, the pitch, vibrato or inflection of the first person's speech. performing audio signal processing of the audio signal to determine at least one of; and determining speech perceived emotion of the first person based on a result of the audio signal processing of the audio signal. contains steps,
the method further comprising modifying the audio signal to increase entrainment between the utterance perceived emotion of the first person and the semantic emotion of the first person;
method. According to claim 3,
modifying the audio signal to produce a modified audio signal comprises modifying audio data of the video signal corresponding to at least one of the pitch, vibrato or tone;
The method,
further comprising providing the modified audio signal to a subsystem associated with the second person participating in the video chat.
method. According to claim 1,
modifying the video signal to produce a modified video signal comprises modifying image data of the video signal corresponding to at least one of a facial expression or a body pose;
The method,
providing the modified video signal to a subsystem associated with the second person participating in the video chat.
method. According to claim 1,
Determining the semantic emotion of the first person based on the audio signal comprises:
performing natural language processing of the audio signal;
determining the semantic emotion of the first person based on a result of the natural language processing of the audio signal;
method. According to claim 1,
determining one or more types of perceived emotion of the first person based on the video signal;
Using a facial circumplex model to quantify positivity and aggressiveness of a facial expression of the first person based on the video signal; or
At least one of using a pose circumplex model to quantify positiveness and aggressiveness of a body pose of the first person based on the video signal;
The step of determining the semantic emotion of the first person based on the audio signal comprises a language circumplex model to quantify the positivity and activeness of the language of the first person based on the audio signal. Including the step of using,
Modifying the video signal to yield a modified video signal comprises:
modifying image data of the video signal to reduce a distance between the positivity and the aggressiveness of the facial expression of the first person and the positivity and the aggressiveness of the language of the first person; or
changing at least one of image data of the video signal to reduce a distance between the positivity and the aggressiveness of the body pose of the first person and the positivity and the aggressiveness of the language of the first person; including,
method. According to claim 7,
The step of determining the one or more types of perceived emotion of the first person is also based on the audio signal, and based on the audio signal, the utterance complex cycle to quantify the positivity and aggressiveness of the utterance of the first person. Using a speech circumplex model,
The method is configured to reduce a distance between the positivity and assertiveness of the utterance of the first person and the positivity and the aggressiveness of the language of the first person, to calculate a modified audio signal, the audio of the audio signal. Further comprising changing the data,
method. According to claim 8,
providing the modified video signal and the modified audio signal to a subsystem associated with the second person participating in the video chat.
method. one or more interfaces configured to receive video signals and audio signals of a first person participating in a video chat with a second person;
communicatively coupled to the one or more interfaces;
determine one or more types of perceived emotion of the first person based on the video signal;
determine the semantic emotion of the first person based on the audio signal;
one or more processors configured to modify the video signal to increase entrainment between at least one of the one or more types of perceived emotion of the first person and the semantic emotion of the first person.
subsystem. According to claim 10,
one or more cameras configured to acquire the video signal;
Further comprising one or more microphones configured to acquire the audio signal.
subsystem. According to claim 10,
the one or more processors comprising one or more neural networks configured to determine the one or more types of perceived emotion of the first person based on the video signal and the semantic emotion of the first person based on the audio signal; to implement,
subsystem. According to claim 10,
wherein the one or more processors implement one or more neural networks configured to modify the video signal to increase entrainment between at least one of the one or more types of perceived emotion of the first person and the semantic emotion of the first person. ,
subsystem. According to claim 10,
To determine the one or more types of perceived emotion of the first person based on the video signal, the one or more processors comprising:
detecting a facial expression of the first person based on the video signal;
determine a facial expression perceived emotion of the first person based on the facial expression of the first person; or
detecting a body pose of the first person based on the video signal;
determine a body pose perceived emotion of the first person based on the body pose of the first person; or
detecting a facial expression and a body pose of the first person based on the video signal;
Determine a facial expression perceived emotion and a body pose perceived emotion of the first person based on the facial expression and the body pose of the first person.
subsystem. According to claim 14,
The one or more processors also
performing audio signal processing on the audio signal to determine at least one of pitch, vibrato, or tone of speech of the first person, and determining a perceived emotion of speech of the first person based on a result of the audio signal processing; do,
Modify the audio signal to increase entrainment between the utterance perceived emotion of the first person and the semantic emotion of the first person.
subsystem. According to claim 15,
The one or more processors,
Modify the audio signal to produce a modified audio signal by modifying audio data of the audio signal corresponding to at least one of the pitch, vibrato or tone.
subsystem. According to claim 10,
The one or more processors,
modifying the video signal to produce a modified video signal by modifying image data of the video signal corresponding to at least one of a facial expression or a body pose;
subsystem. According to claim 10,
The one or more processors,
performing natural language processing of the audio signal;
configured to determine the semantic emotion of the first person based on a result of the natural language processing of the audio signal;
subsystem. According to claim 10,
The one or more processors,
A facial complex circulation model is used to quantify the positivity and aggressiveness of the facial expression of the first person based on the video signal, or
configured to use a pose complex cyclic model to quantify positiveness and aggressiveness of a body pose of the first person based on the video signal;
The one or more processors,
Using a language complex circulation model to quantify the positivity and aggressiveness of the language of the first person based on the audio signal;
a distance between the positivity and the aggressiveness of the facial expression of the first person and the positivity and the aggressiveness of the language of the first person, or the positivity and the aggressiveness of the body pose of the first person and change image data of the video signal to reduce at least one of a distance between the positivity and the assertiveness of the language of the first person.
subsystem. According to claim 19,
The one or more processors also
Based on the audio signal, a utterance complex cycle model is used to quantify the positivity and aggressiveness of the first person's utterance;
Modifying audio data of the audio signal to reduce a distance between the positivity and assertiveness of the utterance of the first person and the positivity and the aggressiveness of the language of the first person,
subsystem. According to claim 20,
a transmitter configured to transmit a modified video signal and a modified audio signal to a subsystem associated with the second person participating in the video chat.
subsystem. A computer readable storage medium storing computer instructions which, when executed by one or more processors, cause the one or more processors to perform a method according to any one of claims 1 to 9. delete delete delete delete

Images