WO2020125493A1

WO2020125493A1 - Wearable augmented-reality remote video system and video call method

Info

Publication number: WO2020125493A1
Application number: PCT/CN2019/124331
Authority: WO
Inventors: 王鹏
Original assignee: 中国科学院深圳先进技术研究院
Priority date: 2018-12-17
Filing date: 2019-12-10
Publication date: 2020-06-25
Also published as: CN109803109B; CN109803109A

Abstract

The present application is applicable to the technical field of wearable devices, and provides a wearable augmented-reality remote video system and a video call method. The system comprises an augmented-reality smart glasses system, a facial expression recognition system and a facial expression construction system, wherein the augmented-reality smart glasses system is used for capturing the current environment information of a user, and constructing, according to the current environment information, a three-dimensional geographic image of the current environment information of the user; the facial expression recognition system is used for recognizing the current facial expression and current emotion of the user; and the facial expression construction system is used for capturing and constructing a three-dimensional virtual facial expression of the user. A holographic video call is realized by means of recognizing the current facial expression and current emotion of the user, and updating the three-dimensional virtual facial expression according to the current facial expression and current emotion of the user, and by means of constructing the three-dimensional geographic image of an environment where the user is located as the background of a video call, and projecting an image of the updated three-dimensional virtual facial expression and the three-dimensional geographic image into a display of an opposite call end.

Description

Wearable augmented reality remote video system and video call method

Technical field

The invention belongs to the technical field of wearable devices, and particularly relates to a wearable augmented reality remote video system and a video call method.

Background technique

With the continuous development of mobile communication technology, mobile terminals now have the function of remote video calling. The user selects the object of the remote video call, after dialing the video call of the other party, through the front camera of the mobile terminal during the chat process The facial expressions of the user can be obtained, and the facial expressions of the other party can be displayed through the display screen of the mobile terminal, so the face-to-face chat experience can be enjoyed. In order to enable the user to see the other party and the other party's facial expressions in real time during the call, the user needs to hold the mobile terminal and face the screen. To enable the user to see the user and the user's facial expressions in real time, the user needs to hold the mobile terminal and make the mobile terminal 'S camera is aimed at the user's face.

In summary, at present, when a user performs a remote video call, it is necessary to hold the mobile terminal and make the mobile terminal face the user's face to make an effective video call.

technical problem

In view of this, embodiments of the present invention provide a wearable augmented reality remote video system and a video call method to solve the problem that the current user needs to hold the mobile terminal and make the mobile terminal face the user's face in order to perform effective video when making a remote video call The problem with the call.

Technical solution

The first aspect of the present invention provides a wearable augmented reality remote video system, which includes an augmented reality smart glasses system, a facial expression recognition system, and a facial expression construction system;

The augmented reality smart glasses system and the facial expression recognition system are set in the augmented reality smart glasses, and the facial expression construction system is set in the wearable smart bracelet; the augmented reality smart glasses system and the facial expression recognition system are respectively A communication connection with the facial expression construction system, and a communication connection with the facial expression recognition system and the facial expression construction system;

The augmented reality smart glasses system is used to capture the current environment information of the user and construct a three-dimensional geographic image of the current environment information of the user according to the current environment, and project the three-dimensional geographic image to the augmented reality display of the opposite end of the call;

The facial expression recognition system is used to recognize the user's current facial expression and current emotion;

The facial expression construction system is used to capture and construct a three-dimensional virtual facial expression of a user.

A second aspect of the present invention provides a video call method, including:

Capture and construct the user's three-dimensional virtual facial expression image through the facial expression construction system;

Identifying the user's current facial expression and current emotion through the facial expression recognition system;

The augmented reality smart glasses system is used to capture the current environment information of the user and construct a three-dimensional geographic image of the current environment information of the user according to the current environment;

Updating the three-dimensional virtual facial expression image based on the user's current facial expression and current emotion recognized by the facial expression recognition system, and fusing the updated three-dimensional virtual facial expression image with the three-dimensional geographic image to generate virtual video data;

Project the virtual video data into the augmented reality display at the opposite end of the call.

Beneficial effect

The present invention provides a wearable augmented reality remote video system and a video call method, by constructing a three-dimensional virtual facial expression image, and then identifying the user's current facial expression and current emotion based on the collected physiological signals, thereby driving the facial expression construction system to The user's current facial expressions and current emotions update the three-dimensional virtual facial expressions, and by constructing a three-dimensional geographic image of the user's environment as the background of the video call, the updated three-dimensional virtual facial expression image and the three-dimensional geographic image are projected to the opposite end of the call The display realizes a holographic and live video call with the opposite end of the call, which effectively solves the problem that the current user needs to hold the mobile terminal and make the mobile terminal face the user's face to make an effective video call when making a remote video call.

BRIEF DESCRIPTION

In order to more clearly explain the technical solutions in the embodiments of the present invention, the drawings required in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only for the invention. In some embodiments, for those of ordinary skill in the art, without paying creative labor, other drawings may be obtained based on these drawings.

1 is a schematic structural diagram of a wearable augmented reality remote video system according to Embodiment 1 of the present invention;

2 is a schematic structural diagram of an augmented reality smart glasses system 110 of a wearable augmented reality remote video system according to Embodiment 1 of the present invention;

FIG. 3a is a schematic diagram of a facial expression recognition system 120 of a wearable augmented reality remote video system provided in Embodiment 1 of the present invention on augmented reality smart glasses;

FIG. 3b is a schematic diagram of the distribution of the collection positions of the face corresponding to the facial expression recognition system 120 of the wearable augmented reality remote video system provided in Embodiment 1 of the present invention;

FIG. 4 is a schematic structural diagram of a facial expression construction system 130 of a wearable augmented reality remote video system according to Embodiment 1 of the present invention;

5 is a schematic diagram of an implementation process of a video call method according to Embodiment 2 of the present invention;

6 is a schematic flowchart of an implementation process corresponding to step S101 of Embodiment 2 provided by Embodiment 3 of the present invention;

7 is a schematic flowchart of an implementation process corresponding to step S102 of Embodiment 2 provided by Embodiment 4 of the present invention;

8 is a schematic structural diagram of a terminal device according to Embodiment 5 of the present invention.

Embodiments of the invention

In the following description, for the purpose of illustration rather than limitation, specific details such as specific system structures and technologies are proposed to thoroughly understand the embodiments of the present invention. However, those skilled in the art should understand that the present invention can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, systems, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary details.

In order to explain the technical solution of the present invention, the following will be described with specific embodiments.

Example one:

As shown in FIG. 1, this embodiment provides a wearable augmented reality remote video system 10, which includes an augmented reality smart glasses system 110, a facial expression recognition system 120, and a facial expression construction system 130;

The augmented reality smart glasses system 110 and the facial expression recognition system 120 are set in the augmented reality smart glasses, and the facial expression construction system 130 is set in the wearable smart bracelet; the augmented reality smart glasses system 110 is respectively connected with the facial expression recognition system 120 and the facial expression The construction system 130 is in communication connection, and the facial expression recognition system 120 and the facial expression construction system 130 are in communication connection.

The augmented reality smart glasses system 110 is used to capture the user's current environmental information and construct a three-dimensional geographic image of the user's current environmental information according to the current environmental information, and project the three-dimensional geographic image onto the augmented reality display of the opposite end of the call.

The facial expression recognition system 120 is used to recognize the user's current facial expression and current emotion.

The facial expression construction system 130 is used to capture and construct a three-dimensional virtual facial expression of the user.

It should be noted that the augmented reality smart glasses system 110, the facial expression recognition system 120, and the facial expression construction system 130 may be wired communication connections or wireless communication connections, which are not limited herein. The facial expression recognition system 120 and the facial expression construction system 130 may be a wired communication connection or a wireless communication connection, which is not limited herein.

It should also be noted that a wireless connection can be established between the above-mentioned augmented reality smart glasses and the above-mentioned wearable smart bracelet to achieve communication and data transmission. The wireless connection methods include but are not limited to: wireless Bluetooth connection method, wireless LAN connection method, The short-distance wireless communication method and the like enhance display how the smart glasses and the wearable smart bracelet realize the wireless connection, and how to perform communication and data transmission through the wireless connection link, as existing technical means, which will not be repeated here.

In specific applications, the above-mentioned augmented reality smart glasses are worn on the user's face, and the current environment information of the user is obtained through the dual current-view wide-angle camera of the augmented reality smart glasses, and through the facial expression recognition system provided on the side of the augmented reality smart glasses near the face The user's physiological information, the wearable smart bracelet is worn on the user's wrist, and the user's face image information is collected through the 3D structured light collection camera provided on the wearable smart bracelet, and according to the collected face image information Construct three-dimensional virtual facial expressions.

In one embodiment, the augmented reality smart glasses system 110 includes augmented reality smart glasses 111 and a host 112.

In one embodiment, the augmented reality smart glasses 111 include an optical fiber scanning projector 111A, a binocular infrared gesture recognition camera 111B, an eye tracker 111C, a dual current viewing wide-angle camera 111D, an acceleration sensor 111E, a bone conduction headset, and a noise reduction microphone 111F .

The above-mentioned host 112 includes a processor 112A, a controller 112A and an optical fiber scanning projector 111A, a binocular infrared gesture recognition camera 111B, an eye tracker 111C, a dual current viewing wide-angle camera 111D, an acceleration sensor 111E, a bone conduction headset and a noise reduction microphone 111F is electrically connected.

The host 112 is used to collect the projection light source and project the three-dimensional geographic image on the augmented reality display at the opposite end of the call through the optical fiber.

In specific applications, the processor 112A controls the optical fiber scanning projector 111A, the binocular infrared gesture recognition camera 111B, the eye tracker 111C, the dual current viewing wide-angle camera 111D, the acceleration sensor 111E, the bone conduction headset and the noise reduction microphone 111F to work , And receive the data acquired by the optical fiber scanning projector 111A, the binocular infrared gesture recognition camera 111B, the eye tracker 111C, the dual-view wide-angle camera 111D, the acceleration sensor 111E, the bone conduction headset, and the noise reduction microphone 111F.

As shown in FIG. 2, the optical fiber scanning projector 111A, the binocular infrared gesture recognition camera 111B, the eye tracker 111C, the dual current viewing wide-angle camera 111D, the acceleration sensor 111E, the bone conduction headset, and the noise reduction microphone 111F are respectively provided in the above enhancement Reality smart glasses 111 on.

The above-mentioned optical fiber scanning projector 111A is used for optical fiber scanning imaging, the binocular infrared gesture recognition camera 111B is used for photographing and recognizing the user's gestures, the eye tracker 111C is used to capture the intersection point where the user's eyes are focused, and the dual current wide-angle camera 111D It is used to capture the current environment and construct a three-dimensional geographic image. The acceleration sensor 111E is used to detect the user's head posture. The bone conduction headset and noise reduction microphone 111F are used to play voice data of the opposite end of the call and receive voice data input by the user.

In one embodiment, the facial expression recognition system 120 includes an EMG sensor 121, an ESR sensor 122, an ECG signal sensor 123, a respiratory signal sensor 124, and a data processor 125; the data processor 125 and the EMG sensor 121, respectively, The skin electrical response sensor 122, the electrocardiographic signal sensor 123, and the respiratory signal sensor 124 are electrically connected.

As shown in FIG. 3a, the facial expression recognition system includes 4 pairs of 8 sensors, namely a first myoelectric sensor 121A, a second myoelectric sensor 121B, a first cutaneous electrical response sensor 122A, and a second cutaneous electrical sensor 122B, The first ECG signal sensor 123A, the second ECG signal sensor 123B, the first respiration signal sensor 124A, and the second respiration signal sensor 124B, the above eight sensors are respectively in contact with the user's face muscle body surface, and are used to obtain the user's Physiological signals. The physiological signals include skin electrical response signals, myoelectric signals, respiratory signals, and electrocardiographic signals. The data processor 125 performs data processing and analysis on the physiological signals collected by each sensor to identify the user's current facial expression and current emotion.

As shown in FIG. 3b, the facial expression recognition system collects physiological signals corresponding to 8 collection positions through 8 sensors respectively. The specific first EMG sensor 121A collects EMG signals at 01 collection positions, and the second EMG sensor 121B collects 02 The EMG signal at the collection position, the first galvanic skin response sensor 122A acquires the galvanic skin response signal at the 03 acquisition location, the second galvanic skin response sensor 122B acquires the galvanic skin response signal at the 04 acquisition location, and the first ECG signal sensor 123A acquires the 05 The ECG signal of the collection position, the second ECG signal sensor 123B collects the ECG signal of 06 collection position, the first respiratory signal sensor 124A collects the respiratory signal of position 07, and the second respiratory signal sensor 124B collects the 08 respiratory signal of collection position. It should be noted that the above sensors can use existing sensors to collect physiological signals.

In specific applications, after the above 8 sensors respectively correspond to the physiological signals of the 8 collection positions of the face, the muscle tension of each collection position is analyzed according to the physiological signals of each collection position, and then the muscle tension of the 8 collection positions is combined The analysis determines the current facial expression of the user. It should be noted that the foregoing analysis of the muscle tension according to physiological signals and the analysis of the muscle tension according to the 8 collection positions to determine the current facial expression of the user can be implemented by an existing analysis algorithm, which will not be repeated here.

In one embodiment, the facial expression construction system 130 includes a 3D structured light collection camera 131 and an EMG signal sensor 132.

As shown in FIG. 4, the above 3D structured light collection camera 131 and EMG signal sensor 132 are integrated in a wearable smart bracelet. The 3D structured light collection camera 131 is used to collect user's face image information, and the EMG signal sensor 132 is used Used to obtain user's gesture information.

In one embodiment, the above-mentioned augmented reality smart glasses system further includes an augmented reality display, and the augmented reality display is used to display virtual video data sent from a call end.

The wearable augmented reality remote video system provided in this embodiment constructs a three-dimensional virtual facial expression image through a facial expression construction system, and then recognizes the user's current facial expression and current emotion according to the physiological signals collected by the facial expression recognition system, thereby driving facial expressions The construction system updates the three-dimensional virtual facial expression according to the user's current facial expression and current emotion, and constructs a three-dimensional geographic image of the user's environment through the augmented reality smart glasses system, which serves as the background of the video call. The updated three-dimensional virtual facial expression image and The three-dimensional geographic image is projected on the display of the opposite end of the call to realize a holographic and live video call with the opposite end of the call, effectively solving the problem that the current user needs to hold the mobile terminal and make the mobile terminal face the user's face when making a remote video call. Problems with effective video calls.

Example two:

As shown in FIG. 5, this embodiment provides a video call method. The above video call method is implemented based on the above-mentioned wearable enhanced display remote video system, which specifically includes:

Step S101: Capture and construct a three-dimensional virtual facial expression image of the user through the facial expression construction system.

In specific applications, the facial expression construction system includes a 3D structured light collection camera, which scans the user's face through the 3D structured light collection camera to obtain the user's expression information, and constructs a three-dimensional in a virtual environment based on the user's expression information Virtual facial expression image.

Step S102: Recognize the user's current facial expression and current emotion through the facial expression recognition system.

In a specific application, after the three-dimensional virtual facial expression image is generated, the user's current facial expression and current emotion are recognized through the facial expression recognition system. The user's physiological information is collected through the facial expression recognition system, and the user's current facial expression and current emotion are analyzed according to the physiological information.

Step S103: Capture the current environment information of the user through the augmented reality smart glasses system and construct a three-dimensional geographic image of the current environment information of the user according to the current environment.

In specific applications, the front-view binocular wide-angle camera of the augmented reality smart glasses system of the augmented reality smart glasses system captures the current environment information of the user, and constructs a three-dimensional geographic image of the current environment of the user according to the binocular vision 3D reconstruction algorithm as a chat background . At the same time, the user's three-dimensional virtual facial expression image can be mapped to the three-dimensional geographic image, that is, the user's avatar can be mapped to the chat background.

In a specific application, the dual-view camera also captures the user's gestures and interacts with the call peer according to the user's gestures.

In a specific application, an eye tracker captures the intersection area focused by the user's eyes in real time, and switches the chat background according to the intersection area focused by the user's eyes.

Step S104: Update the three-dimensional virtual facial expression image based on the user's current facial expression and current emotion recognized by the facial expression recognition system, and merge the updated three-dimensional virtual facial expression image with the three-dimensional geographic image to generate virtual video data .

In specific applications, the user's physiological information is collected through the facial expression recognition system, and the user's current facial expression and current emotion are analyzed according to the physiological information to drive the facial expression construction system to update the user's three-dimensional virtual facial expression image. Specifically, the current facial expression of the user is determined by collecting the myoelectric signal and skin reaction signal of the face, and the collected human respiratory signal and the electrocardiographic signal are jointly identified by the emotion recognition algorithm in multiple dimensions to determine the user's current facial expression mood.

In specific applications, the updated 3D virtual facial expression image is fused with the 3D geographic image at the corresponding moment, that is, the 3D geographic image of the current environment is obtained in real time, and then the 3D virtual facial expression image corresponding to the current facial expression and the current emotion is projected Go to the three-dimensional geographic image of the current environment and generate virtual video data.

Step S105: Project the virtual video data to the augmented reality display at the opposite end of the call.

In a specific application, the virtual video information with the three-dimensional virtual facial expression image and the three-dimensional geographic image is sent to the call peer through the communication link with the call peer, and displayed through the enhanced display monitor of the call peer.

In specific applications, you can communicate with the augmented reality smart glasses at the opposite end of the call through augmented reality smart glasses, or you can communicate with the augmented reality smart glasses at the opposite end of the call through the wearable smart bracelet, and you can also communicate with the call through the augmented reality smart glasses The communication of the wearable smart bracelet at the opposite end may also be performed through the wearable smart bracelet and the wearable smart bracelet at the opposite end of the call, which is not limited herein.

In specific applications, while projecting virtual video data to the augmented reality display at the opposite end of the call, the synchronized audio data is also transmitted to the opposite end of the call, and the synchronized audio is played through the bone conduction headset of the augmented reality smart glasses at the opposite end of the call data.

In one embodiment, the above-mentioned video call method before step S101 further includes:

Step S106: the user's gesture action is captured through the wearable smart bracelet worn on the user's wrist, and the call object is switched according to the gesture action.

In a specific application, the user's gestures and movements are recognized by the myoelectric sensor on the wrist, and the call object is switched according to the gestures of the user's left and right waving hands.

In specific applications, motion information is obtained through the myoelectric sensor of the wrist, such as the angle of rotation of the wrist, the swing amplitude of the wrist up and down, and the swing width of the wrist, etc., and the user's gesture movement is analyzed and recognized based on the motion information, and the user's gesture movement Switch to the corresponding call partner.

Exemplarily, if the gesture action is to swing the arm to the left, then select the previous call object of the current call object from the call list, and cut off the video call with the current call object, and establish a video call with the previous call object; if the gesture The action is to swing the arm to the right, then select the next call object of the current call object from the call list, cut off the video call with the current call object, and establish a video call with the next call object. It should be noted that the correspondence between the gesture action and the switching of the call object can be determined according to the settings of the user, and is not limited here.

In specific applications, the above-mentioned wearable augmented reality remote video system can also detect the user's head posture through the acceleration sensor of the augmented reality smart glasses, and recognize the user's gestures through the binocular infrared gesture recognition camera of the augmented reality smart glasses, thereby achieving Real-time interaction with the opposite end of the call.

The video call method provided in this embodiment can also construct a three-dimensional virtual facial expression image through the facial expression construction system, and then recognize the user's current facial expression and current emotion according to the physiological signals collected by the facial expression recognition system, thereby driving the facial expression construction system Update the three-dimensional virtual facial expression according to the user's current facial expression and current emotion, and construct the three-dimensional geographic image of the user's environment through the augmented reality smart glasses system, as the background of the video call, the updated three-dimensional virtual facial expression image and the three-dimensional geographic The image is projected on the display of the opposite end of the call to realize a holographic and live video call with the opposite end of the call, effectively solving the problem that the current user needs to hold the mobile terminal and make the mobile terminal face the user's face in order to perform effective video when making a remote video call The problem with the call.

Example three:

As shown in FIG. 6, in this embodiment, step S101 in Embodiment 1 specifically includes:

Step S201: Scan the user's facial expression through the 3D structured light collection camera of the facial expression construction system.

Step S202: Construct a three-dimensional virtual facial image of the user in the virtual space according to the scanned facial expression of the user.

In specific applications, the user's face is scanned by the 3D structured light collection camera to obtain the user's expression information, and a three-dimensional virtual facial expression image is constructed in the virtual environment according to the user's expression information.

Example 4:

As shown in FIG. 7, in this embodiment, step S102 in Embodiment 1 specifically includes:

Step S301: Collect physiological signals of the user, the physiological signals including skin electrical response signals, myoelectric signals, respiratory signals and electrocardiographic signals.

In specific applications, the physiological information of the user is collected through the eight sensors of the facial expression recognition system, and the physiological signals include skin electrical response signals, myoelectric signals, respiratory signals, and electrocardiographic signals. Specifically, the EMG signals are collected by the first EMG sensor and the second EMG sensor, the skin reaction signals are collected by the first EEG sensor and the second EEG sensor, and the first EKG signal sensor and the second EKG signal are collected The signal sensor collects the electrocardiographic signal, and the first respiratory signal sensor and the second respiratory signal sensor collect the respiratory signal.

Step S302: Perform multi-dimensional joint recognition based on the user's physiological signal to determine the current facial expression and current emotion of the current user.

In a specific application, analyzing the user's current facial expression and current emotion based on physiological information drives the facial expression construction system to update the user's three-dimensional virtual facial expression image.

Specifically, the current facial expression of the user is determined by collecting the myoelectric signal and the skin reaction signal of the face, and the collected human respiratory signal and the electrocardiographic signal are jointly identified by the emotion recognition algorithm in multiple dimensions to determine the user's current facial expression mood.

In one embodiment, the above step S302 specifically includes the following steps:

Step S3021: Initialize the physiological signal, filter out interference signals and perform data sampling to obtain discrete physiological signals. Step S3022: Extract feature values of various discrete physiological signals.

In specific applications, the feature value of each discrete physiological signal is extracted through a statistical feature algorithm, and the feature value of the physiological signal is extracted by specific classification, that is, the feature value of the myoelectric signal, the feature value of the skin reaction signal, and the feature of the respiratory signal are extracted Value and extract the characteristic value of ECG signal.

Step S3023: Based on the discrete binary particle swarm optimization algorithm, the effective characteristic signals of various discrete physiological signals are determined according to the characteristic values of various discrete physiological signals.

In specific applications, in order to reduce the amount of calculation, based on the discrete binary particle swarm algorithm, the effective characteristic signals of various discrete physiological signals are determined according to the characteristic values of various discrete physiological signals. Effective feature signals refer to signal features useful for classifying emotions.

Step S3024: Perform emotion recognition based on the effective characteristic signals of the various types of discrete physiological signals, and obtain corresponding emotion categories.

In specific applications, the effective feature signals of various discrete physiological signals are screened out, and a feature set of valid feature signals is formed, and emotion recognition is performed according to the feature set, and the category to which the pattern with the highest similarity belongs belongs is output as the recognition result. The similarity calculation is performed by various emotion feature values of the emotion database and the effective feature signals of various discrete physiological signals, and the emotion attribute with the highest similarity is used as the emotion category of the current user.

In specific applications, the specific implementation manners for extracting the characteristic values of various types of discrete physiological signals are as follows:

(1) Coding: According to the characteristics of feature selection, each feature is defined as a one-dimensional discrete binary variable of the granule, the variable length is equal to the number of all features, if the i-th bit is 1, then the i-th feature is selected, Otherwise, this feature is blocked. Therefore, each particle represents a different subset of features.

(2) Initialization: In order to obtain different numbers of features, the number of "1"s contained in each particle is first randomly generated, and then these "1"s are randomly distributed in all dimensions of this particle.

(3) Evaluation of fitness: Search for the target based on the BPSO algorithm to obtain the global minimum. The evaluation of fitness above includes two parts:

(a) Classification error rate. Use the features determined in the feature subset to train the classifier, use the results of the classification to evaluate the performance of the particles, and use this to guide the target search of the BPSO algorithm;

(b) The number of features used. Each feature subset contains a certain number of features. If the two feature subsets have the same accuracy, the subset with fewer features is selected.

It should be noted that in the two factors of accuracy and number of features, the accuracy is the key consideration, so the fitness function is determined as follows:

fitness＝10 ⁵ ×(1-Accuracy)+k×Ones

Accuracy in the formula represents the accuracy rate obtained by each individual. Ones is the number of "1"s contained in the particle position, that is, the number of selected features. The BPSO algorithm is to find the global minimum value and locate the weight of the accuracy rate to 100,000 to increase the importance of the accuracy rate. High accuracy means that the fitness value is small, and this feature subset is likely to win the competition. k is a constant greater than O, and is a compromise parameter between accuracy and the number of features. The larger the value, the more attention is paid to the number of features. In this embodiment, K takes a value of 0.5.

In specific applications, the feature selection method based on the BPSO algorithm is as follows:

First of all, in the discrete binary PSO algorithm, the feature vector can use a binary vector to represent particles. The expression is as follows:

X = (x ₁ , x ₂ ,..., x _D ) ^T , x _i ∈{0, 1}

X represents the number of original features. If the i-th bit of x is l, the feature is selected, if it is O, the feature is not selected, and T (speed) represents the probability that the feature is selected.

Therefore, the above steps for extracting feature values are as follows:

Step 1: Set the group size to m, the maximum number of iterations, the iteration closing value t and the minimum error threshold;

Step 2: Initialize the position and velocity of the particle swarm, calculate the fitness of each particle as the initial individual extremum pbesti, and assign the minimum value of all pbesti to gbest as the initial global extremum, the initial value of Vi is set to O;

Step 3: According to the classification result of the nearest neighbor classifier, perform performance evaluation on the particle swarm;

Step 4: Update the speed and position of particles;

Step 5: Use Hamming distance to judge the similarity of particles;

Step 6: Determine whether the minimum error threshold or the maximum number of iterations is reached. If so, perform step 7; otherwise, go to step 3;

Step 7: Output the optimal target value and particle, and the feature corresponding to the particle position is the optimal feature combination found.

Example 5:

8 is a schematic diagram of a terminal device according to Embodiment 7 of the present invention. As shown in FIG. 8, the terminal device 8 of this embodiment includes: a processor 80, a memory 81, and a computer program 82 stored in the memory 81 and executable on the processor 80, for example, a program. When the processor 80 executes the computer program 82, the steps in the above embodiments of each picture processing method are implemented, for example, steps S101 to S105 shown in FIG. 5.

Exemplarily, the computer program 82 may be divided into one or more modules/units, and the one or more modules/units are stored in the memory 81 and executed by the processor 80 to complete this invention. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 82 in the terminal device 8. For example, the computer program 82 may be divided into an expression construction module, a recognition module, an environment construction module, a fusion module, and a projection module. The specific functions of each module are as follows:

Expression construction module, used to capture and construct the user's three-dimensional virtual facial expression image through the facial expression construction system;

Recognition module, used to recognize the user's current facial expression and current emotion through the facial expression recognition system;

An environment construction module, configured to capture the user's current environment information through the augmented reality smart glasses system and construct a three-dimensional geographic image of the user's current environment information according to the current environment;

The fusion module is used to update the three-dimensional virtual facial expression image based on the current facial expression and current emotion of the user recognized by the facial expression recognition system, and fuse the updated three-dimensional virtual facial expression image with the three-dimensional geographic image to generate a virtual Video data

The projection module is used to project the virtual video data to the augmented reality display at the opposite end of the call.

The terminal device 8 may be a computing device such as a desktop computer, a notebook, a palmtop computer, and a cloud management server. The terminal device may include, but is not limited to, a processor 80 and a memory 81. Those skilled in the art may understand that FIG. 8 is only an example of the terminal device 8 and does not constitute a limitation on the terminal device 8, and may include more or less components than the illustration, or a combination of certain components or different components. For example, the terminal device may further include an input and output device, a network access device, a bus, and the like.

The so-called processor 80 may be a central processing unit (Central Processing Unit, CPU), or other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application-specific integrated circuits (Application Specific Integrated Circuit, ASIC), Ready-made programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory 81 may be an internal storage unit of the terminal device 8, such as a hard disk or a memory of the terminal device 8. The memory 81 may also be an external storage device of the terminal device 8, for example, a plug-in hard disk equipped on the terminal device 8, a smart memory card (Smart, Media, Card, SMC), and a secure digital (SD) Cards, flash cards, etc. Further, the memory 81 may also include both an internal storage unit of the terminal device 8 and an external storage device. The memory 81 is used to store the computer program and other programs and data required by the terminal device. The memory 81 can also be used to temporarily store data that has been or will be output.

Those skilled in the art can clearly understand that, for convenience and conciseness of description, only the above-mentioned division of each functional unit and module is used as an example for illustration. In practical applications, the above-mentioned functions can be allocated by different functional units, Module completion means that the internal structure of the system is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above integrated unit may use hardware It can also be implemented in the form of software functional units. In addition, the specific names of each functional unit and module are only for the purpose of distinguishing each other, and are not used to limit the protection scope of the present application. For the specific working processes of the above units and modules in the wireless terminal, reference may be made to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

In the above embodiments, the description of each embodiment has its own emphasis. For a part that is not detailed or recorded in an embodiment, you can refer to the related descriptions of other embodiments.

Those of ordinary skill in the art may realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application of the technical solution and design constraints. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

In the embodiments provided by the present invention, it should be understood that the disclosed system/terminal device and method may be implemented in other ways. For example, the system/terminal device embodiments described above are only schematic. For example, the division of the module or unit is only a logical function division. In actual implementation, there may be another division manner, such as multiple units Or components can be combined or integrated into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, systems or units, and may be in electrical, mechanical or other forms.

The unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or may be distributed to multiple network units on. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above integrated unit can be implemented in the form of hardware or software function unit.

If the integrated module/unit is implemented in the form of a software functional unit and set as an independent product for sale or use, it may be stored in a computer-readable storage medium. Based on this understanding, the present invention can realize all or part of the processes in the methods of the above embodiments, and can also be completed by a computer program instructing relevant hardware. The computer program can be stored in a computer-readable storage medium. When the program is executed by the processor, the steps of the foregoing method embodiments may be implemented. Wherein, the computer program includes computer program code, and the computer program code may be in a source code form, an object code form, an executable file, or some intermediate form, etc. The computer-readable medium may include any entity or system capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM, Read-Only Memory) , Random Access Memory (RAM, Random Access Memory), electrical carrier signals, telecommunications signals and software distribution media, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in jurisdictions. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media Excluded are electrical carrier signals and telecommunications signals.

The above-mentioned embodiments are only used to illustrate the technical solutions of the present invention, not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, persons of ordinary skill in the art should understand that they can still implement the foregoing The technical solutions described in the examples are modified, or some of the technical features are equivalently replaced; and these modifications or replacements do not deviate from the essence and scope of the technical solutions of the embodiments of the present invention, and should be included in Within the protection scope of the present invention.

Claims

A wearable augmented reality remote video system, characterized in that the wearable augmented reality remote video system includes an augmented reality smart glasses system, a facial expression recognition system, and a facial expression construction system;

The augmented reality smart glasses system and the facial expression recognition system are set in the augmented reality smart glasses, and the facial expression construction system is set in the wearable smart bracelet; the augmented reality smart glasses system and the facial expression recognition system are respectively A communication connection with the facial expression construction system, and a communication connection with the facial expression recognition system and the facial expression construction system;

The augmented reality smart glasses system is used to capture the current environment information of the user and construct a three-dimensional geographic image of the current environment information of the user according to the current environment, and project the three-dimensional geographic image to the augmented reality display of the opposite end of the call;

The facial expression recognition system is used to recognize the user's current facial expression and current emotion;

The facial expression construction system is used to capture and construct a three-dimensional virtual facial expression of a user.
The wearable augmented reality remote video system according to claim 1, wherein the augmented reality smart glasses system includes augmented reality smart glasses and a host;

The augmented reality smart glasses include an optical fiber scanning projector, a binocular infrared gesture recognition camera, an eye tracker, a dual current viewing wide-angle camera, an acceleration sensor, a bone conduction headset, and a noise reduction microphone;

The host includes a processor, the controller and the optical fiber scanning projector, the binocular infrared gesture recognition camera, the eye tracker, the dual current viewing wide-angle camera, the acceleration sensor, the The bone conduction earphone and the noise reduction microphone are electrically connected;

The host is used to collect the projection light source and project the three-dimensional geographic image on the augmented reality display at the opposite end of the call through the optical fiber.
The wearable augmented reality remote video system according to claim 1, wherein the facial expression recognition system includes an electromyography sensor, a skin electrical response sensor, an electrocardiographic signal sensor, a respiratory signal sensor, and a data processor;

The data processor is electrically connected to the myoelectric sensor, the skin electrical response sensor, the electrocardiographic signal sensor and the respiration signal sensor, respectively.
The wearable augmented reality remote video system according to claim 1, wherein the facial expression construction system includes a 3D structured light collection camera and an EMG signal sensor.
The wearable augmented reality remote video system according to claim 4, wherein the 3D structured light collection camera and the EMG signal sensor are integrated in a wearable smart bracelet.
A video call method based on the wearable augmented reality remote video system according to claim 1, characterized in that it includes:

Capture and construct the user's three-dimensional virtual facial expression image through the facial expression construction system;

Recognize the user's current facial expression and current emotion through the facial expression recognition system;

Capture the current environment information of the user through the augmented reality smart glasses system and construct a three-dimensional geographic image of the current environment information of the user according to the current environment;

Updating the three-dimensional virtual facial expression image based on the user's current facial expression and current emotion recognized by the facial expression recognition system, and fusing the updated three-dimensional virtual facial expression image with the three-dimensional geographic image to generate virtual video data;

Project the virtual video data into the augmented reality display at the opposite end of the call.
The video call method according to claim 6, further comprising:

The wearable smart bracelet worn on the user's wrist captures the user's gestures and switches the call object according to the gestures.
The video call method according to claim 6, wherein the capturing and constructing the three-dimensional virtual facial expression image of the user through the facial expression construction system includes:

Scan the user's facial expressions through the 3D structured light collection camera of the facial expression construction system;

According to the scanned facial expressions of the user, a three-dimensional virtual facial image of the user is constructed in the virtual space.
The video call method according to claim 6, wherein the recognizing the user's current facial expression and current emotion through the facial expression recognition system includes:

Collecting physiological signals of the user, the physiological signals including skin electrical response signals, myoelectric signals, respiratory signals and electrocardiographic signals;

Multi-dimensional joint recognition is performed based on the user's physiological signals to determine the current facial expression and current emotion of the current user.
The video call method according to claim 9, wherein the multi-dimensional joint recognition is performed based on the user's physiological signal to determine the current facial expression and current emotion of the current user, including:

Initialize the physiological signals, filter out interference signals and perform data sampling to obtain discrete physiological signals;

Extract the characteristic values of various discrete physiological signals;

Based on discrete binary particle swarm optimization algorithm to determine the effective characteristic signals of various discrete physiological signals based on the characteristic values of various discrete physiological signals

Emotion recognition is performed according to the effective characteristic signals of the various types of discrete physiological signals to obtain corresponding emotion categories.