CN113610018A - VR real-time communication interactive system and method combining 5G, expression tracking and beautifying - Google Patents
VR real-time communication interactive system and method combining 5G, expression tracking and beautifying Download PDFInfo
- Publication number
- CN113610018A CN113610018A CN202110918746.0A CN202110918746A CN113610018A CN 113610018 A CN113610018 A CN 113610018A CN 202110918746 A CN202110918746 A CN 202110918746A CN 113610018 A CN113610018 A CN 113610018A
- Authority
- CN
- China
- Prior art keywords
- user
- communication
- model
- expression
- tracking
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000004891 communication Methods 0.000 title claims abstract description 185
- 238000000034 method Methods 0.000 title claims abstract description 41
- 230000002452 interceptive effect Effects 0.000 title description 21
- 230000003993 interaction Effects 0.000 claims abstract description 93
- 238000012545 processing Methods 0.000 claims abstract description 62
- 238000004088 simulation Methods 0.000 claims abstract description 38
- 230000003796 beauty Effects 0.000 claims abstract description 27
- 238000012549 training Methods 0.000 claims description 21
- 230000033001 locomotion Effects 0.000 claims description 17
- 210000005252 bulbus oculi Anatomy 0.000 claims description 11
- 210000003128 head Anatomy 0.000 claims description 11
- 238000009877 rendering Methods 0.000 claims description 11
- 230000008569 process Effects 0.000 claims description 10
- 230000008451 emotion Effects 0.000 claims description 8
- 230000015572 biosynthetic process Effects 0.000 claims description 7
- 230000008859 change Effects 0.000 claims description 7
- 239000006260 foam Substances 0.000 claims description 7
- 238000001914 filtration Methods 0.000 claims description 6
- 238000005286 illumination Methods 0.000 claims description 6
- 238000010801 machine learning Methods 0.000 claims description 6
- 210000001747 pupil Anatomy 0.000 claims description 6
- 238000000227 grinding Methods 0.000 claims description 5
- 238000013507 mapping Methods 0.000 claims description 5
- 230000002087 whitening effect Effects 0.000 claims description 5
- 238000013135 deep learning Methods 0.000 claims description 4
- 230000007613 environmental effect Effects 0.000 claims description 4
- 210000001508 eye Anatomy 0.000 claims description 4
- 238000003786 synthesis reaction Methods 0.000 claims description 4
- 230000010354 integration Effects 0.000 claims description 3
- 230000004807 localization Effects 0.000 claims description 3
- 238000001228 spectrum Methods 0.000 claims description 3
- 238000012360 testing method Methods 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 description 19
- 239000011521 glass Substances 0.000 description 13
- 230000006698 induction Effects 0.000 description 8
- 239000000463 material Substances 0.000 description 6
- 230000005540 biological transmission Effects 0.000 description 5
- 238000013527 convolutional neural network Methods 0.000 description 5
- 238000001514 detection method Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 4
- 239000000243 solution Substances 0.000 description 4
- 238000006467 substitution reaction Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000010276 construction Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 239000004205 dimethyl polysiloxane Substances 0.000 description 3
- 235000013870 dimethyl polysiloxane Nutrition 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- CXQXSVUQTKDNFP-UHFFFAOYSA-N octamethyltrisiloxane Chemical compound C[Si](C)(C)O[Si](C)(C)O[Si](C)(C)C CXQXSVUQTKDNFP-UHFFFAOYSA-N 0.000 description 3
- 238000004987 plasma desorption mass spectroscopy Methods 0.000 description 3
- 229920000435 poly(dimethylsiloxane) Polymers 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 210000000887 face Anatomy 0.000 description 2
- 230000001815 facial effect Effects 0.000 description 2
- 238000013508 migration Methods 0.000 description 2
- 230000005012 migration Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000008447 perception Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- FGUUSXIOTUKUDN-IBGZPJMESA-N C1(=CC=CC=C1)N1C2=C(NC([C@H](C1)NC=1OC(=NN=1)C1=CC=CC=C1)=O)C=CC=C2 Chemical compound C1(=CC=CC=C1)N1C2=C(NC([C@H](C1)NC=1OC(=NN=1)C1=CC=CC=C1)=O)C=CC=C2 FGUUSXIOTUKUDN-IBGZPJMESA-N 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000002146 bilateral effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000012938 design process Methods 0.000 description 1
- 238000004134 energy conservation Methods 0.000 description 1
- 230000004424 eye movement Effects 0.000 description 1
- 230000008921 facial expression Effects 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 238000007654 immersion Methods 0.000 description 1
- 238000002347 injection Methods 0.000 description 1
- 239000007924 injection Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 210000000697 sensory organ Anatomy 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/77—Retouching; Inpainting; Scratch removal
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
- H04L51/04—Real-time or near real-time messaging, e.g. instant messaging [IM]
- H04L51/046—Interoperability with other network applications or services
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The invention discloses a VR real-time communication interaction system and method combining 5G, expression tracking and beauty, the system is provided with a cloud server, a plurality of cluster processing devices, a plurality of communication rooms and a plurality of user interaction devices, the cloud server is respectively connected with the cluster processing devices, each cluster processing device is respectively connected with a plurality of adjacent user interaction devices, so that a plurality of area processing stations are formed, each user interaction device is arranged in the corresponding communication room, the cluster processing devices are used for matching and combining a face model and a general body model to obtain a user simulation model, and the user simulation model is forwarded to a communication party with established communication connection through the cloud server. The invention generates stronger substituting feeling on interaction by modeling the user and environment establishing communication interaction and restoring the expression and the gesture of the communication party.
Description
Technical Field
The invention relates to the technical field of virtual reality, in particular to a VR real-time communication interaction system and method combining 5G, expression tracking and beauty.
Background
VR refers to virtual reality technology, which mainly uses a multi-technology fusion approach including, but not limited to, computer vision technology, wide-angle display technology, dynamic motion capture technology, etc. by using VR, we can create and experience a virtual environment that relies on a computer-generated simulated reality environment, and provide multi-dimensional perception information such as vision, hearing, and touch to people, so that people can generate substitution feeling.
In recent years, with the continuous development and innovation of VR technology, VR technology has also been widely applied in various fields, such as being used in game entertainment, making up for the deficiency of 3D immersion of games, being also used in the field of medical simulation training, providing real medical training scenes, etc.; however, in general, most VR usage scenarios in the market are based on personal experience of a single user, and lack usage scenarios of multiple users, and lack online interaction and real-time communication between different users. In addition, under the background of the modern times, an instant messaging and interactive device based on a VR technology for meeting the use scenes of multiple users can undoubtedly provide perception information with more dimensions for users compared with an instant messaging and interactive device such as a smart phone, so that the users have a sense of being personally on the scene, and various entertainment and communication requirements of different users separating two places are met.
However, in the research of the prior VR-based instant messaging and interactive device, such as a street call kiosk VR sharing device (patent application No. CN202020822636.5), and a VR-based method, device and system with watch (patent application No. CN202011182157.2), the former is only an instant messaging and interactive device based on the prior call kiosk combined with VR glasses and a small number of sensors, which only meets the communication requirement and is difficult to provide entertainment content, and the latter only provides virtual scenes. In most of the current patents, the 5G technology is less used as the main wireless communication means, and it is difficult to meet the requirement of multi-user instant online.
Disclosure of Invention
In order to overcome the defects and shortcomings of the prior art, the invention provides a VR real-time communication interaction system combining 5G, expression tracking and beauty, which generates stronger substitution feeling on interaction by modeling a user and an environment for establishing communication interaction and accurately restoring the expression and the posture of a communication party, and simultaneously meets various requirements of multi-user communication, entertainment and the like.
The invention aims to provide a VR real-time communication interaction method combining 5G, expression tracking and beauty.
In order to achieve the first purpose, the invention adopts the following technical scheme:
a VR real-time communication interaction system combining 5G, expression tracking and beauty is provided with a cloud server, a plurality of cluster processing devices, a plurality of communication rooms and a plurality of user interaction devices, wherein the cloud server is respectively connected with the cluster processing devices, each cluster processing device is respectively connected with a plurality of adjacent user interaction devices so as to form a plurality of area processing stations, each user interaction device is arranged in a corresponding communication room, each user interaction device comprises a first image acquisition device, a user processor, a head-mounted VR interaction device and a first communication module, the user processor is respectively connected with the first image acquisition device and the first communication module, the first communication module is further connected with the cluster processing devices, and the user processor is in data transmission with the cluster processing devices through the first communication module;
the head-mounted VR interaction device is provided with a display device, a second image acquisition device, a micro electrode plate and a second communication module, wherein the micro electrode plate, the display device and the second image acquisition device are respectively connected with the second communication module, and the second communication module is also connected with the first communication module in a wireless manner;
the microelectrode sheet is used for collecting EOG signals;
the first image acquisition device is used for acquiring an environment image containing limbs of a user, the second image acquisition device is used for acquiring a face image of the user, and the second image acquisition device is arranged on the head-mounted VR interaction device;
the user processor is used for creating a general body model and a face model, recognizing the gesture and the expression of a user to obtain gesture tracking information and an expression recognition type, wherein the gesture information comprises head gesture information, hand gesture information and limb gesture information;
the cluster processing device is used for receiving the general body model, the face model, the posture tracking information and the expression recognition type, matching and combining the face model and the general body model to obtain a user simulation model, and forwarding the user simulation model to a communication party with established communication connection through the cloud server;
the cloud server is used for forwarding data and pre-building an environment scene model and a user simulation model;
the display device is used for displaying the environmental scene model and the user simulation model;
and the cluster processing device connected with the communication party updates the gesture and the expression of the user simulation model according to the gesture tracking information and the expression recognition type.
As a preferred technical scheme, the cloud server is used for matching a communication party establishing connection with a current user, wherein the communication party is one or more communication users establishing communication interaction with the current user;
the user processor receives preset environment scene types input by a user, the preset environment scene types are forwarded to the cloud server through the cluster processing device, the cloud server stores a plurality of preset environment scene types, the cloud server generates an environment scene model according to the preset environment scene types selected by the user through the user processor, a communication party connected with the current user is matched, and the environment scene model, the user simulation model, the posture tracking information and the expression recognition types are sent to the cluster processing device connected with the communication party.
As a preferred technical scheme, the user interaction device further comprises a laser sensor, the laser sensor is arranged in the communication room, and the laser sensor is connected with the user processor;
the laser sensor is used for positioning the position of a user in a communication room, the user processor acquires the user position through the laser sensor, the user position is transmitted to the cloud server through the cluster processing device through the first communication module, the cloud server forwards the user position to the cluster processing device connected with a communication party, the communication party is one or any plurality of communication users which establish communication interaction with the current user, the cluster processing device connected with the communication party updates the relative position of the user simulation model in the environment scene model according to the user position, and the position updating information is sent to the user processor of the communication party.
As a preferred technical scheme, the user interaction device further comprises a photosensitive sensor, a laser lighthouse base station and an induction handle, wherein the laser lighthouse base station is arranged at two ends of a diagonal line of the user interaction device, each laser lighthouse base station is about 2m high, a laser sensor array for positioning is arranged in each laser lighthouse base station, and the photosensitive sensor is arranged on the ground of the communication room and the induction handle, so that the laser lighthouse laser positioning system is formed by the laser sensor array and the photosensitive sensor.
As a preferred technical solution, the user interaction device further comprises an ambient light sensor, the ambient light sensor is disposed in the communication room, and the ambient light sensor is connected to the user processor;
the communication system comprises an ambient light sensor, a user processor, a first communication module, a cloud server and a communication party, wherein the ambient light sensor is used for sensing the light conditions around users in a communication room, the user processor acquires ambient light information of the users by using the ambient light sensor, the ambient light information of the users is transmitted to the cloud server through the cluster processing device through the first communication module, the cloud server forwards the ambient light information of the users to the cluster processing device connected with the communication party, the communication party is one or any plurality of communication users which have established communication interaction with the current user, and the display brightness of an ambient scene model is adjusted by the user processor of the communication party according to the ambient light information of the users.
As a preferred technical solution, the cluster processing device and the user interaction device are in a one-to-many relationship, and the data of each user interaction device in an area is resolved and distributed by one cluster processing device deployed in the area.
As a preferred technical scheme, the head-mounted VR interaction device further comprises a mirror frame, the micro electrode plate, the display device, the second image acquisition device and the second communication module are all arranged on the mirror frame, the mirror frame is provided with a plurality of foam areas, and the foam areas are used for mounting the micro electrode plate.
In order to achieve the second object, the invention adopts the following technical scheme:
a VR real-time communication interaction method combining 5G, expression tracking and beauty comprises the following steps:
model building steps: creating a general body model and a face model, and matching and combining the face model and the general body model to obtain a user simulation model;
a scene building step: selecting a target scene type, and generating an environment scene model;
a data acquisition step: acquiring an environment image containing limbs of a user, a face image of the user and an EOG signal;
updating model information: separating an environment image containing user limbs to obtain a user posture image, tracking the head, the hand and the limb movement of a user according to the user posture image to obtain posture tracking information, tracking the expression of the user according to a user face image to obtain eyeball characteristics, and inputting the eyeball characteristics and an EOG signal into an expression recognition model to predict to obtain an expression recognition type;
the expression recognition model is obtained by machine learning through a plurality of groups of training data, each group of training data comprises expression characteristics and label information for identifying expression recognition types corresponding to the expression characteristics, and the expression recognition types are identifiers of a plurality of expression types;
a step of image formation adjustment: and beautifying the user simulation model by rendering according to the preset beautifying type selected by the user, wherein the beautifying comprises face thinning, skin grinding and whitening.
User synchronization step: and matching a communication party connected with the current user, sending the environment scene model, the user simulation model, the posture tracking information and the expression recognition type to the communication party, and updating the posture and the expression of the user simulation model by the communication party according to the posture tracking information and the expression recognition type.
As a preferred technical scheme, the expression recognition model is obtained by performing machine learning through multiple groups of training data, and the specific process includes:
selecting the pupil diameter and the fixation frequency as eyeball characteristics, and reducing the influence of illumination on the change of the pupil diameter by constructing an illumination reflection model;
filtering the EOG signal under a 0.05-50 Hz band-pass filter to obtain the EOG signal with power spectrum change conditions under different scanning frequencies;
acquiring eye characteristics and EOG signals of anger, pain, disgust, sadness, fear, surprise, happiness and joy through different external video and sound tests, and forming emotion training set data by taking corresponding expression types as label information;
and finding the optimal characteristic value of the two kinds of characteristic data synthesis corresponding to different emotions through an SVM classifier to complete the classification of the expression recognition types to obtain an expression recognition model.
As a preferred technical solution, the obtaining of the posture tracking information by performing the head tracking, the hand tracking and the limb movement tracking of the user according to the user posture image specifically includes performing motion capture and posture estimation by using a segmented 3D posture estimation method, where the 3D posture estimation method includes the following steps:
acquiring depth image information and plane information of a current camera shooting area by using a depth camera;
extracting the plane information by using a VGG network to obtain a characteristic value;
the method comprises the following steps of completing person target tracking of a single person or a plurality of persons by using a target tracking algorithm based on deep learning: inputting the extracted characteristic value, predicting the 2D joint position in real time on a human target rectangular frame by using a CNN network to obtain corresponding Heatmaps-H{j,t}；
Based on Heatmaps-H{j,t}Extracting the connection relation of each joint from the depth image information;
will Heatmaps-H{j,t}And the connection relation of each joint is used as input, and the mapping of the 2D picture pixels and the 3D surface model points is carried out by utilizing a pre-trained RCNN to obtain the corresponding 3D joint positions;
integration of 3D joint positions to yield location-maps-X{j,t}、Y{j,t}、Z{j,t}(ii) a Wherein j represents a joint point, and t represents time;
and extracting based on Heatmaps and localization maps to obtain 2D key points and 3D posture estimation, and combining the two to obtain final motion capture and posture estimation data.
Compared with the prior art, the invention has the following advantages and beneficial effects:
(1) according to the invention, the first image acquisition device, the second image acquisition device and the microelectrode sheet are arranged on the user interaction device in the communication room to acquire data of body data of a user, so that 3D models of the two parties can be quickly and accurately established, and then the face of the user can be modeled by acquiring facial information and is led into the face of the body model, so that the whole person can be accurately restored; then, wireless transmission is carried out through the first communication module and the second communication module, so that the models of the two parties are led into a self-selected scene, and the two parties in communication have the feeling of face-to-face conversation; in general, the invention makes both communication parties be on the scene, and solves the gap feeling brought to people by the existing communication method.
(2) Compared with the traditional mode of distributing calculated data in cloud computing, the VR real-time communication interactive system combining 5G, expression tracking and beauty in the invention has the advantages that the graph rendering on the edge cloud correspondingly calculates and feeds back delay-sensitive information such as equipment head tracking, controller tracking, hand tracking and motion tracking, thereby achieving lower delay; data processing work in the interactive system is completed through the cloud, and the setting cost of hardware such as a processor is reduced.
(3) Based on the technical scheme that the 5G interaction technology is applied to the VR real-time communication interaction system combining 5G, expression tracking and beautifying, the technical scheme of the invention provides comprehensive support for the instant interaction of the VR interaction system and the terminal equipment, solves the problem of the demand of strong operation and network support for prior information (face, body data and the like of a user) acquired in advance by the VR system during the instant interaction, feedback data generated in the experimental process and the like, and compared with the traditional mobile communication network, the 5G network can bear exponential data quantity and has good performances in the aspects of accelerating data transmission rate, reducing congestion delay, improving safety and stability and the like.
Drawings
Fig. 1 is a schematic structural diagram of a VR real-time communication interactive system combining 5G, expression tracking, and beauty in embodiment 1 of the present invention;
FIG. 2 is a schematic structural diagram of a user interaction device in embodiment 1 of the present invention;
fig. 3 is a flowchart illustrating steps of a VR real-time communication interaction method combining 5G, expression tracking, and beauty in embodiment 2 of the present invention;
fig. 4 is a schematic diagram of scene update in embodiment 2 of the present invention.
Detailed Description
In the description of the present disclosure, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings, and are only for convenience of describing and simplifying the present disclosure, but do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and thus, should not be construed as limiting the present disclosure.
Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. Also, the use of the terms "a," "an," or "the" and similar referents do not denote a limitation of quantity, but rather denote the presence of at least one. The word "comprising" or "comprises", and the like, means that the element or item appearing before the word covers the element or item listed after the word and its equivalents, but does not exclude other elements or items. The terms "connected" or "coupled" and the like are not restricted to physical or mechanical connections, but may include electrical connections, whether direct or indirect.
In the description of the present disclosure, it is to be noted that the terms "mounted," "connected," and "connected" are to be construed broadly unless otherwise explicitly stated or limited. For example, the connection can be fixed, detachable or integrated; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meaning of the above terms in the present disclosure can be understood in specific instances by those of ordinary skill in the art. In addition, technical features involved in different embodiments of the present disclosure described below may be combined with each other as long as they do not conflict with each other.
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Examples
Example 1
The embodiment is described by taking an actual application scenario of a telephone booth which is centrally arranged at a street and combines a 5G, expression tracking and beauty VR real-time communication interactive system as an example. Compared with a common telephone booth, the communication interaction method can meet the requirements of face-to-face experience and customization of user communication interaction. The small-size convenient telephone booth is lower in cost, good in adaptability and strong in popularization. The method has certain guiding significance for the practical application of the VR real-time communication interactive system combining 5G, expression tracking and beauty.
As shown in fig. 1, the present embodiment provides a VR real-time communication interactive system combining 5G, expression tracking and beauty, and the present embodiment takes VR glasses as a head-mounted VR interactive device for explanation: the system is provided with a cloud server, a plurality of cluster processing devices, a plurality of communication rooms and a plurality of user interaction devices, wherein the cloud server is respectively connected with the cluster processing devices, each cluster processing device is respectively connected with a plurality of adjacent user interaction devices, so that a plurality of area processing stations are formed, and each user interaction device is arranged in the corresponding communication room.
In this embodiment, the cluster processing device and the user interaction device are in a one-to-many relationship, that is, data of each user interaction device in an area is resolved and distributed by one cluster processing device deployed in the area.
As shown in fig. 2, the user interaction device includes a first image acquisition device, a user processor, VR glasses, and a first communication module, the user processor is connected to the first image acquisition device and the first communication module, the first communication module is further connected to the cluster processing device, and the user processor performs data transmission with the cluster processing device through the first communication module;
the VR glasses are provided with a glasses frame, a display device, a second image acquisition device, a micro electrode plate and a second communication module, the micro electrode plate, the display device, the second image acquisition device and the second communication module are all arranged on the glasses frame, the micro electrode plate, the display device and the second image acquisition device are respectively connected with the second communication module, and the second communication module is further connected with the first communication module in a wireless mode. The lens frame is provided with a plurality of foam areas, the foam areas are used for installing micro electrode pieces, the micro electrode pieces are uniformly distributed in the foam areas and surround the foam areas to form a ring shape, and the micro electrode pieces are used for collecting EOG signals. In practical application, 8 micro-electrode sheets are arranged; the second image acquisition device adopts an eye tracker, in particular an infrared camera; the first communication module and the second communication module both adopt 5G communication modules, wireless transmission is carried out by utilizing a 5G instant communication technology, a base station deployment mode integrating a wireless access network and a backhaul is adopted, more flexible and low-cost base station deployment is realized, and real-time backhaul of signals is realized by means of a 5G link, so that time delay is reduced and higher real-time performance is achieved.
In this embodiment, the first image capturing device is used to capture an image of an environment including limbs of a user, and the second image capturing device is used to capture an image of a face of the user, wherein the second image capturing device is disposed on the VR glasses. During practical application, the first image acquisition device adopts a multi-view camera.
The user processor is used for creating a general body model and a face model, and sending the general body model and the face model to the cluster processing device through the first communication module.
In this embodiment, the user processor is further configured to recognize the gesture and the expression of the user to obtain gesture tracking information and an expression recognition type. Specifically, an environment image containing the limbs of a user is separated to obtain a user posture image, head tracking, hand tracking and limb movement tracking of the user are carried out according to the user posture image to obtain posture tracking information, an EOG signal collected by a microelectrode is received, expression tracking of the user is carried out according to a face image of the user to obtain eyeball characteristics, expression recognition types are obtained by combining the eyeball characteristics and the EOG signal, the posture tracking information and the expression recognition types are sent to a cluster processing device through a first communication module, and the posture information comprises head posture information, hand posture information and limb posture information.
In this embodiment, the cluster processing device is used as an edge computing processing device connected to a plurality of adjacent communication rooms, and computes a large amount of data in real time, and is specifically deployed in edge computer rooms in various places. The cluster processing device is used for receiving the general body model, the face model, the posture tracking information and the expression recognition type, matching and combining the face model and the general body model to obtain a user simulation model, and forwarding the user simulation model to the communication party with the established communication connection through the cloud server.
In this embodiment, the display device of the VR glasses is used to show the user with the virtual reality-based generated environment scene model and the user simulation model.
In this embodiment, the cloud server stores a plurality of preset environment scene types, generates an environment scene model according to the preset environment scene types selected by the user through the user processor, matches a communication party connected with the current user, and sends the environment scene model, the user simulation model, the posture tracking information, and the expression recognition types to the cluster processing device connected with the communication party, where the communication party is one or any plurality of communication users who have established communication interaction with the current user. The cluster processing device connected with the communication party updates the gesture and the expression of the user simulation model according to the gesture tracking information and the expression recognition type, and the user processor of the communication party receives the updated user simulation model and displays the user simulation model through the display device of the VR glasses. In addition, the cluster processing device is also used for beautifying the user simulation model through rendering according to the preset beautifying type selected by the user for the image of the communication party. Wherein the beautifying treatment comprises face thinning, skin grinding and skin whitening. In actual application, the user processor receives the preset environment scene type input by the user and forwards the preset environment scene type to the cloud server through the cluster processing device.
In this embodiment, the user interaction device further includes a laser sensor and an ambient light sensor, the laser sensor and the ambient light sensor are disposed in the communication room, and the laser sensor and the ambient light sensor are respectively connected to the user processor.
The laser sensor is used for positioning the position of a user in a communication room, the user processor acquires the position of the user through the laser sensor, the position of the user is transmitted to the cloud server through the cluster processing device through the first communication module, the cloud server forwards the position of the user to the cluster processing device connected with the communication party, the communication party is one or any multiple communication users which establish communication interaction with the current user, the cluster processing device connected with the communication party updates the relative position of the user simulation model in the environment scene model according to the position of the user, the position updating information is sent to the user processor of the communication party, and the user processor is displayed through the display device.
The environment light sensor is used for sensing light conditions around users in a communication room, the user processor acquires user environment light information by using the environment light sensor, the user environment light information is transmitted to the cloud server through the cluster processing device through the first communication module, the cloud server forwards the user environment light information to the cluster processing device connected with the communication party, the communication party is one or any plurality of communication users who establish communication interaction with the current user, the display brightness of the environment scene model is adjusted by the user processor of the communication party according to the user environment light information, and therefore the comfort level of the user on visual experience is improved while energy conservation of VR glasses is achieved.
In practical application, 1 laser lighthouse base station is respectively arranged at two ends of a diagonal line of the user interaction device, each laser lighthouse base station is about 2m high, a laser sensor array for positioning is arranged in each laser lighthouse base station, and the laser sensor array and photosensitive sensors distributed on the head-wearing VR interaction device, the ground and the induction handle form a whole set of laser lighthouse laser positioning system; the first image acquisition devices are depth cameras which are 8 and are uniformly distributed above the periphery of the user interaction device; the ambient light sensor sets up 2, installs respectively on the wall of user interaction device both sides. The induction handle is a game handle.
Furthermore, those skilled in the art will appreciate that the communication party establishing the communication interaction is one or any of a plurality of user interaction devices communicating through the cloud server.
It should be noted that, a person skilled in the art may replace the VR glasses with other head-mounted VR interaction devices according to actual situations, and set the micro-electrode sheet, the display device, the second image capturing device, and the second communication module on the interaction device accordingly, which is not limited herein.
In this embodiment, from the perspective of the VR device end, the VR device can be divided into three parts, the first part is a display technology, and based on knowledge related to computer graphics and an operating system, the device engine renders a VR picture through a low-latency rendering pipeline, a fixation point rendering, a light field display, and the like, and then performs on-screen display after computer processing processes such as inverse distortion, synthesis, position prediction, and the like. In order to improve the rendering efficiency, a point-of-regard rendering technology needs to be introduced. The actions operated by each user can be fed back into the VR space in real time, and each user participating in the interaction can observe the actions of others in real time. When the user visually reaches a certain position in the three-dimensional space, rays emitted by the main camera on the VR glasses hit an object which is visually involved by the user, and the VR system can respond to, activate and display the hit object in real time. In addition, related technologies of computer graphics, such as rendering technologies, can be used for beautifying the face and body images according to preset beautifying types selected by users, so that the requirements of the users on the image generation can be set as much as possible; the second part is a positioning technology, and in view of the high real-time performance of the VR real-time communication interaction system combining 5G, expression tracking and beauty, an external laser positioning mode such as HTC five is used, and after a transmitter scans a space, the time difference of laser signals received by a helmet and an induction handle is calculated, so that the space coordinate of the equipment is deduced. The touch feedback is carried out by means of external auxiliary props such as an induction handle, the specific grasping force is obtained by arranging a pressure sensor on the external auxiliary props, and the more accurate and complex interaction action is realized by utilizing buttons and vibration feedback arranged on the induction handle through 6-degree-of-freedom space tracking with two separated hands, namely 3 rotational degrees of freedom and 3 translational degrees of freedom, so that the interaction of each user and a virtual environment in full dimension is realized; the third part is recognition technology, in order to quickly and accurately establish the 3D models of the two interactive parties, prior information such as body data and facial data is collected in advance by combining a VR real-time communication interactive system with 5G, expression tracking and beauty, a general character model of a related character group is established, and gesture recognition, body posture recognition, expression recognition, eye movement tracking and the like are carried out by wearing gloves or using an induction handle.
It should be noted that, as will be understood by those skilled in the art, the VR real-time communication interactive system combining 5G, expression tracking and beauty can be used not only in an application scenario of instant call such as a telephone booth, but also in an application scenario of entertainment, and the embodiment is not limited herein.
Example 2
Referring to fig. 3 and 4, the present embodiment provides a VR real-time communication interaction method combining 5G, expression tracking and beauty, including the following steps:
model building steps: creating a general body model and a face model, and matching and combining the face model and the general body model to obtain a user simulation model;
a scene building step: selecting a target scene type, and generating an environment scene model;
a data acquisition step: acquiring an environment image containing limbs of a user, a face image of the user and an EOG signal;
updating model information: separating an environment image containing user limbs to obtain a user posture image, tracking the head, the hand and the limb movement of a user according to the user posture image to obtain posture tracking information, tracking the expression of the user according to a user face image to obtain eyeball characteristics, and inputting the eyeball characteristics and an EOG signal into an expression recognition model to predict to obtain an expression recognition type;
the expression recognition model is obtained by performing machine learning through multiple groups of training data, each group of training data comprises expression features and label information for identifying expression recognition types corresponding to the expression features, the expression recognition types are identifiers of multiple expression types, and the expression types specifically comprise anger, pain, disgust, sadness, fear, surprise, happiness and joy.
User synchronization step: and matching a communication party connected with the current user, sending the environment scene model, the user simulation model, the posture tracking information and the expression recognition type to the communication party, and updating the posture and the expression of the user simulation model by the communication party according to the posture tracking information and the expression recognition type.
A step of image formation adjustment: and beautifying the user simulation model through rendering according to the preset beautifying type selected by the user for the image of the communication party, so as to further improve the interactive experience of the user. Wherein the beautifying treatment comprises face thinning, skin grinding and skin whitening.
In this embodiment, in order to further improve the real feeling of the user interaction experience, the user synchronization step further includes: the user position is obtained by utilizing the laser sensor and adopting a lighthouse laser positioning method, and the relative position of the user simulation model in the environmental scene model is updated according to the user position, so that the synchronization effect of small-range walking of the user in a space range is met.
In this embodiment, the VR real-time communication interaction method combining 5G, expression tracking and beauty further includes an ambient light adjustment step, where the ambient light adjustment step specifically includes: the user processor acquires user environment light information by using the environment light sensor, and adjusts the display brightness of the environment scene model according to the user environment light information, so that the energy of VR glasses is saved, and the comfort level of a user in visual experience is improved.
In this embodiment, the model building step includes the following specific steps:
generating a universal body model based on the user of the matched user classification label, wherein the user classification label comprises age, gender, race and height;
extracting face front image information and depth feature points from a user face image acquired by a second image acquisition device, establishing a face general model based on the face front image information, matching the depth feature points with the face general model, and smoothing to obtain a face model;
and loading the face model to the general body model, and matching and combining the face model and the general body model to obtain the user simulation model.
In this embodiment, the construction process of the face modeling system of the face model is as follows:
using a high-resolution camera to respectively shoot a plurality of high-resolution face photos with different face characteristics from the side and the front, wherein the face characteristics comprise age, skin color, mouth shape, nose shape and the like;
marking the characteristic points of the human face in the photo by using a manual marking method, wherein the characteristics comprise but are not limited to contour points and five sense organ position points of the human face;
introducing a plurality of marked faces into a unified topological structure to obtain an original training set;
importing an original training set into a neural network based on GAN for training to obtain a group of mapping functions from two-dimensional photos based on human faces to three-dimensional human face models, wherein the training process specifically comprises the following steps: obtaining a 3-dimensional face image with simple texture according to the input high-resolution image by using a 3MSS algorithm; extracting depth information by using a super-resolution model to further improve the resolution of the input image; and updating the 3-dimensional face image by using the depth information to obtain a 3-dimensional face model with accurate textures, and finally obtaining a mapping relation between the input image and the 3-dimensional face model with the accurate textures.
In practical application, the scene building step adopts a pre-constructed mode, and achieves balance of data volume and precision by using professional modeling tools such as PDMS and SP3D and combining a deep learning algorithm. The three-dimensional factory design system PDMS establishes an element library covering the required element size information and a callable level library which is in certain correlation with the element library to form an accurate and comprehensive independent database. The method comprises the steps of establishing equipment, an attached ladder stand, an attached platform and the like in a parameterization mode by setting the name, the type and the placement point of the equipment, and carrying out construction adjustment, additional pipeline support and various information of an injection pipeline according to related commands of the pipeline on the PID, so that the establishment of a model and the output of a plan view are realized. In the interactive design process, collision check is carried out in real time, and errors generated in the pipeline branch generation are timely found and corrected on the premise of consistent data. The advantages of network resource sharing and interactive communication are fully utilized, the high real-time performance of the VR real-time communication interactive system combined with 5G, expression tracking and beauty is matched, secondary development is supported according to VR environmental requirements, and the dynamic model information system with all-round overall optimization and reasonable layout is realized. SP3D has an automatic function and a rule-driven technology, and remarkably improves the production efficiency, design and engineering quality by cooperatively designing various specialties and performing unified management on data. After the relevant parameters constructed by the model are summarized into the table pipeline table for processing, the table pipeline table is introduced into the pipeline system at one time, processing tasks are planned in batches, and the pipeline system is established efficiently. The comprehensive collision detection of the server side and the real-time collision detection in the working process are used for respectively realizing the detection of a VR real-time communication interaction system combining 5G, expression tracking and beauty, the positioning of collision points and the real-time collision detection in the working process.
When constructing the scene, the method comprises the following steps:
a. selecting a plurality of typical objects as materials of a model library;
b. b, shooting a high-precision image of the material in the step a to obtain parameters such as a real-scene texture shape and the like;
c. obtaining accurate data of the size of each material by measuring the data of each material on site;
d. modeling the typical objects by using 3DMAX, SP3D and PDMS to construct a model library;
e. b, acquiring high-precision pictures of the rest possible texture shapes of the typical object in the step a by shooting;
f. using the picture in the step e as a training set, and training a Convolutional Neural Network (CNN) as a style migration network;
g. carrying out style migration on the model in the model library in the step d by using the convolutional neural network obtained by training in the step f to obtain a model expansion material library with different textures
e. And selecting a proper model from the material library and importing the model into unity to complete the design and construction of the 3D scene.
Based on a model obtained by combining the face information and the body information, the face beautifying effect is realized by using the rendering technology of computer graphics. The face thinning function is realized by shifting the pixel positions: and calculating the coordinates before deformation based on the inverse change of the coordinates after deformation, and interpolating to obtain RGB pixel values of the coordinates and using the RGB pixel values as pixel values of the coordinates for use after deformation. And carrying out skin color detection on the image, and realizing the skin grinding effect by utilizing various filtering algorithms. For example, the weight information composed of the gaussian component and the gradient component blurs and smoothes but can retain the bilateral filtering of the edge; and guiding filtering with high speed based on mean convolution, and the like. By raising the value of R, G, B, A in the image to some extent, the image whitening effect is achieved. The user can adjust the beautification degree according to the self requirement.
In this embodiment, the method of obtaining the gesture tracking information by performing the head tracking, the hand tracking, and the limb movement tracking of the user according to the user gesture image specifically includes performing motion capture and gesture estimation by using a segmented 3D gesture estimation method, where the 3D gesture estimation method includes the following steps:
acquiring depth image information and plane information of a current camera shooting area by using a depth camera;
extracting the plane information by using a VGG network to obtain a characteristic value;
the method comprises the following steps of completing person target tracking of a single person or a plurality of persons by using a target tracking algorithm based on deep learning: inputting the extracted characteristic value, predicting the 2D joint position in real time on a human target rectangular frame by using a CNN network to obtain corresponding Heatmaps-H{j,t}；
Based on Heatmaps-H{j,t}Extracting the connection relation of each joint from the depth image information;
will Heatmaps-H{j,t}And the connection relation of each joint is used as input, and the mapping of the 2D picture pixels and the 3D surface model points is carried out by utilizing a pre-trained RCNN to obtain the corresponding 3D joint positions;
integration of 3D joint positions to yield location-maps-X{j,t}、Y{j,t}、Z{j,t}(ii) a Wherein j represents a joint point, and t represents time;
and extracting and obtaining 2D key points and 3D posture estimation based on Heatmaps and localization maps, and combining the two to obtain final motion capture and posture estimation data, thereby completing the 3D human body posture estimation and motion real-time capture.
In practical application, data of motion capture and attitude estimation are fed back to simulation software such as CATIA, DELMIA and JACK in real time, and updating of the model and realization of interactive action are completed by calculating the object relationship between the current task model and the background model.
During practical application, the expression recognition model is obtained by machine learning through a plurality of groups of training data, and the specific process comprises the following steps:
selecting the pupil diameter and the fixation frequency as eyeball characteristics, and reducing the influence of illumination on the change of the pupil diameter by constructing an illumination reflection model;
filtering the EOG signal under a 0.05-50 Hz band-pass filter to obtain the EOG signal with power spectrum change conditions under different scanning frequencies;
acquiring eye characteristics and EOG signals of anger, pain, disgust, sadness, fear, surprise, happiness and joy through different external video and sound tests, and forming emotion training set data by taking corresponding expression types as label information;
and finding out the optimal characteristic value of the two kinds of characteristic data synthesis corresponding to different emotions through an SVM classifier to complete the classification of eight kinds of basic emotions to obtain an expression recognition model.
In the subsequent process of using the expression recognition model, the facial expression of the person can be updated in real time only by judging according to the classification result and the data received by the sensor to obtain the expression with the maximum probability.
It should be noted that, as will be understood by those skilled in the art, the VR real-time communication interaction method combining 5G, expression tracking and beauty can be used not only in an application scenario of an instant call such as a telephone booth, but also in an application scenario of an entertainment activity, and the embodiment is not limited herein.
The above embodiments are preferred embodiments of the present invention, but the present invention is not limited to the above embodiments, and any other changes, modifications, substitutions, combinations, and simplifications which do not depart from the spirit and principle of the present invention should be construed as equivalents thereof, and all such changes, modifications, substitutions, combinations, and simplifications are intended to be included in the scope of the present invention.
Claims (10)
1. A VR real-time communication interaction system combining 5G, expression tracking and beauty is provided with a cloud server, a plurality of cluster processing devices, a plurality of communication rooms and a plurality of user interaction devices, wherein the cloud server is respectively connected with the cluster processing devices, each cluster processing device is respectively connected with a plurality of adjacent user interaction devices so as to form a plurality of area processing stations, and each user interaction device is arranged in a corresponding communication room;
the head-mounted VR interaction device is provided with a display device, a second image acquisition device, a micro electrode plate and a second communication module, wherein the micro electrode plate, the display device and the second image acquisition device are respectively connected with the second communication module, and the second communication module is also connected with the first communication module in a wireless manner;
the microelectrode sheet is used for collecting EOG signals;
the first image acquisition device is used for acquiring an environment image containing limbs of a user, the second image acquisition device is used for acquiring a face image of the user, and the second image acquisition device is arranged on the head-mounted VR interaction device;
the user processor is used for creating a general body model and a face model, recognizing the gesture and the expression of a user to obtain gesture tracking information and an expression recognition type, wherein the gesture information comprises head gesture information, hand gesture information and limb gesture information;
the cloud server is used for forwarding data and pre-building an environment scene model and a user simulation model;
the display device is used for displaying the environmental scene model and the user simulation model;
and the cluster processing device connected with the communication party updates the gesture and the expression of the user simulation model according to the gesture tracking information and the expression recognition type.
2. The VR real-time communication interaction system of claim 1 in combination with 5G, emotion tracking and beauty, wherein the cloud server is configured to match a correspondent party that establishes a connection with a current user, the correspondent party being one or any of a plurality of correspondent users that have established communication interaction with the current user;
the user processor receives preset environment scene types input by a user, the preset environment scene types are forwarded to the cloud server through the cluster processing device, the cloud server stores a plurality of preset environment scene types, the cloud server generates an environment scene model according to the preset environment scene types selected by the user through the user processor, a communication party connected with the current user is matched, and the environment scene model, the user simulation model, the posture tracking information and the expression recognition types are sent to the cluster processing device connected with the communication party.
3. The VR real-time communication interaction system combining 5G, expression tracking and beauty as claimed in claim 1, wherein the user interaction device further comprises a laser sensor disposed in the communication room, the laser sensor coupled to the user processor;
the laser sensor is used for positioning the position of a user in a communication room, the user processor acquires the user position through the laser sensor, the user position is transmitted to the cloud server through the cluster processing device through the first communication module, the cloud server forwards the user position to the cluster processing device connected with a communication party, the communication party is one or any plurality of communication users which establish communication interaction with the current user, the cluster processing device connected with the communication party updates the relative position of the user simulation model in the environment scene model according to the user position, and the position updating information is sent to the user processor of the communication party.
4. The VR real-time communication interaction system of claim 3 combining 5G, expression tracking and beauty, wherein the user interaction device further comprises a photosensitive sensor, a laser lighthouse base station and a sensing handle, the laser lighthouse base station is arranged at two ends of a diagonal line of the user interaction device, a laser sensor array for positioning is arranged in the laser lighthouse base station, the photosensitive sensor is arranged on the ground of the communication room and the sensing handle, and therefore the laser sensor array and the photosensitive sensor form a laser lighthouse laser positioning system.
5. The VR real-time communication interaction system with 5G, expression tracking and beauty as claimed in claim 1, wherein the user interaction device further comprises an ambient light sensor, the ambient light sensor is disposed in the communication room, the ambient light sensor is connected to the user processor;
the communication system comprises an ambient light sensor, a user processor, a first communication module, a cloud server and a communication party, wherein the ambient light sensor is used for sensing the light conditions around users in a communication room, the user processor acquires ambient light information of the users by using the ambient light sensor, the ambient light information of the users is transmitted to the cloud server through the cluster processing device through the first communication module, the cloud server forwards the ambient light information of the users to the cluster processing device connected with the communication party, the communication party is one or any plurality of communication users which have established communication interaction with the current user, and the display brightness of an ambient scene model is adjusted by the user processor of the communication party according to the ambient light information of the users.
6. The VR real-time communication interaction system of claim 1 wherein the cluster processing device and the user interaction devices are in a one-to-many relationship, and data of each user interaction device in an area is resolved and distributed by one cluster processing device deployed in the area.
7. The VR real-time communication interaction system combining 5G, expression tracking and beauty as claimed in claim 1, wherein the head-mounted VR interaction device further comprises a mirror frame, the micro electrode pads, the display device, the second image capture device and the second communication module are all disposed on the mirror frame, and the mirror frame is provided with a plurality of foam areas for mounting the micro electrode pads.
8. A VR real-time communication interaction method combining 5G, expression tracking and beauty is characterized by comprising the following steps:
model building steps: creating a general body model and a face model, and matching and combining the face model and the general body model to obtain a user simulation model;
a scene building step: selecting a target scene type, and generating an environment scene model;
a data acquisition step: acquiring an environment image containing limbs of a user, a face image of the user and an EOG signal;
updating model information: separating an environment image containing user limbs to obtain a user posture image, tracking the head, the hand and the limb movement of a user according to the user posture image to obtain posture tracking information, tracking the expression of the user according to a user face image to obtain eyeball characteristics, and inputting the eyeball characteristics and an EOG signal into an expression recognition model to predict to obtain an expression recognition type;
the expression recognition model is obtained by machine learning through a plurality of groups of training data, each group of training data comprises expression characteristics and label information for identifying expression recognition types corresponding to the expression characteristics, and the expression recognition types are identifiers of a plurality of expression types;
user synchronization step: matching a communication party connected with the current user, sending the environment scene model, the user simulation model, the posture tracking information and the expression recognition type to the communication party, and updating the posture and the expression of the user simulation model by the communication party according to the posture tracking information and the expression recognition type;
a step of image formation adjustment: and beautifying the user simulation model by rendering according to the preset beautifying type selected by the user, wherein the beautifying comprises face thinning, skin grinding and whitening.
9. The VR real-time communication interaction method of claim 8, wherein the expression recognition model is obtained by machine learning through multiple sets of training data, and the specific process includes:
selecting the pupil diameter and the fixation frequency as eyeball characteristics, and reducing the influence of illumination on the change of the pupil diameter by constructing an illumination reflection model;
filtering the EOG signal under a 0.05-50 Hz band-pass filter to obtain the EOG signal with power spectrum change conditions under different scanning frequencies;
acquiring eye characteristics and EOG signals of anger, pain, disgust, sadness, fear, surprise, happiness and joy through different external video and sound tests, and forming emotion training set data by taking corresponding expression types as label information;
and finding the optimal characteristic value of the two kinds of characteristic data synthesis corresponding to different emotions through an SVM classifier to complete the classification of the expression recognition types to obtain an expression recognition model.
10. The VR real-time communication interaction method of claim 8, wherein the obtaining of the pose tracking information by tracking the head, hand and limb movements of the user according to the pose image of the user is performed by performing motion capture and pose estimation using a segmented 3D pose estimation method, and the 3D pose estimation method comprises the following steps:
acquiring depth image information and plane information of a current camera shooting area by using a depth camera;
extracting the plane information by using a VGG network to obtain a characteristic value;
the method comprises the following steps of completing person target tracking of a single person or a plurality of persons by using a target tracking algorithm based on deep learning: input extractedCharacteristic value, using CNN network to predict 2D joint position in real time to obtain corresponding Heatmaps-H{j，t}；
Based on Heatmaps-H{j，t}Extracting the connection relation of each joint from the depth image information;
will Heatmaps-H{j，t}And the connection relation of each joint is used as input, and the mapping of the 2D picture pixels and the 3D surface model points is carried out by utilizing a pre-trained RCNN to obtain the corresponding 3D joint positions;
integration of 3D joint positions to yield location-maps-X{j，t}、Y{j，t}、Z{j，t}(ii) a Wherein j represents a joint point, and t represents time;
and extracting based on Heatmaps and localization maps to obtain 2D key points and 3D posture estimation, and combining the two to obtain final motion capture and posture estimation data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110918746.0A CN113610018A (en) | 2021-08-11 | 2021-08-11 | VR real-time communication interactive system and method combining 5G, expression tracking and beautifying |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110918746.0A CN113610018A (en) | 2021-08-11 | 2021-08-11 | VR real-time communication interactive system and method combining 5G, expression tracking and beautifying |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113610018A true CN113610018A (en) | 2021-11-05 |
Family
ID=78340270
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110918746.0A Pending CN113610018A (en) | 2021-08-11 | 2021-08-11 | VR real-time communication interactive system and method combining 5G, expression tracking and beautifying |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113610018A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115463413A (en) * | 2022-09-30 | 2022-12-13 | 小派科技（上海）有限责任公司 | Interaction device, control method and interaction system applied to multi-person interaction scene |
| CN116546182A (en) * | 2023-07-05 | 2023-08-04 | 中数元宇数字科技(上海)有限公司 | Video processing method, device, equipment and storage medium |
-
2021
- 2021-08-11 CN CN202110918746.0A patent/CN113610018A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115463413A (en) * | 2022-09-30 | 2022-12-13 | 小派科技（上海）有限责任公司 | Interaction device, control method and interaction system applied to multi-person interaction scene |
| CN116546182A (en) * | 2023-07-05 | 2023-08-04 | 中数元宇数字科技(上海)有限公司 | Video processing method, device, equipment and storage medium |
| CN116546182B (en) * | 2023-07-05 | 2023-09-12 | 中数元宇数字科技(上海)有限公司 | Video processing method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11262841B2 (en) | Wireless wrist computing and control device and method for 3D imaging, mapping, networking and interfacing | |
| US20210358214A1 (en) | Matching meshes for virtual avatars | |
| US20190082211A1 (en) | Producing realistic body movement using body Images | |
| US20210177124A1 (en) | Information processing apparatus, information processing method, and computer-readable storage medium | |
| US20230123433A1 (en) | Artificial intelligence-based animation character drive method and related apparatus | |
| CN105608746B (en) | A method of reality is subjected to Virtual Realization | |
| CN117572954A (en) | System and method for augmented reality | |
| KR20180100476A (en) | Virtual reality-based apparatus and method to generate a three dimensional(3d) human face model using image and depth data | |
| CN114766038A (en) | Individual views in a shared space | |
| KR20230044041A (en) | System and method for augmented and virtual reality | |
| KR102148151B1 (en) | Intelligent chat based on digital communication network | |
| CN113610018A (en) | VR real-time communication interactive system and method combining 5G, expression tracking and beautifying | |
| CN115909015B (en) | Method and device for constructing deformable nerve radiation field network | |
| CN111383313B (en) | Virtual model rendering method, device, equipment and readable storage medium | |
| CN112330753A (en) | Target detection method of augmented reality system | |
| CN108416255B (en) | System and method for capturing real-time facial expression animation of character based on three-dimensional animation | |
| CN109685911A (en) | A kind of the AR glasses and its implementation of achievable virtual fitting | |
| CN108830901A (en) | A kind of image processing method and electronic equipment | |
| CN104715505A (en) | Three-dimensional head portrait generating system and generating device and generating method thereof | |
| CN113342167B (en) | Space interaction AR realization method and system based on multi-person visual angle positioning | |
| CN107527033A (en) | Camera module and social intercourse system | |
| CN117765157A (en) | Virtual character expression driving method and reconstruction equipment | |
| CN116129487A (en) | Three-dimensional image pronunciation head posture simulation method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |