CN116880701A

CN116880701A - Multimode interaction method and system based on holographic equipment

Info

Publication number: CN116880701A
Application number: CN202311148693.4A
Authority: CN
Inventors: 张雪兵
Original assignee: Shenzhen Euclideon Technology Co ltd
Current assignee: Shenzhen Euclideon Technology Co ltd
Priority date: 2023-09-07
Filing date: 2023-09-07
Publication date: 2023-10-13
Anticipated expiration: 2043-09-07
Also published as: CN116880701B

Abstract

The invention relates to the technical field of holographic images, and discloses a multi-mode interaction method and system based on holographic equipment, which are used for improving display timeliness and display accuracy when multi-mode interaction is performed based on the holographic equipment. Comprising the following steps: collecting infrared reflection signals to obtain infrared reflection signals; collecting a target visual line and a brain wave data set; performing time sequence alignment processing to obtain target fusion data; extracting multi-type feature data to obtain position feature data, concerned position feature data and emotion feature data; space division is carried out on the three-dimensional display space to obtain a target space region; generating an interaction instruction for the voice control instruction to generate a target interaction instruction; constructing holographic display space coordinates, generating a holographic display space coordinate set, and carrying out holographic equipment matching to determine a target holographic equipment set; and constructing a three-dimensional virtual reality space, and transmitting the three-dimensional virtual reality space to the virtual reality glasses for holographic image display.

Description

Multimode interaction method and system based on holographic equipment

Technical Field

The invention relates to the technical field of holographic images, in particular to a multimode interaction method and system based on holographic equipment.

Background

With the continuous development of technology, there is an increasing demand for interactive experience and immersive technology. The multi-mode interaction technology based on the holographic equipment provides a brand new interaction mode, and more real and natural interaction experience can be realized in the virtual reality space.

However, in the processing of multi-modal data, time series alignment is a critical step in the prior art for aligning the time points of different data sources. However, existing alignment algorithms may suffer from insufficient accuracy in processing complex multimodal data, requiring more accurate and reliable alignment methods. The existing feature extraction method and analysis model may have the problems of non-ideal extraction effect or insufficient feature fusion under the condition of multi-modal data,

disclosure of Invention

The invention provides a multi-mode interaction method and a multi-mode interaction system based on holographic equipment, which are used for improving display timeliness and display accuracy when multi-mode interaction is performed based on the holographic equipment.

The first aspect of the present invention provides a multi-modal interaction method based on holographic equipment, the multi-modal interaction method based on holographic equipment comprising: collecting infrared reflection signals of a handheld handle device of a target user to obtain infrared reflection signals;

Acquiring a visual line of the target user through preset virtual reality glasses to obtain a target visual line, and acquiring brain wave data of the target user through the virtual reality glasses to obtain a brain wave data set;

performing time sequence alignment processing on the infrared reflection signals, the target visual line and the brain wave data set to obtain target fusion data;

extracting multi-type feature data from the target fusion data to obtain position feature data, concerned position feature data and emotion feature data;

space division is carried out on a preset three-dimensional display space through the position characteristic data and the concerned position characteristic data, so that a corresponding target space region is obtained;

collecting a voice control instruction of the target user, and generating an interaction instruction for the voice control instruction through the emotion characteristic data to generate a target interaction instruction;

constructing holographic display space coordinates through the target interaction instruction, generating a holographic display space coordinate set, carrying out holographic equipment matching through the target interaction instruction, and determining a target holographic equipment set;

and constructing a three-dimensional virtual reality space through the target holographic equipment set based on the holographic display space coordinate set, and transmitting the three-dimensional virtual reality space to virtual reality glasses of the target user for holographic image display.

With reference to the first aspect, in a first implementation manner of the first aspect of the present invention, performing a time sequence alignment process on the infrared reflection signal, the target visual line, and the brain wave data set to obtain target fusion data includes:

respectively carrying out time stamp calibration on the infrared reflection signal, the target visual line and the brain wave data set, and determining a first initial time stamp corresponding to the infrared reflection signal, a second initial time stamp corresponding to the target visual line and a third initial time stamp corresponding to the brain wave data set;

performing time axis analysis on the first initial time stamp, the second initial time stamp and the third initial time stamp to determine at least one time axis data;

performing time axis alignment processing based on linear interpolation on the at least one time axis data to determine a target time axis;

and based on the target time axis, carrying out data fusion on the infrared reflection signal, the target visual line and the brain wave data set to generate target fusion data.

With reference to the first aspect, in a second implementation manner of the first aspect of the present invention, the extracting the multi-type feature data from the target fusion data to obtain location feature data, attention location feature data, and emotion feature data includes:

Performing data filtering processing on the target fusion data to generate filtering fusion data;

carrying out data layering on the filtering fusion data to determine multi-layer filtering fusion data to be processed;

and respectively extracting multi-type feature data of each layer of the filtering fusion data to be processed to obtain the position feature data, the concerned position feature data and the emotion feature data.

With reference to the second implementation manner of the first aspect, in a third implementation manner of the first aspect of the present invention, the extracting multi-type feature data of each layer of the filtering fusion data to be processed to obtain the location feature data, the attention location feature data, and the emotion feature data includes:

carrying out hierarchical standard matching on each layer of the filtering fusion data to be processed, and determining the data type corresponding to each layer of the filtering fusion data to be processed;

performing feature extraction algorithm matching based on the data types corresponding to the to-be-processed filter fusion data of each layer, and determining a feature extraction algorithm corresponding to the to-be-processed filter fusion data of each layer;

and extracting multi-type feature data of each layer of the filtering fusion data to be processed through a feature extraction algorithm corresponding to each layer of the filtering fusion data to be processed, so as to obtain the position feature data, the concerned position feature data and the emotion feature data.

With reference to the first aspect, in a fourth implementation manner of the first aspect of the present invention, the performing spatial segmentation on a preset three-dimensional display space by using the position feature data and the attention position feature data to obtain a corresponding target spatial region includes:

constructing the three-dimensional virtual point cloud of the position characteristic data to obtain a first candidate three-dimensional virtual point cloud;

performing three-dimensional virtual point cloud construction on the concerned position feature data to obtain a second candidate three-dimensional virtual point cloud;

performing virtual three-dimensional space mapping on the first candidate three-dimensional virtual point cloud to obtain a first virtual three-dimensional space;

performing virtual three-dimensional space mapping on the second candidate three-dimensional virtual point cloud to obtain a second virtual three-dimensional space;

performing space division point matching on the first virtual three-dimensional space and the second virtual three-dimensional space to determine a space division point set;

and carrying out space division on the three-dimensional display space based on the space division point set to obtain the target space region.

With reference to the first aspect, in a fifth implementation manner of the first aspect of the present invention, the constructing holographic display space coordinates by the target interaction instruction, generating a holographic display space coordinate set, and performing holographic device matching by the target interaction instruction, to determine a target holographic device set includes:

Performing interaction position mapping on the target interaction instruction to determine interaction position data;

performing coordinate mapping on the interaction position data based on a preset space coordinate system to generate the holographic display space coordinate set;

performing equipment identification analysis on the target interaction instruction to determine a target equipment identification set;

and carrying out holographic equipment matching through the target equipment identification set, and determining the target holographic equipment set.

With reference to the first aspect, in a sixth implementation manner of the first aspect of the present invention, the constructing, by the target holographic device set, a three-dimensional virtual reality space based on the holographic display space coordinate set, and transmitting the three-dimensional virtual reality space to virtual reality glasses of the target user for holographic image display includes:

extracting geometric elements from the holographic display space coordinate set through the target holographic equipment set to determine a geometric element set;

generating a virtual space of the geometric element set to obtain the three-dimensional virtual reality space;

and transmitting the three-dimensional virtual reality space to virtual reality glasses of the target user for holographic image display.

The second aspect of the present invention provides a multi-modal interaction apparatus based on a holographic device, the multi-modal interaction apparatus based on a holographic device comprising:

the first acquisition module is used for acquiring infrared reflection signals of the handheld handle equipment of the target user to obtain infrared reflection signals;

the second acquisition module is used for acquiring the visual line of the target user through preset virtual reality glasses to obtain a target visual line, and acquiring brain wave data of the target user through the virtual reality glasses to obtain a brain wave data set;

the alignment module is used for carrying out time sequence alignment processing on the infrared reflection signals, the target visual line and the brain wave data set to obtain target fusion data;

the extraction module is used for extracting multi-type feature data of the target fusion data to obtain position feature data, concerned position feature data and emotion feature data;

the segmentation module is used for carrying out space segmentation on a preset three-dimensional display space through the position characteristic data and the concerned position characteristic data to obtain a corresponding target space region;

the generating module is used for collecting the voice control instruction of the target user, generating the interaction instruction of the voice control instruction through the emotion characteristic data and generating a target interaction instruction;

The matching module is used for constructing holographic display space coordinates through the target interaction instruction, generating a holographic display space coordinate set, carrying out holographic equipment matching through the target interaction instruction, and determining a target holographic equipment set;

and the display module is used for constructing a three-dimensional virtual reality space through the target holographic equipment set based on the holographic display space coordinate set, and transmitting the three-dimensional virtual reality space to the virtual reality glasses of the target user for holographic image display.

A third aspect of the present invention provides a multi-modal interaction device based on holographic devices, comprising: a memory and at least one processor, the memory having instructions stored therein; the at least one processor invokes the instructions in the memory to cause the holographic device-based multimodal interaction apparatus to perform the holographic device-based multimodal interaction method described above.

A fourth aspect of the invention provides a computer readable storage medium having instructions stored therein which, when run on a computer, cause the computer to perform the above-described holographic device-based multimodal interaction method.

In the technical scheme provided by the application, the infrared reflection signals are acquired to obtain the infrared reflection signals; acquiring a visual line of a target user to obtain a target visual line, and acquiring brain wave data of the target user to obtain a brain wave data set; performing time sequence alignment processing to obtain target fusion data; extracting multi-type feature data from the target fusion data to obtain position feature data, concerned position feature data and emotion feature data; space division is carried out on the three-dimensional display space, and a target space area is obtained; generating interactive instructions for the voice control instructions through the emotion characteristic data to generate target interactive instructions; constructing holographic display space coordinates, generating a holographic display space coordinate set, carrying out holographic equipment matching, and determining a target holographic equipment set; and constructing a three-dimensional virtual reality space, and transmitting the three-dimensional virtual reality space to virtual reality glasses of a target user for holographic image display. In the embodiment of the application, the more immersive interaction experience is realized through the comprehensive application of the multi-mode data such as the infrared reflection signal acquisition, the visual line acquisition, the brain wave data acquisition and the like. The user interacts with the virtual environment in real time through the modes of the handle equipment, the visual line, the brain waves and the like, and more natural and real interaction experience is provided. And extracting multi-type feature data from the target fusion data to obtain position feature data, concerned position feature data and emotion feature data. These feature data can be used for deeper user behavior analysis, emotion recognition and personalized interactions, providing more personalized, intelligent services. And obtaining a target space region by space division of the three-dimensional display space. The virtual scene is divided into different areas, so that the interactive behavior of the user can be conveniently positioned and managed, and more accurate and targeted interactive experience is provided. And generating an interaction instruction for the voice control instruction through the emotion characteristic data, generating a target interaction instruction, carrying out holographic equipment matching, and determining a target holographic equipment set. This helps to intelligently select the appropriate devices and interaction modes according to the emotional state and voice instructions of the user, providing a personalized, intelligent interaction experience. The three-dimensional virtual reality space is constructed and transmitted to the virtual reality glasses of the target user to display the holographic image, so that the holographic image display of realism and immersion is realized. The user views and interacts with the holographic image through the virtual reality glasses, providing a more realistic and vivid visual experience.

Drawings

FIG. 1 is a schematic diagram of one embodiment of a multi-modal interaction method based on holographic devices in an embodiment of the present invention;

FIG. 2 is a flowchart of extracting multi-type feature data from target fusion data according to an embodiment of the present invention;

FIG. 3 is a flowchart of extracting multiple types of feature data for each layer of filtering fusion data to be processed according to an embodiment of the present invention;

FIG. 4 is a flowchart of the spatial division of a preset three-dimensional display space by position feature data and attention position feature data according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of an embodiment of a multi-modal interaction means based on holographic devices in an embodiment of the present invention;

FIG. 6 is a schematic diagram of an embodiment of a multi-modal interaction device based on holographic devices in an embodiment of the invention.

Detailed Description

The embodiment of the invention provides a multi-mode interaction method and a multi-mode interaction system based on holographic equipment, which are used for improving display timeliness and display accuracy when multi-mode interaction is performed based on the holographic equipment.

The terms "first," "second," "third," "fourth" and the like in the description and in the claims and in the above drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments described herein may be implemented in other sequences than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed or inherent to such process, method, article, or apparatus.

For ease of understanding, a specific flow of an embodiment of the present invention is described below with reference to fig. 1, where an embodiment of a multi-modal interaction method based on a holographic device in an embodiment of the present invention includes:

s101, collecting infrared reflection signals of a handheld handle device of a target user to obtain infrared reflection signals;

it is to be understood that the execution subject of the present invention may be a multi-modal interaction device based on a holographic device, and may also be a terminal or a server, which is not limited herein. The embodiment of the invention is described by taking a server as an execution main body as an example.

Specifically, an infrared emitter and an infrared sensor are mounted on the handle device. These hardware modules will be used to send and receive infrared signals. A specific set of infrared emission signal sequences is designed to represent different handle actions or states. For example, pressing button a and button B may have different sequences of infrared emission signals. When a user uses the handle device, the infrared transmitter transmits a specific sequence of infrared signals that are reflected at the surface of the handle. The infrared sensor will receive these reflected signals and convert them into electrical signals. The electrical signals collected by the infrared sensors are processed and decoded to extract information of the transmitted infrared signals. The processing may include filtering, amplifying, and analog to digital conversion steps to obtain a clean signal. The obtained infrared reflection signal can be further subjected to data analysis and recording. By identifying and recording the handle actions or states corresponding to the different infrared signal sequences, a mapping relationship can be established, and the infrared signals are associated with the operation of the handle device. The server can track the actions of the user on the handle in real time and accurately capture the interaction actions of the user. For example, suppose a server is designing a virtual game based on holographic devices. The player holds a specially designed game handle, and an infrared emitter and an infrared sensor are integrated in the game handle. The character in the game performs different operations by different handle actions, such as pressing button a is a jump and pressing button B is an attack. When the player presses button a, the infrared transmitter transmits a predetermined sequence of infrared transmission signals. These signals are reflected at the handle surface and then captured by an infrared sensor within the handle and converted into electrical signals. By processing and decoding the electrical signals, the game system recognizes that this is a jumping action, and performs a jumping operation on the character in the virtual reality space. Similarly, if the player presses button B, the infrared transmitter transmits another set of infrared transmission signal sequences, and the game system parses the signals to allow the character to attack. Through the collection and analysis of the infrared reflection signals, players can use handles to conduct multi-mode interaction in the virtual game, so that game experience is richer and immersive. The method can also be applied to other fields, such as virtual reality training, simulation exercise and the like, and the interactivity of the user and the system is enhanced.

S102, acquiring a visual line of a target user through preset virtual reality glasses to obtain a target visual line, and acquiring brain wave data of the target user through the virtual reality glasses to obtain a brain wave data set;

specifically, a preset virtual reality glasses is prepared. These glasses typically have a high resolution display and built-in sensors that can track the user's head movements and gaze direction. The data of these sensors will be used to capture the visual line of the user in a virtual reality environment. The head movement and the eye gaze point of the user can be tracked in real time by the built-in sensor of the virtual reality glasses. When the user wears virtual reality glasses and enters the virtual environment, the system begins to record the rotation and movement of the user's gaze point and head in the virtual scene. The data may be represented as a visual line of sight in a three-dimensional space, revealing a visual point of interest to the user in a virtual reality environment. The brain wave data of the target user is collected, and a brain-computer interface technology is required to be integrated in the virtual reality glasses. The brain-computer interface records the brain electrical activity of the user through a set of brain electrodes or other biological sensors. When the user wears the virtual reality glasses and interacts, the brain wave data of the user are recorded by the brain sensor. The obtained visual line data and brain wave data are transmitted to a computer for processing and analysis. For the visual line-of-sight data, an algorithm and a calculation method can be used to extract key points and gaze trajectories, and further the visual line-of-sight of the target user is obtained. For brain wave data, signal processing and feature extraction are performed to acquire information about a user's cognition, emotion, or intention. After the visual line and brain wave data of the target user are obtained, these data can be combined into a comprehensive data set. This data set may be used to construct a personalized model of the user reflecting the user's attention, emotion and cognitive state in the virtual reality environment. For example, suppose a server is developing an application for virtual reality training that can help a user learn how to assemble a complex mechanical device. The user wears preset virtual reality glasses and performs virtual training. In this process, the virtual reality glasses record a visual line of sight of the user in the virtual scene, for example, the user may focus on the body of the device and then focus on the assembly position of the parts. Meanwhile, a brain-computer interface integrated in the virtual reality glasses starts to collect brain wave data of the user. When the user thinks how to assemble the parts, the electroencephalogram sensor records the electroencephalogram activity of the user. The brain wave data can reflect the cognitive state and emotional response of the user in the learning process, and the application program performs personalized adjustment according to the attention focus and the cognitive state of the user by analyzing the visual line and the brain wave data of the user. For example, if the system finds that the user has difficulty in assembling a certain part, it determines from the brain wave data that the user may feel frustrated or tired, and then provides additional prompts and encouragement to the user to assist the user in completing the task smoothly.

S103, performing time sequence alignment processing on the infrared reflection signals, the target visual line and the brain wave data set to obtain target fusion data;

specifically, data of the target user is collected, including infrared reflection signals, target visual line-of-sight and brain wave data sets. These data can be collected through preset virtual reality glasses and brain-computer interface technology. And respectively performing time stamp calibration on the collected infrared reflection signals, the target visual line and the brain wave data set. The time stamp calibration is to correspond the time information in the data to the time when the data is actually acquired, and determine the initial time stamp of each data set. This is done to ensure that the time references of the data are consistent. A time axis analysis was performed. In the time axis analysis, at least one time axis data needs to be selected for the subsequent time axis alignment processing. This selection may be based on the sampling frequency and accuracy of the data sets, selecting the data set that is most suitable as the target timeline. For example, since the sampling frequency of the brain wave data is high, the time axis of the brain wave data can be selected as the target time axis. A time axis alignment process based on linear interpolation is performed on the selected target time axis data. The time axis alignment process is to ensure that the different data sets remain consistent in time so that they can correspond to the same point in time. The time intervals between the data are filled by linear interpolation so that the data can be aligned smoothly in time. And based on the target time axis, carrying out data fusion on the infrared reflection signal, the target visual line and the brain wave data set. The data fusion can adopt simple splicing, weighted average or other fusion algorithms to combine the information of different data sets onto a target time axis to generate target fusion data. The server obtains target fusion data integrating infrared reflection signals, target visual lines and brain wave data, and can be used for subsequent analysis and application. For example, suppose that the server is developing a holographic-device-based medical assistance system intended to assist a doctor in performing a surgical procedure. The server interacts with the doctor through virtual reality glasses and the biosensor. The doctor performs the surgical operation using the virtual reality glasses and the handle device. The virtual reality glasses track the visual line of the doctor, recording the organ or area of interest to the doctor during the procedure. Meanwhile, the brain wave data of the doctor are collected by the brain wave sensor, and the cognitive state and emotion change of the doctor are recorded. And determining initial time stamps of the infrared reflection signals, the target visual line and the brain wave data set through time stamp calibration. A time axis analysis is performed, and a time axis of brain wave data is selected as a target time axis because it has the highest sampling frequency. The time stamps of the infrared reflected signal and the target visual line are linearly interpolated to align them with the target time axis. And carrying out data fusion on the infrared reflection signals subjected to the time axis alignment processing, the target visual line and the brain wave data set to generate target fusion data. These fused data may be provided to a physician for assisting a surgical procedure, such as displaying the position and state of key organs in a virtual reality scene, while analyzing the physician's cognitive state and emotional response from the brain wave data, providing real-time feedback and guidance. Through time sequence alignment processing, the infrared reflection signals, the target visual line and the brain wave data set are fused to a unified target time axis, so that comprehensive analysis and interaction of multi-mode data are realized, and a more comprehensive and personalized function is provided for the medical auxiliary system.

S104, extracting multi-type feature data from the target fusion data to obtain position feature data, concerned position feature data and emotion feature data;

specifically, data filtering processing is performed on the target fusion data, and filtering fusion data is generated. The data filtering is to remove noise and interference in the data, making the data smoother and more reliable. The common filtering methods include low-pass filtering, high-pass filtering, band-pass filtering and the like, and the proper filtering method is selected according to the characteristics of the data. And carrying out data layering on the filtering fusion data to determine multi-layer filtering fusion data to be processed. The data layering is to divide the filtering fusion data into a plurality of layers, and the layering is performed according to the importance and time sequence relation of the data. Thus, different feature extraction and analysis can be performed for different levels of data. And respectively extracting the multi-type characteristic data from each layer of filtering fusion data to be processed. Feature data extraction is the extraction of features with representativeness and separability from raw data for subsequent analysis and application. Different levels of data may extract different types of features, such as location features, location of interest features, emotional features, and so forth. For example, assume that the target fusion data of the server is medical assistance data collected through virtual reality glasses and brain-computer interface technology. In the medical operation process, a doctor wears virtual reality glasses and carries brain sensors at the same time, the virtual reality glasses record visual lines and operation scenes of the doctor, and the brain sensors collect brain wave data of the doctor. And carrying out data filtering processing on the target fusion data, and removing possible noise and interference to obtain filtering fusion data. For example, band-pass filtering is performed on brain wave data to remove high-frequency and low-frequency noise and retain signals of a specific frequency band. And carrying out data layering on the filtering fusion data to determine multi-layer filtering fusion data to be processed. For example, the filter fusion data is divided into a low frequency layer and a high frequency layer, and the division is performed according to the frequency range of the brain wave data. And extracting multi-type characteristic data from each layer of filtering fusion data to be processed. And extracting position characteristic data from the data of the low-frequency layer, and analyzing the operation position and the movement track of the handle in the operation process of a doctor. And extracting the attention position characteristic data of the high-frequency layer, and identifying attention points and visual interest areas of doctors in the operation scene. And meanwhile, extracting emotion characteristic data from the brain wave data, and analyzing the emotion state and the cognition level of a doctor in the operation process by analyzing the spectrum characteristic and waveform form of the brain wave data. Through such multi-type feature data extraction, the server obtains important features such as position information, a region of interest, and an emotional state of a doctor during an operation from the medical assistance data. These features can help doctors improve surgical skills, optimize surgical procedures, and provide real-time feedback and guidance for surgical procedures. Meanwhile, the feature data extraction method can also be applied to other fields, such as user experience research, virtual reality games and the like, and provides more personalized and intelligent experience for multi-modal interaction.

And carrying out hierarchical standard matching on each layer of filtering fusion data to be processed, and determining the data type corresponding to each layer of filtering fusion data to be processed. Hierarchical standard matching is to allocate different types of data to corresponding hierarchies, and matching is performed according to the characteristics and purposes of the data. For example, the low frequency part in the filter fusion data is matched as a position feature data layer, the high frequency part is matched as a focus position feature data layer, and the brain wave data is matched as an emotion feature data layer. And carrying out feature extraction algorithm matching based on the data types corresponding to the to-be-processed filtering fusion data of each layer, and determining the feature extraction algorithm corresponding to the to-be-processed filtering fusion data of each layer. Different types of data may require different feature extraction algorithms to extract the most representative and distinguishing features. For example, for a layer of positional feature data, a motion trajectory analysis algorithm may be used to extract the position and motion information of the physician's handle manipulation; for the focused position feature data layer, a focused point recognition algorithm can be used for extracting a focused region of a doctor in a scene; for the emotion feature data layer, algorithms such as spectrum analysis and waveform morphology analysis can be used to extract emotion features in brain wave data. And extracting the multi-type feature data of each layer of the to-be-processed filter fusion data through a feature extraction algorithm corresponding to each layer of the to-be-processed filter fusion data, so as to obtain the position feature data, the concerned position feature data and the emotion feature data. The result of the feature data extraction will be a specific feature value or feature vector for each data type. For example, taking a virtual reality medical assistance system as an example, assume that the server has three layers of filtering fusion data to be processed, namely, handle operation position data of a low-frequency layer, scene attention point data of a high-frequency layer and brain wave data layer. And matching the handle operation position data to a position feature data layer, matching the scene attention point data to an attention position feature data layer and matching the brain wave data to an emotion feature data layer through layering standard matching. Performing feature extraction algorithm matching based on the data types, and extracting the position and motion information of the operation of the doctor handle by using a motion trail analysis algorithm aiming at the position feature data layer; extracting a region of interest of a doctor in a scene by using a gaze point identification algorithm aiming at the attention position characteristic data layer; and extracting emotion features in the brain wave data by using algorithms such as frequency spectrum analysis, waveform morphology analysis and the like aiming at the emotion feature data layer. And respectively extracting multi-type feature data of the three layers of filtering fusion data to be processed through a feature extraction algorithm to obtain position feature data, concerned position feature data and emotion feature data. These feature data will be used for subsequent data analysis and application, for example in a virtual reality medical assistance system, to optimize the presentation of the virtual scene according to the location features and the location features of interest of the physician, while providing real-time feedback and adjustment according to the emotional features to enhance the physician's surgical experience and efficiency.

S105, carrying out space division on a preset three-dimensional display space through the position characteristic data and the attention position characteristic data to obtain a corresponding target space region;

specifically, three-dimensional virtual point cloud construction is carried out on the position characteristic data, and a first candidate three-dimensional virtual point cloud is obtained. The position characteristic data may include position and motion information of the user handle operation, and by constructing point clouds of the data in a three-dimensional space, corresponding virtual point cloud representations are obtained. And constructing the three-dimensional virtual point cloud of the concerned position characteristic data to obtain a second candidate three-dimensional virtual point cloud. The location of interest feature data may contain information about the region or keypoint of interest of the user in the scene, which data is also represented by building a virtual point cloud in three-dimensional space. And performing virtual three-dimensional space mapping on the first candidate three-dimensional virtual point cloud to obtain a first virtual three-dimensional space. In this embodiment, the first candidate three-dimensional virtual point cloud is mapped into the three-dimensional display space, and a virtual three-dimensional space model is created. And performing virtual three-dimensional space mapping on the second candidate three-dimensional virtual point cloud to obtain a second virtual three-dimensional space. And mapping the second candidate three-dimensional virtual point cloud into the three-dimensional display space to obtain another virtual three-dimensional space model. And performing space division point matching on the first virtual three-dimensional space and the second virtual three-dimensional space, and determining a space division point set. In this step, a matching set of points is found by comparing the point cloud data in the two virtual space models. These matching points represent regions of similar locations and points of interest in the two virtual spaces. And carrying out space division on the three-dimensional display space based on the space division point set to obtain the target space region. Based on the matched set of points, the three-dimensional display space may be divided into a plurality of target spatial regions, each region containing a set of points having similar locations and features of interest. For example, assuming that the server is developing a virtual reality game, the user controls the game through the handle device, and the virtual reality glasses track the user's head movements and gaze point. And recording the position and motion information of the user in the game through the handle equipment, and constructing the three-dimensional virtual point cloud by using the data to obtain a first candidate three-dimensional virtual point cloud. The virtual reality glasses record information of the concerned region or key points of the user in the game, and construct a three-dimensional virtual point cloud to obtain a second candidate three-dimensional virtual point cloud. And mapping the first candidate three-dimensional virtual point cloud into a three-dimensional display space of the game to obtain a first virtual three-dimensional space model. And mapping the second candidate three-dimensional virtual point cloud into the three-dimensional display space of the game to obtain a second virtual three-dimensional space model. And (3) by comparing the point cloud data in the first virtual three-dimensional space and the second virtual three-dimensional space, finding a matched point set, and determining a space division point set. The three-dimensional display space of the game is divided into a plurality of target spatial regions according to the spatially-divided point sets, each region containing a point set having similar positions and features of interest. These target spatial regions may be used for interactive design in a game, such as adjusting object effects in a game scene according to a user's gaze point, providing a more personalized and immersive game experience.

S106, collecting voice control instructions of a target user, and generating interaction instructions of the voice control instructions through emotion feature data to generate target interaction instructions;

specifically, the system collects voice control instructions of the target user in real time through equipment such as a microphone. The collected voice data is an analog signal, which needs to be converted into a digital signal through analog-to-digital conversion so as to perform subsequent digital signal processing. And performing voice recognition on the digitized voice data. Speech recognition technology is the process of converting speech signals into text or instructions. Modern speech recognition systems are typically based on deep learning techniques such as Recurrent Neural Networks (RNNs) and transcriptional attention mechanisms (Transcription Attention Mechanism). These techniques may perform feature extraction and pattern recognition on voice data to convert a user's voice instructions into a text format. Meanwhile, the system also needs to collect emotion characteristic data of the target user. This is achieved in a number of ways, for example using emotion recognition sensors, facial expression recognition techniques, brain wave data acquisition, etc. The goal of emotion analysis is to determine the current emotional state of the user, such as happy, sad, angry, etc. For speech control scenarios, emotion analysis is achieved by analyzing pitch, pace, intonation, and some linguistic features in speech. Modern emotion analysis techniques typically combine Natural Language Processing (NLP) techniques with machine learning algorithms, such as Support Vector Machines (SVMs) and Deep Neural Networks (DNNs), to emotion classify text or speech data. And combining the text instruction obtained by voice recognition and emotion characteristic data obtained by emotion analysis to form a comprehensive semantic representation. For example, the user speaks "turn on the light" and emotion analysis shows that the user is emotionally active, the system will recognize that the user wants to turn on the light and may feel happy or satisfied. Based on the integrated semantic representation, the system generates corresponding interaction instructions. For example, the system may generate "turn on lights and play happy music" to satisfy the user's positive emotion. If the emotion analysis shows that the user's emotion is low, the system may generate "turn on the lights and dim the lights" to improve the user's emotion. And the system executes corresponding interaction action according to the generated target interaction instruction. This may involve device control, scene switching, music playing, etc. to meet the needs and emotional state of the user. For example, the system may execute an instruction to turn on the light while selecting to play the appropriate music according to the result of the emotion analysis, thereby providing a more intelligent and personalized interactive experience for the user. The goal of the overall process is to generate corresponding interaction instructions based on the user's voice instructions and emotional state to provide a more intelligent and emotional user experience.

S107, constructing holographic display space coordinates through a target interaction instruction, generating a holographic display space coordinate set, carrying out holographic equipment matching through the target interaction instruction, and determining a target holographic equipment set;

specifically, interaction location information is extracted from the target interaction instruction. Based on the semantic analysis of the interaction instructions, the interaction location data involved is identified, which may be a certain area in the virtual reality scene, the location of the virtual object or the actual location of the user. And mapping the interaction position data into a preset holographic display space coordinate system. The holographic display space coordinate system is a virtual three-dimensional coordinate system for locating the position of a virtual object, user or other interactive element. And corresponding the interaction position data and the holographic display space coordinate system to obtain corresponding three-dimensional coordinate information, thereby forming a holographic display space coordinate set. And analyzing the equipment identification information in the target interaction instruction. The device identification may be a unique identifier, such as a device serial number, device name, or other identification code, for identifying a particular holographic device. And obtaining a target equipment identification set by analyzing the equipment identification information in the interaction instruction. And matching the target device identification set with the preconfigured holographic device information. The preconfigured holographic device information contains relevant information such as the position, the function, the attribute and the like of each holographic device. By comparing the target device identification with these configuration information, the set of holographic devices involved in the target interaction instruction can be determined. For example, assuming a virtual reality intelligent assistant system that interacts with voice commands, the user may issue different commands to the system, such as "open application", "close window", and "switch scenes". The user sends out a voice command of opening an application program, and the system recognizes the command and analyzes the emotion of the user to be in a normal state. The system extracts interaction location data from the instruction as a region in the virtual reality scene. The system maps the interaction position data to a holographic display space coordinate system to obtain corresponding three-dimensional coordinates. The device identification in the system analysis voice command is "application". The system determines the holographic device set to which the instruction relates by matching the device identification "application" with the preconfigured holographic device information, thereby performing the operation of opening the application.

S108, constructing a three-dimensional virtual reality space through a target holographic device set based on the holographic display space coordinate set, and transmitting the three-dimensional virtual reality space to virtual reality glasses of a target user for holographic image display.

The device information is extracted from the target holographic device set, and the device information comprises geometric element data such as the position, the shape, the size and the like of the device. Such information may describe the location and characteristics of each device in the holographic display space. These geometric element data are associated with a holographic display space coordinate set according to the device information. The holographic display space coordinate set is a virtual three-dimensional coordinate system for locating the positions of virtual objects, users and interactive elements. And (3) obtaining the representation of the three-dimensional virtual reality space by corresponding the geometric element data of the equipment to the holographic display space coordinate set. For each device, virtual space generation is performed according to the position of its geometric element data in the holographic display space. This means that the device is placed in a suitable position and its size and shape in the virtual reality scene is adjusted so that it can interact with other virtual objects in the virtual world. And transmitting the generated three-dimensional virtual reality space to virtual reality glasses of the target user so as to display the holographic image. Data of the virtual reality scene is transmitted to the glasses device through a wireless transmission technology, and a user can see the holographic image in the glasses after wearing the glasses to interact with the virtual reality scene. For example, consider a virtual reality teaching scenario where students learn scientific knowledge through holographic glasses into the virtual world. In a virtual classroom, there are several holographic devices, including a virtual projector, a virtual laboratory bench, and a virtual interactive screen. The system extracts geometric element data of these devices, including their positions, shapes and sizes, from the set of target holographic devices. The system correlates the geometric element data with the holographic display space coordinate set to obtain a representation of the three-dimensional virtual reality space. The system performs virtual space generation based on the location of each device in the holographic display space. The virtual projector is placed on one wall of the classroom, the virtual laboratory bench is placed in the center of the classroom, and the virtual interactive screen is placed on the other wall of the classroom. According to the geometric element data of the devices, the size and the shape of the devices in the virtual reality scene are adjusted so that the devices can interact with students, for example, the students can perform experimental operations on a virtual experiment table. The system transmits the generated three-dimensional virtual reality space to the virtual reality glasses of the target user, so that the students can see the holographic images in the glasses after wearing the glasses. The students interact with the virtual projector through the interaction function in the glasses to adjust projection contents; performing experimental operation on a virtual experiment table; and the virtual interaction screen is used for carrying out interaction communication with a virtual teacher.

In the embodiment of the application, the infrared reflection signals are acquired to obtain the infrared reflection signals; acquiring a visual line of a target user to obtain a target visual line, and acquiring brain wave data of the target user to obtain a brain wave data set; performing time sequence alignment processing to obtain target fusion data; extracting multi-type feature data from the target fusion data to obtain position feature data, concerned position feature data and emotion feature data; space division is carried out on the three-dimensional display space, and a target space area is obtained; generating interactive instructions for the voice control instructions through the emotion characteristic data to generate target interactive instructions; constructing holographic display space coordinates, generating a holographic display space coordinate set, carrying out holographic equipment matching, and determining a target holographic equipment set; and constructing a three-dimensional virtual reality space, and transmitting the three-dimensional virtual reality space to virtual reality glasses of a target user for holographic image display. In the embodiment of the application, the more immersive interaction experience is realized through the comprehensive application of the multi-mode data such as the infrared reflection signal acquisition, the visual line acquisition, the brain wave data acquisition and the like. The user interacts with the virtual environment in real time through the modes of the handle equipment, the visual line, the brain waves and the like, and more natural and real interaction experience is provided. And extracting multi-type feature data from the target fusion data to obtain position feature data, concerned position feature data and emotion feature data. These feature data can be used for deeper user behavior analysis, emotion recognition and personalized interactions, providing more personalized, intelligent services. And obtaining a target space region by space division of the three-dimensional display space. The virtual scene is divided into different areas, so that the interactive behavior of the user can be conveniently positioned and managed, and more accurate and targeted interactive experience is provided. And generating an interaction instruction for the voice control instruction through the emotion characteristic data, generating a target interaction instruction, carrying out holographic equipment matching, and determining a target holographic equipment set. This helps to intelligently select the appropriate devices and interaction modes according to the emotional state and voice instructions of the user, providing a personalized, intelligent interaction experience. The three-dimensional virtual reality space is constructed and transmitted to the virtual reality glasses of the target user to display the holographic image, so that the holographic image display of realism and immersion is realized. The user views and interacts with the holographic image through the virtual reality glasses, providing a more realistic and vivid visual experience.

In a specific embodiment, the process of executing step S103 may specifically include the following steps:

(1) Respectively performing time stamp calibration on the infrared reflection signal, the target visual line and the brain wave data set, and determining a first initial time stamp corresponding to the infrared reflection signal, a second initial time stamp corresponding to the target visual line and a third initial time stamp corresponding to the brain wave data set;

(2) Performing time axis analysis on the first initial time stamp, the second initial time stamp and the third initial time stamp to determine at least one time axis data;

(3) Performing time axis alignment processing based on linear interpolation on at least one time axis data to determine a target time axis;

(4) And based on the target time axis, carrying out data fusion on the infrared reflection signal, the target visual line and the brain wave data set to generate target fusion data.

Specifically, the time stamp calibration is carried out on the infrared reflection signal, the target visual line and the brain wave data set, and the time stamp of each data point is determined. The time stamp calibration is achieved by recording time information of the data, such as a time stamp or system time obtained from the data acquisition device. Each data point is marked with its corresponding timestamp so that the data points in the different data sets can be correlated to the correct point in time. And performing time axis analysis to obtain time axis conditions, such as start time and end time, of each data set. This allows to understand the distribution of the data points of each data set on the time axis and the time length of the respective data set. According to the result of the time axis analysis, the time axis of one data set may be selected as a target time axis, or the time axes of a plurality of data sets may be combined to obtain a unified time axis. The unified target timeline will ensure that the data points of different data sets are aligned and compared under the same time scale. And performing time axis alignment processing of linear interpolation on the data on the target time axis. This is to address the different sampling rate or timestamp errors that may exist during data acquisition. The data are interpolated and aligned by algorithms such as linear interpolation, so that the infrared reflection signal, the target visual line and the brain wave data can be arranged according to the same time step on the time axis. And based on the target time axis, carrying out data fusion on the infrared reflection signal, the target visual line and the brain wave data set to generate target fusion data. The data fusion is to combine the data points in each data set according to the corresponding positions on the target time axis. The server obtains complete target fusion data, wherein the complete target fusion data comprises fusion information of infrared reflection signals, target visual line and brain wave data. For example, suppose there is a virtual reality game, where a player plays with a virtual reality head display. In a game, a player needs to control the perspective of a game character using head movements while interacting by gazing at different objects, while recording the brain wave data of the player to analyze the emotional state of the player. And performing time stamp calibration on the infrared reflection signal, the target visual line and the brain wave data set. For example, each head movement, gaze motion and brain wave data point is marked with its corresponding timestamp, ensuring that the data points are in one-to-one correspondence with the time it was recorded. And (5) performing time axis analysis to obtain the time axis condition of each data set. Assuming that the time axis range of the head movement data is from the start to the end of the game, the time axis range of the gaze motion data is from the start to the end of the game, and the time axis range of the brain wave data is also from the start to the end of the game. According to the result of the time axis analysis, a time axis of one data set is selected as a target time axis. In this embodiment, the server selects the timeline of head movement data as the target timeline because head movement is the primary input by the player to control the perspective of the game character. And performing time axis alignment processing of linear interpolation on the head motion data. The time axis of the head movement data is aligned with the time axis of the staring action data and the brain wave data through a linear interpolation algorithm, so that comparison and analysis are ensured under the same time scale. And based on the target time axis, performing data fusion on the head movement data, the staring action data and the brain wave data to generate target fusion data. For example, the server integrates head movement data and gaze motion data, analyzes the relationship of the player's visual attention to head movement in the game, and simultaneously analyzes the player's emotional state in different game scenes in combination with brain wave data.

In a specific embodiment, as shown in fig. 2, the process of executing step S104 may specifically include the following steps:

s201, performing data filtering processing on target fusion data to generate filtering fusion data;

s202, carrying out data layering on the filtering fusion data to determine multi-layer filtering fusion data to be processed;

s203, extracting multi-type feature data of each layer of filtering fusion data to be processed to obtain position feature data, concerned position feature data and emotion feature data.

The data filtering process is performed. The data filtering is to remove noise and outliers in the data to improve the quality and accuracy of the data. Common filtering methods include low pass filtering, median filtering, gaussian filtering, etc. These methods can smooth data, eliminate unnecessary fluctuations, while preserving important characteristic information. And the server obtains filtering fusion data through data filtering processing. And carrying out data layering on the filtering fusion data. Data layering is to divide data into different layers in order to further extract feature information of the different layers. Layering occurs according to specific criteria, such as according to time periods, spatial regions, or other relevant attributes. And through data layering, the server determines multi-layer filtering fusion data to be processed. After data layering, the server respectively extracts multi-type characteristic data from each layer of filtering fusion data to be processed. Feature data extraction is to extract meaningful features from the data for subsequent analysis and application. For each layer of data, the server employs a different feature extraction algorithm to obtain location feature data, location feature data of interest, and emotional feature data. For example, in a virtual reality application, a server removes noise due to device errors or motion judder by performing a filtering process on behavior data of a user in a virtual scene. The server layers the behavior data of the user and divides the data into different phases, such as an exploration phase, an interaction phase and a decision phase, according to the time period. The server extracts features for each stage of data separately. For the position feature data, the server extracts a motion trail, a gaze point set and an interaction position set of the user in the virtual scene. These features may help the server to learn the range of motion of the user in the virtual scene, the region of interest, and the interaction with the virtual object. And the attention position characteristic data calculates the attention degree of the user to different areas in the virtual scene by analyzing the attention point set of the user. The server knows which regions are more attractive to the user and optimizes the presentation of the content of the virtual scene according to the user's points of interest. The extraction of the emotion feature data identifies the emotional states of the user at different stages by analyzing the brain wave data. For example, the server extracts the emotional state of the user, such as happiness, tension, relaxation, etc., from the brain wave data through an emotion recognition algorithm. This may help the server learn about the user's emotional changes in the virtual scene, thereby providing a more personalized and emotional virtual experience for the user.

In a specific embodiment, as shown in fig. 3, the process of performing step S203 may specifically include the following steps:

s301, carrying out hierarchical standard matching on each layer of to-be-processed filtering fusion data, and determining a data type corresponding to each layer of to-be-processed filtering fusion data;

s302, performing feature extraction algorithm matching based on the data types corresponding to each layer of to-be-processed filter fusion data, and determining a feature extraction algorithm corresponding to each layer of to-be-processed filter fusion data;

s303, extracting multi-type feature data of each layer of the to-be-processed filter fusion data through a feature extraction algorithm corresponding to each layer of the to-be-processed filter fusion data, and obtaining position feature data, concerned position feature data and emotion feature data.

It should be noted that hierarchical standard matching is performed. And defining corresponding layering standards for each layer of filtering fusion data to be processed according to the characteristics and application requirements of the data. The layering criteria are determined based on temporal, spatial, or other properties. For example, in a virtual reality scenario, a server layers the user's interaction behavior by time periods, such as an exploration phase, a task execution phase, and a rest phase. According to these criteria, the server classifies data for different periods of time into different tiers. And performing feature extraction algorithm matching. And for each layer of filtering fusion data to be processed, the server selects a proper feature extraction algorithm according to the data type to which the server belongs. Different types of data may require different feature extraction methods to extract the most valuable information from them. For example, in the position feature data, the server adopts a track analysis algorithm to extract a motion track and an interaction path of the user in the virtual scene. And for the attention position characteristic data, the server uses an attention point clustering algorithm to identify attention points of the user in the virtual scene. For the emotion feature data, the server adopts an electroencephalogram emotion recognition algorithm to extract the emotion state of the user from the electroencephalogram data. And extracting the multi-type characteristic data of each layer of filtering fusion data to be processed by a characteristic extraction algorithm. This means that the server extracts a variety of meaningful feature information from each layer of data. For example, assume that a server is developing a holographic communication system that allows users to communicate remotely through holographic devices and in real-time in a three-dimensional virtual space. The server matches the appropriate feature extraction algorithm based on the data type of each layer of data. For example, during a stationary phase, the server extracts the user's biometric characteristics as location feature data using a frequency domain analysis algorithm; in the interaction stage, the server adopts an emotion analysis algorithm to extract the emotion state of the user as emotion characteristic data. Through the feature extraction algorithms, the server extracts position feature data, attention position feature data and emotion feature data from each layer of filtering fusion data to be processed respectively. These feature data will help the server better understand the behavior and emotional state of the user in the holographic communication system, thereby providing a more immersive and realistic interaction experience.

In a specific embodiment, as shown in fig. 4, the process of performing step S105 may specifically include the following steps:

s401, performing three-dimensional virtual point cloud construction on the position characteristic data to obtain a first candidate three-dimensional virtual point cloud;

s402, performing three-dimensional virtual point cloud construction on the focused position feature data to obtain a second candidate three-dimensional virtual point cloud;

s403, performing virtual three-dimensional space mapping on the first candidate three-dimensional virtual point cloud to obtain a first virtual three-dimensional space;

s404, performing virtual three-dimensional space mapping on the second candidate three-dimensional virtual point cloud to obtain a second virtual three-dimensional space;

s405, performing space division point matching on the first virtual three-dimensional space and the second virtual three-dimensional space, and determining a space division point set;

s406, space division is carried out on the three-dimensional display space based on the space division point set, and a target space region is obtained.

Specifically, three-dimensional virtual point cloud construction is carried out on the position characteristic data, and a first candidate three-dimensional virtual point cloud is obtained. Each position in the position characteristic data is converted into a three-dimensional coordinate point, and all the points are combined into a virtual point cloud. And constructing the three-dimensional virtual point cloud of the concerned position characteristic data to obtain a second candidate three-dimensional virtual point cloud. Likewise, the server converts each location in the location-of-interest feature data into three-dimensional coordinate points and combines them into another virtual point cloud. And performing virtual three-dimensional space mapping on the first candidate three-dimensional virtual point cloud to obtain a first virtual three-dimensional space. This maps points in the first candidate three-dimensional virtual point cloud into a virtual three-dimensional space, resulting in a virtual space with three-dimensional coordinates. And performing virtual three-dimensional space mapping on the second candidate three-dimensional virtual point cloud to obtain a second virtual three-dimensional space. Similarly, the server maps points in the second candidate three-dimensional virtual point cloud into another virtual three-dimensional space to obtain a second virtual space. After that, the first virtual three-dimensional space and the second virtual three-dimensional space are subjected to space division point matching, and a space division point set is determined. Some points in the first virtual space are matched with corresponding points in the second virtual space to determine the likelihood that they represent the same location or region in real space. And carrying out space division on the three-dimensional display space based on the space division point set to obtain a target space region. The server divides the three-dimensional display space according to the matched point set, so that a target space region which can be seen by a user in the virtual reality glasses is obtained, and immersive holographic image display is provided. For example, assume that a server is developing a virtual reality navigation system. The user views the navigation route through the holographic device and navigates in real time in the virtual reality glasses. In this system, the server constructs two three-dimensional virtual point clouds according to the user's position feature data and the focused position feature data, which represent the user's position and focused position, respectively. Mapping the two virtual point clouds into a virtual three-dimensional space to obtain two virtual spaces. And determining the corresponding relation between the navigation route focused by the user and the surrounding environment by matching points in the two virtual spaces. And carrying out space segmentation on the three-dimensional display space according to the matching point set, so that a user is ensured to only see the navigation route and the related map information in the virtual reality glasses, and better navigation experience is provided.

In a specific embodiment, the process of executing step S107 may specifically include the following steps:

(1) Performing interaction position mapping on the target interaction instruction, and determining interaction position data;

(2) Performing coordinate mapping on the interactive position data based on a preset space coordinate system to generate a holographic display space coordinate set;

(3) Performing equipment identification analysis on the target interaction instruction to determine a target equipment identification set;

(4) And carrying out holographic device matching through the target device identification set, and determining the target holographic device set.

Specifically, the target user expresses his own intent through interaction means (e.g., gestures, voice commands, etc.). For example, the user may point to a particular location through a gesture, or describe a desired operation using voice instructions. The system will process the target interaction instructions and extract interaction location data therefrom. This process involves the capture and interpretation of sensor data to obtain user-specified interaction location coordinates. The system uses a preset spatial coordinate system to coordinate map the interaction location data. The preset spatial coordinate system is a coordinate system of the holographic display space for describing the layout and position of the holographic projection. By mapping the interaction location data into this spatial coordinate system, the system can obtain a coordinate set of the holographic display space, which contains specific location information of the user interaction. Meanwhile, the system analyzes the device identification information in the target interaction instruction to determine which devices the user wishes to interact on. The device identification information may tell the system the user's device preferences, for example, the user may choose to view or manipulate content on a particular holographic device or virtual reality glasses. The system performs holographic device matching according to the device identification set, and determines a holographic device set suitable for user interaction intention. By matching the target device, the system can ensure that the content is displayed or transmitted on the correct device, providing a customized holographic interaction experience for the user. For example, assume that a server is developing a holographic presentation system. The user interacts with the holographic content through gestures and voice commands. In this system, the user can point to a specific location in the holographic projection space with a finger and then trigger an action or present specific content by voice instructions. In this embodiment, the user points to the interaction location using gestures, and the system captures this pointed location information via the sensor and maps it to interaction location data. The system uses a preset space coordinate system to map the interactive position data into holographic display space coordinates, and a specific position pointed by a user is obtained. The user expresses the interactive intention through the voice command, the system analyzes the command and determines the equipment identification set of the user. If the user selects to view the presentation on a particular holographic device by instruction, the system will determine that the holographic device is one of the target devices. If the user chooses to interact using virtual reality glasses, the system will add the virtual reality glasses to the set of target holographic devices. In this embodiment, the system may implement processing of user interaction instructions and matching of holographic devices, thereby providing a customized holographic presentation experience.

In a specific embodiment, the process of executing step S108 may specifically include the following steps:

(1) Extracting geometric elements from the holographic display space coordinate set through the target holographic equipment set to determine the geometric element set;

(2) Generating a virtual space of the geometric element set to obtain a three-dimensional virtual reality space;

(3) And transmitting the three-dimensional virtual reality space to virtual reality glasses of the target user for holographic image display.

Specifically, the target holographic device set includes all holographic devices in the system that meet the requirements of the user, and these devices may be holographic projectors, virtual reality glasses, or other devices that support holographic image display. And the system extracts geometric elements from the holographic display space coordinate set according to the target holographic equipment set. The holographic display space coordinate set contains all the space position information used to construct the hologram. Geometric element extraction is the conversion of positional information in a set of spatial coordinates into elements with geometric shapes, such as points, lines, faces, etc. These geometric elements will serve as the basis for constructing a three-dimensional virtual reality space. The system then uses the extracted set of geometric elements for virtual space generation. In the virtual space generation process, the system constructs a three-dimensional virtual reality space according to the positions and the shapes of the geometric elements. This virtual reality space is made up of geometric elements that can simulate objects, scenes, and environments in the real world. And the system transmits the generated three-dimensional virtual reality space to virtual reality glasses of the target user for holographic image display. Virtual reality glasses are devices where a user interacts with a virtual reality environment, which may project a three-dimensional virtual reality space into the user's visual perception. Through the display of virtual reality glasses, the user can experience the hologram to interact and immersive experience with virtual reality space. For example, assume that a server is developing a medical training system intended to provide a training experience for virtual surgery. The set of target holographic devices includes a holographic projector and a pair of virtual reality glasses. The system extracts geometric elements from the holographic display space coordinate set according to the positions and the attributes of the holographic projector and the virtual reality glasses, and obtains a geometric element set representing surgical instruments, human anatomy structures and the like. The system constructs a three-dimensional virtual reality space according to the geometric element sets, and simulates a surgical environment and an operation scene. The system transmits the generated virtual operation space to the virtual reality glasses, and after the user wears the glasses, the user can observe and operate the virtual operation in the holographic image, so that the practicality and experience effect of medical training are improved. In this embodiment, it can be seen how to extract geometric elements from the holographic display space through the target holographic device set, and transmit the generated three-dimensional virtual reality space to the virtual reality glasses of the user for displaying the holographic image.

Through the steps, through the comprehensive application of the multi-mode data such as infrared reflection signal acquisition, visual line acquisition and brain wave data acquisition, more immersive interaction experience is realized. The user interacts with the virtual environment in real time through the modes of the handle equipment, the visual line, the brain waves and the like, and more natural and real interaction experience is provided. And extracting multi-type feature data from the target fusion data to obtain position feature data, concerned position feature data and emotion feature data. These feature data can be used for deeper user behavior analysis, emotion recognition and personalized interactions, providing more personalized, intelligent services. And obtaining a target space region by space division of the three-dimensional display space. The virtual scene is divided into different areas, so that the interactive behavior of the user can be conveniently positioned and managed, and more accurate and targeted interactive experience is provided. And generating an interaction instruction for the voice control instruction through the emotion characteristic data, generating a target interaction instruction, carrying out holographic equipment matching, and determining a target holographic equipment set. This helps to intelligently select the appropriate devices and interaction modes according to the emotional state and voice instructions of the user, providing a personalized, intelligent interaction experience. The three-dimensional virtual reality space is constructed and transmitted to the virtual reality glasses of the target user to display the holographic image, so that the holographic image display of realism and immersion is realized. The user views and interacts with the holographic image through the virtual reality glasses, providing a more realistic and vivid visual experience.

The multi-modal interaction method based on the holographic device in the embodiment of the present invention is described above, and the multi-modal interaction apparatus based on the holographic device in the embodiment of the present invention is described below, referring to fig. 5, one embodiment of the multi-modal interaction apparatus based on the holographic device in the embodiment of the present invention includes:

the first acquisition module 501 is configured to acquire an infrared reflection signal of a handheld handle device of a target user, so as to obtain the infrared reflection signal;

the second acquisition module 502 is configured to acquire a visual line of the target user through a preset virtual reality glasses, obtain a target visual line, and acquire brain wave data of the target user through the virtual reality glasses, so as to obtain a brain wave data set;

an alignment module 503, configured to perform time sequence alignment processing on the infrared reflection signal, the target visual line, and the brain wave data set, to obtain target fusion data;

the extracting module 504 is configured to extract multi-type feature data from the target fusion data to obtain location feature data, focus location feature data, and emotion feature data;

the segmentation module 505 is configured to spatially segment a preset three-dimensional display space according to the position feature data and the attention position feature data, so as to obtain a corresponding target space region;

The generating module 506 is configured to collect a voice control instruction of the target user, and generate an interaction instruction for the voice control instruction through the emotion feature data, so as to generate a target interaction instruction;

the matching module 507 is configured to construct holographic display space coordinates according to the target interaction instruction, generate a holographic display space coordinate set, and perform holographic device matching according to the target interaction instruction, so as to determine a target holographic device set;

and the display module 508 is configured to construct a three-dimensional virtual reality space through the target holographic device set based on the holographic display space coordinate set, and transmit the three-dimensional virtual reality space to the virtual reality glasses of the target user for holographic image display.

Through the cooperation of the components, the infrared reflection signal is acquired to obtain an infrared reflection signal; acquiring a visual line of a target user to obtain a target visual line, and acquiring brain wave data of the target user to obtain a brain wave data set; performing time sequence alignment processing to obtain target fusion data; extracting multi-type feature data from the target fusion data to obtain position feature data, concerned position feature data and emotion feature data; space division is carried out on the three-dimensional display space, and a target space area is obtained; generating interactive instructions for the voice control instructions through the emotion characteristic data to generate target interactive instructions; constructing holographic display space coordinates, generating a holographic display space coordinate set, carrying out holographic equipment matching, and determining a target holographic equipment set; and constructing a three-dimensional virtual reality space, and transmitting the three-dimensional virtual reality space to virtual reality glasses of a target user for holographic image display. In the embodiment of the application, the more immersive interaction experience is realized through the comprehensive application of the multi-mode data such as the infrared reflection signal acquisition, the visual line acquisition, the brain wave data acquisition and the like. The user interacts with the virtual environment in real time through the modes of the handle equipment, the visual line, the brain waves and the like, and more natural and real interaction experience is provided. And extracting multi-type feature data from the target fusion data to obtain position feature data, concerned position feature data and emotion feature data. These feature data can be used for deeper user behavior analysis, emotion recognition and personalized interactions, providing more personalized, intelligent services. And obtaining a target space region by space division of the three-dimensional display space. The virtual scene is divided into different areas, so that the interactive behavior of the user can be conveniently positioned and managed, and more accurate and targeted interactive experience is provided. And generating an interaction instruction for the voice control instruction through the emotion characteristic data, generating a target interaction instruction, carrying out holographic equipment matching, and determining a target holographic equipment set. This helps to intelligently select the appropriate devices and interaction modes according to the emotional state and voice instructions of the user, providing a personalized, intelligent interaction experience. The three-dimensional virtual reality space is constructed and transmitted to the virtual reality glasses of the target user to display the holographic image, so that the holographic image display of realism and immersion is realized. The user views and interacts with the holographic image through the virtual reality glasses, providing a more realistic and vivid visual experience.

The multi-mode interaction device based on the holographic device in the embodiment of the present invention is described in detail from the point of view of the modularized functional entity in fig. 5 above, and the multi-mode interaction device based on the holographic device in the embodiment of the present invention is described in detail from the point of view of hardware processing below.

Fig. 6 is a schematic structural diagram of a multi-modal interaction device based on a holographic device 600 according to an embodiment of the present invention, where the multi-modal interaction device 600 may have a relatively large difference due to configuration or performance, and may include one or more processors (central processing units, CPU) 610 (e.g., one or more processors) and a memory 620, and one or more storage media 630 (e.g., one or more mass storage devices) storing application programs 633 or data 632. Wherein the memory 620 and the storage medium 630 may be transitory or persistent storage. The program stored in the storage medium 630 may include one or more modules (not shown), each of which may include a series of instruction operations on the holographic-based multi-modal interaction device 600. Still further, the processor 610 may be configured to communicate with the storage medium 630 to execute a series of instruction operations in the storage medium 630 on the holographic-based multi-modal interaction device 600.

The holographic-device-based multimodal interaction apparatus 600 may also include one or more power supplies 640, one or more wired or wireless network interfaces 650, one or more input/output interfaces 660, and/or one or more operating systems 631, such as Windows Server, mac OS X, unix, linux, freeBSD, and the like. It will be appreciated by those skilled in the art that the holographic-device-based multi-modal interaction device structure illustrated in fig. 6 does not constitute a limitation of the holographic-device-based multi-modal interaction device, and may include more or fewer components than illustrated, or may combine certain components, or a different arrangement of components.

The present invention also provides a multi-modal interaction device based on a holographic device, the multi-modal interaction device based on a holographic device comprising a memory and a processor, the memory storing computer readable instructions which, when executed by the processor, cause the processor to perform the steps of the multi-modal interaction method based on a holographic device in the above embodiments.

The present invention also provides a computer readable storage medium, which may be a non-volatile computer readable storage medium, and may also be a volatile computer readable storage medium, having stored therein instructions that, when executed on a computer, cause the computer to perform the steps of the holographic device-based multimodal interaction method.

It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, which are not repeated herein.

The integrated units, if implemented in the form of software functional units and sold or passed as separate products, may be stored in a computer readable storage medium. Based on the understanding that the technical solution of the present invention may be embodied in essence or in a part contributing to the prior art or in whole or in part in the form of a software product stored in a storage medium, comprising instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a read-only memory (ROM), a random access memory (random access memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

The above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. The multi-mode interaction method based on the holographic equipment is characterized by comprising the following steps of:

collecting infrared reflection signals of a handheld handle device of a target user to obtain infrared reflection signals;

2. The multi-modal interaction method based on holographic equipment of claim 1, wherein the performing time series alignment processing on the infrared reflection signal, the target visual line and the brain wave data set to obtain target fusion data comprises:

3. The multi-modal interaction method based on holographic devices of claim 1, wherein the extracting the multi-type feature data from the target fusion data to obtain location feature data, focus location feature data, and emotion feature data comprises:

4. The multi-modal interaction method based on holographic equipment of claim 3, wherein the extracting multi-type feature data from each layer of the filtering fusion data to be processed to obtain the location feature data, the attention location feature data and the emotion feature data includes:

5. The multi-modal interaction method based on holographic equipment of claim 1, wherein the spatially dividing the preset three-dimensional display space by the position feature data and the attention position feature data to obtain the corresponding target space region comprises:

6. The multi-modal interaction method based on holographic devices of claim 1, wherein the constructing holographic display space coordinates by the target interaction instruction, generating a holographic display space coordinate set, and performing holographic device matching by the target interaction instruction, determining a target holographic device set, comprises:

7. The multi-modal interaction method based on holographic devices of claim 1, wherein the constructing a three-dimensional virtual reality space by the set of target holographic devices based on the set of holographic display space coordinates and transmitting the three-dimensional virtual reality space to virtual reality glasses of the target user for holographic image display comprises:

8. A holographic device-based multi-modal interaction apparatus, the holographic device-based multi-modal interaction apparatus comprising:

9. A holographic device-based multi-modal interaction device, the holographic device-based multi-modal interaction device comprising: a memory and at least one processor, the memory having instructions stored therein;

the at least one processor invokes the instructions in the memory to cause the holographic device-based multimodal interaction apparatus to perform the holographic device-based multimodal interaction method of any of claims 1-7.

10. A computer readable storage medium having instructions stored thereon, which when executed by a processor, implement the holographic device-based multimodal interaction method of any of claims 1-7.