CN107680165B

CN107680165B - HoloLens-based computer console holographic display and natural interaction application method

Info

Publication number: CN107680165B
Application number: CN201710872591.5A
Authority: CN
Inventors: 陈玥同; 陈杰
Original assignee: CETC 28 Research Institute
Current assignee: CETC 28 Research Institute
Priority date: 2017-09-25
Filing date: 2017-09-25
Publication date: 2021-01-26
Anticipated expiration: 2037-09-25
Also published as: CN107680165A

Abstract

The invention discloses a holo-graphic display and natural interaction application method of a computer console based on HoloLens, which sequentially comprises the following steps: reducing the size according to the actual computer console in an equal proportion, establishing a computer console model, constructing a virtual scene according to the model, introducing the virtual scene into HoloLens glasses, and displaying the virtual scene in a real space in a holographic manner; environment understanding is carried out through the HoloLens by utilizing an SLAM technology, and a user is positioned to obtain real environment data; and friendly interaction between a user and the virtual computer console is realized through the HoloLens. The invention aims at the 3D display and natural interaction application of the computer console, adjusts the height, the size, the screen inclination, the position and the angle of a single console and the combination and the arrangement among various consoles by environment perception and virtual object projection and by natural interaction modes such as sight line, gestures, voice and the like, and can also change different materials, colors, styles and styles according to the preference of users.

Description

HoloLens-based computer console holographic display and natural interaction application method

Technical Field

The invention relates to a man-machine interaction and artificial intelligence technology, in particular to a hologrAN _ SNhic exhibition and natural interaction AN _ SNplication method of a computer console based on HoloLens.

Background

In the 60 s of the 20 th century, people have a preliminary exploration on an Augmented Reality (AR) technology, in 2010, AR and wearable equipment are rapidly developed, in 2012, the first generation of Google Glass comes out, the intelligent hardware formation of the AR technology is a trend, apples, Intel, highpass, Microsoft and other huge heads are added in succession, the augmented reality equipment comes out endlessly, such as Epson's Moverio Pro BT200, Meta2, Vuzix M100, Daqri intelligent helmet, ODG R-7, HiAR Glasses on a light platform, and the AR prospect is widely seen. The first generation of Google Glass announced failure and production halt in 2015, but the development trend that wearable augmented reality devices are expected to be accepted by the whole society is not hindered, and more wearable devices play a very important role in various industries. In 2017, the Google glasses return 7 months, an enterprise version Google Glass is released, the service life of a battery is prolonged, the comfort level is improved, the glasses have the advantages of being strong in central processing unit, 800-ten-thousand-pixel cameras and the like, the calculation processing is simple, and the product positioning is still used as external extension of equipment such as a mobile phone and a tablet. During the first generation of Google Glass's exit from the market, microsoft introduced a unique Mixed Reality (MR) device, HoloLens, with the reality and naturalness of the wearing experience far exceeding that of any AR glasses on the market.

The iteration of hardware, the customization of software, the continuous promotion of product content for AR technique is known by more and more users to it is completely open head and corner in many fields, for example game product, advertisement marketing, education (early education, toy etc.), travel service, clothing, medical treatment (medical teaching, operation live broadcast, remote medicine prison, remote ward round etc.), manufacturing (intelligent maintenance, electrical machinery etc.), masses spread etc. field.

In the prior art, a holographic and natural interaction technology aiming at a computer console does not exist at present, the computer console is time-consuming and labor-consuming to manufacture, once the model is determined, any small change is difficult to make, particularly, the computer console needs to be produced in quantity for large-scale enterprises, and the computer console is pulled one by one to move the whole body, and the merchants are difficult to meet various user groups, so that the traditional computer console is high in manufacturing cost, long in period, slow in modification and very inconvenient to use.

Disclosure of Invention

The purpose of the invention is as follows: the invention aims to solve the defects in the prior art and provides a holographic display and natural interaction application method of a computer console based on HoloLens.

The technical scheme is as follows: the invention relates to a holographic display and natural interaction application method of a computer console based on HoloLens, which sequentially comprises the following steps:

(1) and (3) holographic display: reducing the size according to the actual computer console in an equal proportion, establishing a computer console model, constructing a virtual scene according to the model, introducing the virtual scene into HoloLens glasses, and displaying the virtual scene in a real space in a holographic manner;

(2) fusing deficiency and excess: environment understanding is carried out through the HoloLens by utilizing an SLAM technology, and a user is positioned to obtain real environment data;

(3) natural human-computer interaction: friendly interaction between a user and a virtual computer console is realized through the HoloLens and an interaction level model;

(4) and (4) performing interactive performance evaluation and information query on the natural interaction in the step (3).

The virtual computer console is projected in a real space by the technical scheme, so that perfect fit with a real environment is achieved; the information can be displayed in various forms for users to inquire and select; a natural and efficient man-machine interaction mode brings brand-new experience, and the style of a computer console is adjusted independently, so that each user is satisfied.

In the step (1), firstly, the size of the computer console is adjusted according to the actual space size of a room, and then the computer console is placed on the ground according to the actual needs of a user: in practical application, a user can place the computer console on the ground and lean against a wall or furniture, the user can independently select a proper and reasonable direction, the size of the computer console can be automatically adjusted according to the size, the space or the open space area of an actual room, the environment layout can be combined, the space is saved to the maximum degree, and the neat and attractive effect is achieved.

In the step (1), equal-proportion modeling is carried out according to a real computer console, a model of the computer console is established through 3Dmax or Maya modeling software (the equal-proportion model comprises the following parameters of integral appearance, built-in parts, built-in structures and real texture maps), and in the modeling process, in order to simplify calculation amount of HoloLens and improve operation speed and rendering strength, a three-dimensional model optimization method comprising geometric feature optimization and texture optimization is adopted. In order to ensure the consistency of the virtual object under different illumination conditions, the problem of maintaining the self-adaptive illumination attribute of the surface of the virtual object is solved by adopting an image-based relighting drawing method, which comprises two steps of mapping an environment light map and synthesizing relighting; and then, importing the optimized model into Unity3D to build a virtual scene, adding a camera and natural illumination into the scene, adjusting the position, angle and size of a virtual object, ensuring that the initial state appears in the middle of the scene, and finally, issuing the virtual scene to HoloLens glasses through wireless transmission to realize 360-degree holographic viewing.

When a virtual scene is built, an interactive object, an interactive mode, a presentation form and the like are designed according to the characteristics and interactive characteristics of virtual objects, a virtual object assembly is combined, a father node is set, physical attributes and interactive functions are added to each group of virtual objects needing interaction, and the physical attributes comprise Mesh render, Text Mesh, Box compiler, Animation and the like. The interactive functions include display, hide, move, rotate, etc. When a virtual scene is built in Unity3D, almost all modules need to be designed to determine the combination (which determines the interactive object), the interactive mode and the presentation form among the virtual parts. In practice, essentially all work except modeling is done in Unity 3D.

The interaction hierarchical model in the step (3) comprises a physical layer, an information layer and a cognitive layer; the output of the computer console application system is a visual holographic three-dimensional interface, the input is a plurality of natural interaction modes, and the physical layer realizes the detection, identification and positioning of the natural interaction modes; the information layer researches and identifies gestures, actions and voice command semantics on the basis of various input information obtained by a physical layer, and completes the realization of virtual object operation through HoloLens glasses; how the cognitive layer realizes the task to complete the self-defining function on the basis of natural interaction.

During natural man-machine interaction, calculating the position of a head cursor, the gesture direction and the action through a gesture distance obtained by depth camera data, a user position obtained by SLAM and the placement position of an object in a virtual scene, and further performing sight tracking and gesture interaction on the virtual object; and performing voice interaction on the virtual object through a voice recognition command of a specific word. The interaction mode in natural interaction comprises the following two combinations:

the first combination mode is as follows: the method is completed through sight and gesture operations: firstly, changing the material, color and style of the computer console according to the self aesthetic value; secondly, adjusting the height, the size and the gradient of the computer console to be suitable for the stature of the user; the second combination mode: the method is completed through sight and voice instructions: comparing computer operation platforms of different models to select the most favorable type with highest price ratio; secondly, the design scheme is modified in real time, parts are replaced, added or deleted, and the working mode which is most suitable for the user is selected according to the habit of the user.

In the step (4), the interactive performance is evaluated by establishing an interactive evaluation index system, and the specific method comprises the following steps: establishing an evaluation system comprising three-level targets, analyzing evaluation tasks of the evaluation sub-targets one by one, giving quantitative reference, formulating a calculation method of evaluation item coefficients, and determining each evaluation item coefficient.

The information query method includes the following two methods:

(A) the method comprises the steps that information content to be inquired and obtained is defined in the HoloLens glasses in advance, a designer considers an object to be inquired, information to be learned and a display form of a favorite in advance, when a virtual scene is built, event response is added to a virtual assembly, and when a user triggers an event through interaction, the HoloLens glasses respond and display a feedback result;

(B) the method comprises the following steps of inquiring in real time by virtue of a search server, taking an inquired object and a display form into consideration by a designer, and returning an acquired information result by the search server, wherein the specific process comprises the following steps: when a virtual scene is built, a virtual assembly and a father node are designed, the virtual assembly and the father node comprise a combination form and a name of a virtual part, a user transmits an object to be inquired to a search server (a PC (personal computer) terminal and a handheld terminal) through a network by interaction, then searches information on the search server, obtains a result, feeds the result back to a HoloLens glasses through the network, and finally presents the result in the HoloLens glasses in a proper presentation form, wherein communication between the HoloLens glasses and the search server is involved.

For example, in practical applications, the holographic display capability based on the HoloLens glasses for users to view the holographic computer console includes: the overall style of the operation table, the internal pattern of opening/closing the drawer, all parts, information prompt and detailed explanation of various parts, comparison among operation tables of different models, combination and arrangement of the computer operation table and the like are the same as the appearance of the operation table in the front, and even the prompt information seen by the user can be the content known only by a user manual, explanation of a salesperson or hundred-degree search of the real operation table.

Has the advantages that: compared with a material object computer console, the holographic display and natural interaction technology based on HoloLens has the following advantages:

(1) the perfect fit between the holographic computer console and the real world enables the size, height, combination quantity, placement angle and the like of the console to be adjusted automatically according to the environment layout, so that the space is saved to the maximum extent, and the neat and beautiful effect is achieved;

(2) the holographic display of the computer operation table not only can enable a user to view the whole style of the operation table in all directions and at multiple angles at 360 degrees, but also can view the internal pattern and part structures of the operation table in multiple granularities and multiple layers;

(3) and a natural man-machine interaction mode enables a user to become a designer to freely change the style and color of the operation table and modify a scheme in real time.

Drawings

FIG. 1 is a schematic flow diagram of the present invention;

FIG. 2 is a schematic diagram illustrating the principle of the fusion between real and virtual entities;

FIG. 3 is a diagram illustrating information querying in an embodiment;

FIG. 4 is a diagram illustrating gesture interaction according to an embodiment;

FIG. 5 is a diagram illustrating voice interaction in an embodiment.

Detailed Description

The technical solution of the present invention is described in detail below, but the scope of the present invention is not limited to the embodiments.

As shown in fig. 1 and fig. 2, the holographic display and natural interaction application method for a computer console based on HoloLens of the present invention sequentially comprises the following steps:

(1) and (3) holographic display: firstly, adjusting the size of a computer console according to the actual space size of a room, then placing the computer console on the ground according to the actual needs of a user, establishing a model of the computer console through 3Dmax or Maya modeling software, introducing the established model into Unity3D to build a virtual scene, and then publishing the virtual scene to HoloLens glasses through wireless transmission to realize 360-degree holographic viewing;

(3) natural human-computer interaction: holographic display and natural interaction are achieved through the HoloLens, when natural man-machine interaction is conducted, the head cursor position, the gesture direction and the movement are calculated through the gesture distance obtained through the depth camera data of the HoloLens, the user position obtained through the SLAM and the placing position of an object in a virtual scene, and then sight line tracking and gesture interaction are conducted on the virtual object; performing voice interaction on the virtual object through a voice recognition command of a specific word;

In the modeling process, in order to simplify the calculation amount of the HoloLens, improve the running speed and the rendering strength of the virtual object and meet the requirement of real-time drawing, various three-dimensional model optimization methods are comprehensively adopted, including two categories of geometric feature optimization and texture optimization.

1) Geometric feature optimization

Model-based geometric feature optimization may be optimized in terms of points, lines, faces, etc. features of the model. The basic operation is simplified by adopting the most models in the current algorithm of a reduction law master. The method achieves the purpose of simplifying the model by repeatedly and sequentially deleting the geometric elements which have small influence on the characteristics of the model and triangularizing the geometric elements again. According to the difference of the deleted geometric elements, methods such as a vertex deletion method, an edge folding algorithm, a triangular patch folding algorithm and the like can be adopted, wherein the vertex deletion method occupies less memory and is high in speed, but has different damage to the display effect of the model and is deficient in the aspect of fidelity; the edge folding method has high fidelity, but has very large computation amount; the triangular patch folding algorithm is a continuation of the edge folding algorithm, and the efficiency is relatively higher. According to the geometric characteristics and the fineness requirement of the model displayed by the computer console, the optimization method is comprehensively used, and the calculated amount is reduced as much as possible on the basis of ensuring the fineness requirement of the model.

2) Texture optimization

The model appearance attribute of the three-dimensional model, particularly the use of texture, has great superiority in the aspects of improving the reality of the model and enhancing the accuracy of the model. The number of polygons of the model can be effectively controlled by using the texture, the complexity of the model is reduced, and the file volume is reduced while the refreshing frequency of the screen is improved. Therefore, in the model, texture becomes one of important factors affecting the amount of data. By using the texture mapping technology, the surface characteristics of the multi-scene model can be described under the conditions of reducing the calculation of the surface characteristics of the model and not increasing the number of polygons, so that the immersion of the scene is enhanced, and the rendering speed is increased.

Texture and mapping are common implementations of texture mapping techniques. The three-dimensional geometric body can be provided with surface characteristic maps such as transparency, diffuse reflection, self-luminescence and the like by utilizing materials, and then the surface texture of the three-dimensional geometric body can be replaced by pictures, so that the surface subtle effect of the three-dimensional geometric body is enhanced under the condition of not increasing the complexity of a model. To maximize the amount of data to be compressed, methods such as repeated mapping, sub-texture techniques, and bulletin boards may be used in the arrangement of texture image data. The modeling efficiency and the utilization rate of system resources are improved.

When the external illumination condition of the virtual object is changed, in order to maintain the illumination attribute of the object, the problem of maintaining the adaptive illumination attribute of the surface of the virtual object is solved by adopting image-based relighting drawing. Image-based relighting rendering illuminates a real or computer-generated scene with an illuminated scene image showing the world, to obtain a natural and realistic virtual scene image, mainly comprising two steps of ambient light map mapping and relighting composition. 1) And mapping the ambient light map. Due to the superpositionability of the light sources, ambient light can be decomposed into a set of point light sources incident from various directions. The decomposition process can easily use spherical mapping to map the environment light image to the two-dimensional image, so that the relighting image can be conveniently obtained by searching the sample image of the corresponding point and performing linear superposition in the relighting calculation process. The acquisition simulation of the light source needs to render the texture, the illumination intensity and the shadow generated between the objects, which are carried by the light source, and can be realized based on a metal mirror surface spherical projection method.

2) And (5) re-illumination synthesis. The process of base function-based re-illumination is the process of combining the sample images according to the new ambient light map. The new ambient light map includes a texture image of the captured image, a point source, a parallel source, or an area light.

As shown in fig. 2, in the virtual-real fusion process of step (2), the SLAM technique with the powerful HoloLens can quickly scan the indoor environment and provide accurate position and object data. The Hololens perfectly fuses virtual objects with the real world, and there are three main presentation forms: (1) the environment is enhanced: in some 'blank' areas, virtual digital content is attached to provide a new virtual content display mode for a user, so that the purpose of enhancing the environment (reality) is achieved. Such as a movie picture displayed on a wall, a human skeleton floating in the air, a puppy playing on the ground. (2) Environment fusion: the essence is to overlay digital content in the real environment. But in contrast, it relies more on superposition with real objects. For example, a motorcycle is designed, a partial frame already exists in the motorcycle, and Hololens superimposes a virtual appearance design on the motorcycle to achieve the overall effect. (3) Virtual space formula: a completely virtual environment is created that has no relationship to the real world at all, allowing the user to appear to be in another world. Such as simulating a spark surface. The invention uses the environment-enhanced presentation form, places the designed virtual computer console on the real ground, clings to the wall or furniture, combines the environment layout, maximally saves the space, and achieves the effects of tidiness and beauty.

And (3) realizing friendly interaction between the user and the virtual computer console through the HoloLens and the interaction level model. The invention adopts an interactive hierarchical model, which comprises a physical layer, an information layer and a cognitive layer. The output of the computer console application system is a visual holographic three-dimensional interface, and the input is a plurality of natural interaction modes. The physical layer considers how to detect, identify and locate natural interaction modes; the information layer is mainly used for researching definition and recognition of semantics such as gestures, actions, voice commands and the like on the basis of various input information obtained by the physical layer, and achieving the operation of the virtual object through the HoloLens glasses; the cognitive layer is used for researching how to realize tasks on the basis of natural interaction and finishing a self-defining function.

And (3) carrying out experimental verification on the human-computer work efficiency of various natural interaction modes and comprehensive application modes by using an efficiency evaluation model, establishing an interaction performance evaluation system of a computer console application system, summarizing and summarizing human-computer interaction design principles and application modes, establishing human-computer interaction method evaluation indexes and carrying out quantitative analysis on evaluation targets one by one. The principle of natural, friendly and smooth interaction in the system is comprehensively considered.

The interactive evaluation system refines an overall evaluation target by adopting evaluation target number based on user interactive requirements and interactive design principles, establishes an evaluation system comprising three-level targets, analyzes evaluation tasks of evaluation sub-targets one by one, gives quantitative reference, formulates a calculation method of evaluation item coefficients, determines each evaluation item coefficient, and establishes an evaluation model as follows: the comprehensive work efficiency performance of the interactive system is tested by carrying out quantitative and qualitative experimental analysis and work efficiency test evaluation on the interactive system, and the naturalness, feasibility and high efficiency of fusion of various interactive models are verified.

The query of information is divided into two forms: firstly, information contents which are written into the glasses in advance and are inquired and obtained by a user are well defined in advance; secondly, real-time query is carried out, information which needs to be queried by a user is firstly transmitted to a search server (such as a PC end), then a result is searched in the search server (such as the PC end), and finally the result is fed back to the glasses and displayed by the glasses, wherein communication from the server to the HoloLens glasses is involved. In short, one is an inquiry inside the glasses, and one is an inquiry of the glasses-server-glasses.

Fig. 3 is a design diagram of information query, and the viewed contents are mainly divided into two categories: the virtual console model and the prompt information and display content related to the computer console are provided.

Viewing a model of a virtual console, comprising: overall/local style, internal pattern of opening/closing the drawer/opening the cabinet door, construction and disassembly structure of all parts, comparison among operation tables of different models, combination and arrangement of computer operation tables and the like. The user can watch the virtual computer console in an omnibearing, multi-angle and multi-granularity manner.

The method for checking the prompt information and the display content of the computer operation table comprises the following steps: brief introduction of computer operation table, instruction manual, notice, function introduction and demonstration of parts, comparison of advantages and disadvantages of different types of operation tables, and the like. The display of information and content has three characteristics of multi-level panel query, diversified forms and classified display. The brief introduction, the instruction manual and the notice of the computer console are displayed in the primary panel, the detailed information is displayed in the secondary panel and is presented in a text form, and a user can roughly know and basically master the computer console according to the prompted information; the function introduction of the parts is shown in the first-level panel, mainly by characters, and the names and function descriptions of the parts are simply introduced. The function demonstration is played in a video mode in the secondary panel. Meanwhile, the operation mechanism is also displayed in a two-level panel in a 3D animation mode, and the internal structure (perspective view), the operation mechanism, the split structure and the like are in a list, so that a user can conveniently understand the principle, and the method is mainly used for teaching, popularizing knowledge and the like; the advantages and the disadvantages of the operation tables of different types are compared in a primary panel in a table form, characters are combined with pictures, the advantages and the disadvantages are described by the characters, the patterns are shown by the pictures, the operation tables of different types are shown by a 3D model in a secondary panel of the patterns, a user can carefully check and compare the operation tables, and a model suitable for the user is selected. The prompt information of the first-level panel is concise and mainly comprises short characters; the information displayed by the secondary panel is rich, detailed, vivid and visual, and mostly appears in the forms of large-segment characters, pictures, videos, 3D animations and models. The displayed information content and form can be increased/deleted/modified/perfected according to the user requirements, and the data of the information content can be derived from a real operation table user manual, a sales promotion person explanation, hundred-degree search or simulation data and the like.

The working principle of the holographic display and natural interaction application method of the computer console based on the HoloLens is as follows:

the method is based on the technical capability of HoloLens glasses, and mainly has three capabilities of virtual-real fusion, holographic display and natural interaction.

The technical capabilities of the HoloLens-based glasses were then:

designing the content and the prompt information viewed by the user, comprising the following steps: brief introduction, instruction manuals, cautions and the like of the computer console are presented in a text form, and detailed information is displayed in the secondary panel; function introduction and demonstration of parts, wherein the function introduction is displayed in a primary panel and mainly takes characters, the function demonstration is displayed in a secondary panel and mainly takes videos; comparing the advantages and the disadvantages of different types of operation tables in a table form; the development/evolution history of the computer console is displayed in the form of pictures combined with characters. The prompt information of the first-level panel is concise and mainly short characters; the information displayed by the secondary panel is rich, detailed, vivid and visual, and is mostly in the form of large-segment characters, pictures and videos. The design of the invention has three characteristics of multi-level panel query, diversified forms and classified display.

And (II) designing gesture operations of the user, including clicking, moving, rotating, zooming, backing, resetting, placing and the like. Wherein, clicking the change of attributes such as material/color of the corresponding computer console, the display of information of each panel, and the like; the movement operation corresponds to the movement of the placing position of the computer console, the opening of a drawer and the like; the rotation operation corresponds to the opening of a cabinet door (left, right and back) of a computer console, the integral rotation of 360 degrees for viewing, and the adjustment of the inclination of components such as a screen/a microphone and the like; the zooming operation corresponds to the change of the whole size of the computer console, the adjustment of the sizes of parts and the like; the backward operation returns the result of the previous step aiming at the misoperation of the user, and 5 steps can be returned at most; if the computer console is reset, returning to the original state of the computer console before any change is made; the virtual computer operation table corresponding to the placing operation is placed on the real ground.

And (III) designing a user voice instruction (English) comprising 'Change Type', 'Compare Difference Models', 'Group/Disperse', 'Add/Delete Microphone/Telephone/Mouse/Keyboard …' and the like. Wherein "Change Type" is a Change Type, one Type appearing at a time; the 'Complex Differencent Models' are Different Models in comparison, and are of various types at the same time, so that the visual comparison is convenient; "Group/dispose" is the same type of combination/individual placement; "Add/Delete Microphone/Telephone/Mouse/Keyboard …" is to Add/Delete each component to conform to the working habit of the individual. In the second design, the modes of movement, rotation, zooming, backing, resetting, placing, etc. can be switched by single-click operation with gestures or by voice commands, and respectively correspond to "Move", "Rotate", "Scale", "Back", "Reset" and "Replace".

Example 1:

in this embodiment, there are three main interaction modes for the HoloLens: gaze (staring ray), gettrue (Gesture), and Voice (Voice).

1. And (2) Gaze: the primary input mode of HoloLens determines the position of the sight line light spot according to the position and the angular direction of the head of the user, and a forward ray extends between the position of the sight line light spot and two eyes of the head of the user, and the ray can identify a collided object. The sight line light point is in a form similar to a cursor of a desktop system and is used for determining and selecting the holographic object of the operation. The embodiment selects the operation object by utilizing the size function of the HoloLens glasses, and is the premise and the basis of gesture interaction and voice interaction.

2. Gesture: when gesture interaction is carried out, a hand needs to be placed in a front area of Hololens, and there are three core interaction gestures: click (press), release (release) and bloom (bloom). Five gesture operation responses are derived by combining three interaction gestures: 1) click (Air tap), i.e. a single click operation is done in the Air for selecting or activating digital content. The operation is as follows: the index finger straightens first and then bends downward and then straightens again. This operation can also be accomplished by the Hololens accessory "HoloLens Clicker"; 2) return (Bloom), which is a system level gesture, the user returns to the Start Menu (Start Menu) as if the Start Menu was launched by clicking the Windows logo key on the keyboard. The operation is as follows: holding the fist with the palm facing upwards, and then spreading the five fingers like flowers blooming; 3) and long press (Hold) for triggering a secondary operation, such as displaying a secondary operation menu. The operation is as follows: clicking and keeping the index finger in a downward state; 4) operation (Manipulation), holding the click gesture, the screen response and the hand motion are calculated in the following manner of 1: 1, absolute movement of the finger relative to the entire AR mixture space; 5) navigation (Navigation), relative motion within standard 3D space, holding a tap gesture.

In the embodiment, the gesture interaction application shown in fig. 4 is designed by combining three core interaction gestures and five combined gesture operations, and is divided into seven semantic gesture actions, namely "click", "back", "reset", "place", "move", "rotate" and "zoom", respectively. Wherein:

(1) the "click" semantic operation is mainly used for: 1) displaying and hiding various types of information, and finally displaying, inquiring and hiding all information such as characters, pictures, tables, videos, 3D animations and the like on a primary panel and a secondary panel, so that a user can deeply know attribute information such as a user manual, function introduction, style comparison and the like of a computer console; 2) the style of the color, the material, the style and the like is changed, the color is ten kinds of choices, namely, milky white, grey blue, sky blue, beige, brown, orange red, light purple, olive and water green, and each color corresponds to the choice of 1-100% of transparency. The material is seven kinds of optional, is "cast iron material", "aluminum alloy", "titanium alloy", "wood", "stainless steel", "macromolecular material", "lean pipe" respectively, and the user can select different materials according to the different subassemblies of computer operation panel, if the operation panel mesa selects "wood", the supporting legs selects "cast iron material", can make up the collocation wantonly. The style can be five options, namely traditional style, fresh style, garden style, European style and simple style, each style has default color matching and fixed material, and a user can change local color or material on the basis of a certain style. The design of each style is determined according to the prior market research and can be changed according to the requirements of field users.

(2) The "back-off" semantic operations are mainly used for: returning to the previous step, the invention stores and records user operation, including all interactive operation of gestures and voice, prevents the user from misoperation or unsatisfied current operation result, and can return to the previous step or previous steps, and at most five steps.

(3) The "reset" semantic operation is mainly used for: returning to the initial state before the user operation, the user may not be satisfied with the current state and the user requirement cannot be restored by the 'back-off' operation after the operations of moving the placement position, changing the size and the dimension, changing the color style and the like, and the 'reset' is the redesign of the initial state.

(4) The "put" semantic operations are mainly used for: the virtual 3D computer console is placed on the real ground, and the virtual console model is perfectly attached to the real environment.

(5) The "move" semantic operations are mainly used for: 1) the changing of the placing position of the virtual object combines the placing operation and the moving operation, and on the basis that the virtual object is placed on the real ground, a user changes the placing position through the moving operation, but the virtual computer console is always placed in a manner of being attached to the ground no matter how the virtual computer console moves; 2) the drawer is pulled open, and a user opens/closes the drawer through the back-and-forth movement of gestures, so that objects in the drawer and the space size of the drawer are further watched carefully; 3) similarly, the user extracts/replaces the parts through the back-and-forth movement of the gestures, and further knows the functional principle, the internal structure, the use purpose and the like of the parts by combining the display of the information in the click.

(6) The "rotate" semantic operations are mainly used for: 1) opening the cabinet door, and finishing the opening/closing of the right/rear cabinet door by ' long pressing ' and moving to the right/left, ' long pressing ' and moving to the left/right ' by a user, wherein the rotation angle of the left/right cabinet door is between 0 and 180 degrees, and the rotation angle of the rear cabinet door is between 0 and 90 degrees; 2) adjusting the inclination of the parts, and enabling a user to press for a long time and move to the right/left to finish forward/backward inclination of the parts, wherein the rotation angle is between-90 degrees and 90 degrees; 3) the whole 360-degree rotation, the whole computer console rotates at any angle, and the whole style and the appearance of the computer console can be observed in all directions on the premise that the position of a user is unchanged.

(7) The "zoom" semantic operation is mainly used for: 1) the size of the whole is changed, the operation table placed on the real ground is changed by zooming in and out in combination with the placing operation, and the operation table is adapted to a real space according to the environment layout. The combination of three operations of placing, moving and zooming saves the space to the maximum extent, and achieves the effects of tidiness and attractiveness. Similarly, no matter how "move" and "zoom", the virtual computer console is always attached to the ground; 2) due to the adjustment of the sizes of the parts, the sizes of the parts can be changed by a user according to personal preference or the surplus degree of spaces such as drawers/cabinets and the like. All the gesture operations are not divided into left hand or right hand.

3. Voice: there are three main forms of speech input: 1) voice commands, when the user speaks the keywords, the preset actions are called; 2) dictation, i.e., voice to text, dictation features are used to convert user voice to text input, while supporting content inference and event registration features; 3) grammar Recognition, relying on a set SRGS (Speech Recognition Grammar Specification Version 1.0) file, a series of Grammar rules are defined in the file for Speech Recognition. The voice interaction and the gesture operation can be defined to correspond to the interaction action with the same meaning, for example, the click and return operation in the gesture interaction can be completed by the voice instruction "Select", "Hey Cortana, Go Home".

In the embodiment, a voice command form is adopted, and the voice interaction application shown in fig. 5 is designed, and there are mainly five voice instructions:

(1) the Change Type voice interaction instruction is mainly used for changing types/models of computer operation platforms, only one Type of computer operation platform appears in a scene at a time, and a user can conveniently check the operation platforms of different models repeatedly. The invention comprises three types of operation platforms, wherein each type of operation platform is respectively divided into different models, and the specific voice interaction is divided into two forms: 1) type (b). The voice instructions of "Change Type One", "Change Type Two" and "Change Type Three" correspond to Three different types of operation platforms respectively, and the presented model is the default model of each Type of operation platform; 2) the specific model name. The voice instruction 'Change Type xxx' corresponds to an operation table of a specific model, the presented model is the operation table of the model, and xxx is the model name of the operation table.

(2) The 'Complex Difference Models' voice interaction instruction is mainly used for comparing operation platforms of Different Models/types, and the operation platforms of various Models appear in a scene at the same time, so that the voice interaction instruction is convenient for visual comparison, and is particularly suitable for the situation that a user selects a plurality of more favorite computer operation platforms and cannot take the operation regularly. Specific voice interactions fall into two forms: 1) all are enumerated. The voice command 'Complex Difference Models' displays all Models of operation platforms; 2) some of which are listed. When the user only needs to Compare a few of the Models, the user only needs to know the corresponding model names, and adds and before the last model name, such as "match differential Models xxx yyyyzzz and www", wherein xxx, yyy, zzz and www are all console model names.

(3) The Group/Disperse voice interaction instruction is mainly used for combined placement and independent placement of computer operation tables of the same model, one row (multiple rows are combined and placed side by side) or multiple rows of computer operation tables may be needed for enterprise-level users, and the Group/reverse voice instruction is convenient for users to quickly watch the combined placement effect, the overall attractiveness and the occupied space of the operation tables. The specific usage is that only one Type of operation console appears in a scene after a "Change Type" voice instruction, and a "Group" operation instruction can be used for randomly combining the Type of operation console. The method comprises the following specific steps: 1) by default. The voice command 'Group' is that two voice commands are combined side by side and placed together by default; 2) the number of combinations is specified. The voice command "Group x", where x is "Three", "Four", "Five", and is combined together in up to Five side-by-side arrangement. No matter how many operation platforms of the same type are combined together, only one computer operation platform exists in a scene after the 'dispose' instruction.

(4) The voice interaction instruction of 'Add/Delete Microphone/Telephone/Mouse/Keyboard …' is mainly used for adding/deleting parts such as a Microphone/Telephone/Mouse/Keyboard and the like so as to accord with the working habits of individuals. The "Add xx" voice command interacts in conjunction with the "move" gesture, and parts can be added anywhere, regardless of how many identical parts are added, after "Delete xx" which is a Microphone, Telephone, Mouse, etc., all such parts clear. For example, two words of voice command "Add Microphone", two microphones are added in the scene, and after "Delete Microphone", the two microphones disappear completely, and other parts are similar.

(5) The 'Move/Rotate/Scale/Back/Reset/Replace' voice interaction command is mainly used for switching modes of movement, rotation, zooming, backing, resetting, placing and the like in gesture interaction. In the gesture interaction mode switching, a mode combining sight line and gesture clicking can be selected, and a voice instruction can also be utilized. All the voice commands can be defined and designed according to the requirements of users.

It can be seen from the above embodiments that, in the invention, based on the powerful computing, displaying, interacting and other capabilities of the HoloLens, the content viewed by the user, the prompted information, the displayed style, the user gesture interaction mode, the voice command and the like are designed, and the displayed content includes whole/local viewing, various comparison, introduction of a computer console, a usage manual, functional introduction and demonstration of parts and components and the like; the display style has three characteristics of multi-level panels, diversified forms and classified display; the gesture interaction and the voice command are used for controlling the computer console, and comprise changing the size/pose, adjusting the comfort level, adjusting the placing layout, changing the material/color, adding/deleting parts and the like.

In conclusion, the user can select the manufacturer and then make the computer operation table in one go, so that the manufacturer greatly saves cost, reduces the expenditure of manpower, material resources and financial resources, and can select the computer operation table which is most suitable for the user, time waste and waiting caused by equipment modification are not needed, and the satisfaction degree of the user is greatly improved.

Claims

1. A holographic display and natural interaction application method of a computer console based on HoloLens is characterized by comprising the following steps: the method sequentially comprises the following steps:

(3) natural human-computer interaction: friendly interaction between a user and a virtual computer console is realized through the HoloLens and an interaction level model; during natural human-computer interaction, calculating the position of a head cursor, the gesture direction and the action by the gesture distance obtained by the depth camera data of the HoloLens, the user position obtained by the SLAM and the placing position of an object in a virtual scene, and further carrying out sight line tracking and gesture interaction on the virtual object; performing voice interaction on the virtual object through a voice recognition command of a specific word; the interaction mode in natural interaction comprises the following two combinations:

the first combination mode is as follows: the method is completed through sight and gesture operations: firstly, changing the material, color and style of the computer console according to the self aesthetic value; secondly, adjusting the height, the size and the gradient of the computer console to be suitable for the stature of the user;

the second combination mode: the method is completed through sight and voice instructions: comparing computer operation platforms of different models to select the most favorable type with highest price ratio; modifying the design scheme in real time, replacing, adding/deleting parts, and selecting the working mode most suitable for the user according to the habit of the user;

(4) performing interactive performance evaluation and information query on the natural interaction in the step (3); the method specifically comprises the following steps:

(4.1) evaluating the interactive performance by establishing an interactive evaluation index system: establishing an evaluation system comprising three-level targets, analyzing evaluation tasks of the evaluation sub-targets one by one, giving quantitative reference, formulating a calculation method of evaluation item coefficients, and determining each evaluation item coefficient;

(4.2) the information query modes include the following two modes:

(B) the method comprises the following steps of inquiring in real time by virtue of a search server, taking an inquired object and a display form into consideration by a designer, and returning an acquired information result by the search server, wherein the specific process comprises the following steps: when a virtual scene is built, a virtual assembly and a father node are designed, the virtual assembly and the father node comprise the combination form and the name of a virtual part, a user transmits an object to be inquired to a search server through interaction through a network, then searches information on the search server, obtains a result, feeds the result back to the HoloLens glasses through the network, and finally presents the result in the HoloLens glasses in a proper display form.

2. The holo-graphic presentation and natural interaction application method of the computer console based on the holo lens of claim 1, characterized in that: in the step (1), the size of the computer console is adjusted according to the actual space size of the room, and then the computer console is placed on the ground according to the actual needs of the user.

3. The holo-graphic presentation and natural interaction application method of the computer console based on the holo lens of claim 1, characterized in that: in the step (1), equal-proportion modeling is carried out according to a real computer console, a model of the computer console is established through 3Dmax or Maya modeling software, and in the modeling process, in order to simplify calculation amount of Hololens and improve operation speed and rendering strength, a three-dimensional model optimization method comprising geometric feature optimization and texture optimization is adopted; in order to ensure the consistency of the virtual object under different illumination conditions, the problem of maintaining the self-adaptive illumination attribute of the surface of the virtual object is solved by adopting an image-based relighting drawing method, which comprises two steps of mapping an environment light map and synthesizing relighting; and then, importing the optimized model into Unity3D to build a virtual scene, adding a camera and natural illumination into the scene, adjusting the position, angle and size of a virtual object, ensuring that the initial state appears in the middle of the scene, and finally, issuing the virtual scene to HoloLens glasses through wireless transmission to realize 360-degree holographic viewing.

4. The holo-graphic representation and natural interaction application method of the computer console based on the holo lens of claim 3, characterized in that: the model of the computer console comprises the following parameters: overall appearance, built-in parts, built-in structures and real texture maps.

5. The holo-graphic representation and natural interaction application method of the computer console based on the holo lens of claim 3, characterized in that: when a virtual scene is built, an interactive object, an interactive mode and a presentation form are designed according to the characteristics and interactive characteristics of virtual objects, a virtual object assembly is combined, a father node is set, physical attributes and interactive functions are added to each group of virtual objects needing interaction, and the physical attributes comprise Mesh render, Text Mesh, Box compiler and Animation; the interactive functions include display, hide, move, rotate.

6. The holo-graphic presentation and natural interaction application method of the computer console based on the holo lens of claim 1, characterized in that: the interaction hierarchical model in the step (3) comprises a physical layer, an information layer and a cognitive layer; the output of the computer console application system is a visual holographic three-dimensional interface, the input is a plurality of natural interaction modes, and the physical layer realizes the detection, identification and positioning of the natural interaction modes; the information layer researches and identifies gestures, actions and voice command semantics on the basis of various input information obtained by a physical layer, and completes the realization of virtual object operation through HoloLens glasses; the cognitive layer researches how to realize tasks on the basis of natural interaction and completes the self-defining function.