CN109085925A - It is a kind of realize MR mixed reality interaction method, storage medium - Google Patents
It is a kind of realize MR mixed reality interaction method, storage medium Download PDFInfo
- Publication number
- CN109085925A CN109085925A CN201810954228.2A CN201810954228A CN109085925A CN 109085925 A CN109085925 A CN 109085925A CN 201810954228 A CN201810954228 A CN 201810954228A CN 109085925 A CN109085925 A CN 109085925A
- Authority
- CN
- China
- Prior art keywords
- data
- holoens
- client
- handle
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
- G06F2203/01—Indexing scheme relating to G06F3/01
- G06F2203/012—Walk-in-place systems for allowing a user to walk in a virtual environment while constraining him to a given position in the physical environment
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The present invention provides a kind of method of Holoens combination HTCVive realization MR mixed reality interaction, storage medium, method includes: to be wrapped into forming data packet being sent to Holoens client after server obtains the handle data that HTC handle includes event information and spatial information；Holoens client parses the data packet, obtains the handle data, and apply it in MR scene.The present invention not only improves user and manipulates flexibility；And greatly alleviate the fatigable problem of user's operation；Further, moreover it is possible to promote manipulation accuracy of identification, and promote the sense of reality that user interacts with MR application, significantly optimization customer experience.
Description
Technical field
The present invention relates to MR mixed reality fields, and in particular to it is a kind of realize MR mixed reality interaction method, storage be situated between
Matter.
Background technique
At present Hololens equipment provide several ways come in equipment application interact, respectively include voice,
It stares and gesture.The mode that the application mainly expands is the gesture operation that Hololens is provided.Hololens equipment is supported at present
Several gesture operations be respectively Bloom gesture and Airtap gesture.Bloom gesture is to stretch out your right hand, and finger closes up the palm
The heart is upward, and then opening palm just will appear the same menu interface of win10, or directly voice " hello Cortana ", comes
Trigger Bloom operation.Airtap gesture moves on to cursor above application icon primarily directed to the operation of application, by index finger and
Thumb stretches out, and opens, then closes up first, can trigger Airtap operation.Although mode of operation have it is so several, still
It has the disadvantage that:
1, it is worn for a long time in user's needs and uses equipment, and the application that user uses is when needing a large amount of gesture operations.
For example the application of shooting game, player need the continuous point of use hitter gesture in game process to realize firing operation.This
Player is required arm to be maintained in equipment scanning range for a long time, it is easy to cause sense tired out to player.
2, when using gesture operation, Hololens can not identification very gesture operation each time, and user needs
Hand is placed on to the front of equipment as far as possible, the identification probability of gesture could be improved in this way.Once user by hand out of
The scanning range of Hololens, Hololens equipment will no longer respond user gesture operation.
3, in order to solve above-mentioned two problems, Hololens itself provides a Click equipment, is used to simulation part and says good-bye
Gesture operation.But Click equipment is to be connect by bluetooth with Hololens, so click event itself can only be transmitted, but not
The spatial information of click event, the i.e. spatial position of click event generation can be transmitted.Due to being lost spatial information, user and MR
It is substantially reduced using the interactive sense of reality.
Therefore, it is necessary to which providing one kind can be good at solving that above-mentioned user's fatiguability, identification range are limited causes to identify
Not precisely, Click device losses spatial information and lead to problems such as the sense of reality reduce Holoens combination HTCVive realization MR
The method of mixed reality interaction, storage medium.
Summary of the invention
The technical problems to be solved by the present invention are: providing a kind of Holoens combination HTCVive realization MR mixed reality friendship
Mutual method, storage medium can promote the sense of reality that user interacts with MR application, while also greatly improve user's operation
Flexibility.
In order to solve the above-mentioned technical problem, the technical solution adopted by the present invention are as follows:
A kind of method that Holoens combination HTCVive realizes the interaction of MR mixed reality, comprising:
After server obtains the handle data that HTC handle includes event information and spatial information, it is wrapped into forming data
Packet is sent to Holoens client；
Holoens client parses the data packet, obtains the handle data, and apply it in MR scene.
Another technical solution provided by the invention are as follows:
A kind of computer readable storage medium is stored thereon with computer program, described program when being executed by processor,
It can be realized a kind of above-mentioned Holoens combination HTCVive and realize the step of method of MR mixed reality interaction is included.
The beneficial effects of the present invention are: by Holoens in conjunction with HTCVive, Click equipment is substituted using HTC handle,
Spatial information can be obtained simultaneously, realized and be obviously improved the sense of reality that user interacts with MR application；Meanwhile the introducing of HTC handle, greatly
Operation identification range is extended greatly, so that gesture operation is no longer limited to set identification range and limited gesture-type, no
The accuracy of identification of gesture is only significantly improved, and can also greatly alleviate operation and give user's bring sense tired out, to greatly mention
The flexibility of user's operation is risen.
Detailed description of the invention
Fig. 1 is the process signal for the method that a kind of Holoens combination HTCVive of the present invention realizes the interaction of MR mixed reality
Figure；
Fig. 2 is the method flow schematic diagram of the embodiment of the present invention one；
Fig. 3 is the data transmission stream journey schematic diagram of the embodiment of the present invention one；
Fig. 4 is the server operational process schematic diagram of the embodiment of the present invention two；
Fig. 5 is the client operational process schematic diagram of the embodiment of the present invention two；
Fig. 6 is the flow diagram that two client of the embodiment of the present invention parses handle data；
Fig. 7 is the contrast schematic diagram of two server coordinates system and client coordinates system of the embodiment of the present invention；
Fig. 8 is directly to sit up straight with server coordinates system after handle data with client in the case that Fig. 7 coordinate system is inconsistent
Mark the performance contrast schematic diagram of system；
Fig. 9 and Figure 10 is the motion track of the HCT handle and virtual handle comparison in the case where Fig. 7 coordinate system is inconsistent
Schematic diagram；
Figure 11 is the data analysis process schematic diagram that handle position is corrected in embodiment two；
Figure 12 is the MR scene of game effect diagram of embodiment three.
Specific embodiment
To explain the technical content, the achieved purpose and the effect of the present invention in detail, below in conjunction with embodiment and cooperate attached
Figure is explained.
The most critical design of the present invention is: promoting what user interacted with MR application by Holoens in conjunction with HTCVive
The sense of reality, while also greatly improving the flexibility of user's operation.
Fig. 1 is please referred to, the present invention provides a kind of method that Holoens combination HTCVive realizes the interaction of MR mixed reality, packet
It includes:
After server obtains the handle data that HTC handle includes event information and spatial information, it is wrapped into forming data
Packet is sent to Holoens client；
Holoens client parses the data packet, obtains the handle data, and apply it in MR scene.
As can be seen from the above description, the beneficial effects of the present invention are: believed using recordable and transmitting spatial information and event
The HTCVive equipment replacement Click equipment of breath is realized in combination with the server that the Unity engine that can be realized MR is built
The acquisition of HTC handle data, and be packaged to be transferred to after Holoens client parses and be applied in MR scene, while solving user
Fatiguability, identification not precisely, Click device losses spatial information the problems such as, acquisition be obviously improved user gesture operating flexibility,
And the advantages of promoting the sense of reality that user interacts with MR application.
Further, further includes:
Server monitors the connection request that Holoens client is sent by main thread；
After server receives connection request, a sub thread is created；
Server sends connection data to the corresponding Holoens client of the connection request by the sub thread；
The Holoens client for receiving the connection data is established and the TCP connection of server.
Seen from the above description, TCP connection is used between server and client, it is ensured that the data that client receives
Integrality；Meanwhile it distributing different threads and executing different task respectively, information transmit-receive is clearly ensured in time by the division of labor and receiving and dispatching
It is errorless.
Further, server sends the data packet to Holoens client by the sub thread.
Seen from the above description, a corresponding client, server-side will open a new thread targetedly to transmit two
Interaction data between person, to ensure the real-time and accuracy of data transmission.
Further, it is described be wrapped into forming data packet be sent to client, specifically:
The handle data are packed into including size of data, data type and data by server according to preset agreement
After the data packet of content, Holoens client is sent it to；The data type is Data；The data content is handle
Data.
Seen from the above description, it is passed again after collected handle data being carried out packing processing according to the agreement of agreement
It is defeated, so that client can accurately parse after receiving data according to agreement, data needed for obtaining；Transmission specification can also be made simultaneously
Change.
Further, the Holoens client parses the data packet, obtains the handle data, specifically:
Holoens client obtains size of data and data type from the data packet；
Data content is read from data packet according to the size of data；
According to the data type for being Data, the data content is parsed, obtains handle data.
Seen from the above description, client can quickly and accurately parse acquisition according to agreement after receiving data packet
Handle data.
Further, described to obtain the handle data, and apply it in MR scene, specifically:
Obtain the spatial information in handle data；
According to the spatial information, change position of the corresponding virtual object of the HTC handle in MR scene；Meanwhile
The event information in handle data is obtained, the event information in one frame of tabular form journal successively to trigger
Specific event information；
The corresponding each specific event information of event information described in sequence trigger switch in MR scene.
Seen from the above description, the corresponding operation of HTC handle can be shown in MR scene, and it is flexible greatly to promote manipulation
Property.
Further, if the space coordinates of server and the space coordinates of Holoens client are inconsistent, by hand
Spatial information in handle data applies to before MR scene, further includes:
The spatial information of the space coordinates of corresponding server is converted to the space coordinates of corresponding Holoens client
Spatial information.
Further, the spatial information of the space coordinates by corresponding server is converted to corresponding Holoens client
The spatial information of the space coordinates at end, specifically:
Obtain initial spatial information of the HTC handle in the space coordinates of server in Holoens client, and by its
It is assigned to the corresponding virtual object of HTC handle described in the space coordinates of Holoens client；
Obtain the virtual object correspond to Holoens client space coordinates the first forward direction vector V1 and first position
P1；
It obtains through control HTC handle, the virtual object is moved to the space coordinates for being preset in Holoens client
In a marker be overlapped after, the marker corresponds to the second forward direction vector of the space coordinates of Holoens client
V2 and second position P2；
According to the first forward direction vector V1 and position P1 and the second forward direction vector V2 and position P2, be calculated the first forward direction to
Measure the vertical vector V3 of V1 and the second forward direction vector V2；
According to vertical vector V3, the angle theta of the first forward direction vector V1 and the second forward direction vector V2 is calculated；
According to vertical vector V3 and angle theta, four element Q are calculated；
Spin matrix R is converted by the four elements Q；
The position P1 is converted into the third place P3 according to spin matrix R；
According to the third place P3 and second position P2, positional shift O is calculated；
According to the positional shift O and the third place P3, HTC handle is calculated and corresponds to Holoens client
Initial spatial information after correction in space coordinates；
According to the spin matrix R and positional shift O, acquired spatial information is corrected.
Seen from the above description, if the space coordinates of server are consistent with the space coordinates of Holoens client,
The spatial information application that directly parsing can be obtained is into the scene of client；If inconsistent, need to correct spatial information
It uses again afterwards, the position to ensure the corresponding client scene in position of the handle in server scene is error free, to protect
Card handle manipulates the consistency of the virtual object manipulation of corresponding MR scene.
Further, the handle data are JSON format.
Seen from the above description, data save as JSON format, more succinct compared to traditional XML, and inspection is facilitated to arrange
It is wrong；And interminable label symbol is not needed；Further, moreover it is possible to be supported by more mainstream speeches, the conversion of data is facilitated to conciliate
Analysis.
Another technical solution provided by the invention are as follows:
A kind of computer readable storage medium is stored thereon with computer program, described program when being executed by processor,
It can be realized a kind of above-mentioned Holoens combination HTCVive and realize the step of method of MR mixed reality interaction is included.
As can be seen from the above description, the beneficial effects of the present invention are: it is corresponding those of ordinary skill in the art will appreciate that real
All or part of the process in existing above-mentioned technical proposal, is that relevant hardware can be instructed to realize by computer program
, the program can be stored in a computer-readable storage medium, and the program is when being executed, it may include such as above-mentioned each
The process of method.
Wherein, the storage medium can be disk, optical disc, read-only memory (Read-Only Memory,
) or random access memory (Random Access Memory, RAM) etc. ROM.
Embodiment one
Referring to figure 2., the present embodiment provides a kind of methods that Holoens combination HTCVive realizes the interaction of MR mixed reality.
Using recordable and transmitting spatial information and event information HTC VIVE equipment, in conjunction with the Unity for being suitable for mobile terminal exploitation
Engine is developed to replace Click equipment.
Wherein, HTC VIVE equipment includes a helmet, a pair of of handle and two scanning signal towers, can be realized event letter
It is obtained while breath and spatial information, user will be promoted after being used in MR scene with MR using the sense of reality interacted；Meanwhile also
The triggering mode that user's operation can be improved, greatly promotes operating flexibility.
In addition, also being realized above equipment institute collection event information and space letter server and client side is built respectively
Breath is transmitted in the application at the end Hololens.Specifically, server-side is used to obtain, be packaged and transmit the spatial information and thing of handle
Part information, client are used to receive and parse these data informations.
The present embodiment is connect by using the HTCVive equipment of complete set in conjunction with the server-side that Unity engine is built
After entering SteamVR component, so that it may obtain the spatial information and event information of HTC handle.Then the information that server-side will acquire
According to protocol packing and it is transmitted in the application at the end Hololens (Holoens client), using after receiving the data according to association
View parsing data, finally update obtained data into the scene of application.The brief process of data transmission is as shown in Figure 3.
Specifically, referring to Fig. 2, the method for the present embodiment includes:
S1: server monitors the connection request that Holoens client is sent by main thread；
Server-side is opened server and is monitored, wait the connection of client according to information such as the ports set.
S2: client is opened according to information such as the ports set, request is initiated the connection to server-side；
S3: after server receives connection request, a sub thread is created；
S4: server sends connection data to the Holoens client for initiating the connection request by the sub thread；
S5: the Holoens client for receiving the connection data is established and the TCP connection of server.
If zero accident, after server receives connection request, " successful connection " information is sent to client.Client's termination
After the message for receiving successful connection, corresponding reaction is made, and prepares to receive the event information and sky of the handle from server
Between information.If client is to receive connection data, connection failure within a preset period of time.
S6: it after server obtains the handle data that HTC handle includes event information and spatial information, is wrapped into forming number
Holoens client is sent to according to packet；
Successful connection and after having sent the message of " successful connection ", during client connection, server will be supervised persistently
The event information and spatial information of HTC handle are listened and collected, then sends customer side by protocol packing for these information.Its
In, server sends connection data and data packet to Holoens client by the sub thread.
Wherein, handle data are packed into data packet and are sent to client by server, specifically:
The handle data are packed into including size of data, data type and data by server according to preset agreement
After the data packet of content, Holoens client is sent it to；Since packaged content is handle data, corresponding number
It is Data according to type.
Specifically, needing to execute according to the agreement of agreement when server-side obtains data and client parsing data.Association
The structure of view is broadly divided into three parts: size of data, data type, data content.Data content includes link information or sky
Between information and event information handle data.Wherein link information contains connection failure and successful connection；Spatial information includes
Handle is relative to the position in helmet space, rotation, scaling；Event information include specific lever knob event (Menu key,
System key, trigger key, disk key, disk upper and lower, left and right key, key is held in side).
Link information, for judging whether successful connection in client and server connection.Originally in the present embodiment
When secondary successful connection, server can send the data comprising link information and tell client successful connection to client.Such as
Fruit client initiates the connection request and does not receive the data comprising link information whithin a period of time, then judges that connection is lost
It loses.
Spatial information, for the letter about handle space position that after client connects upper server-side, server-side is sent
Breath.After client receives these data, needs to parse these data and be then applied in corresponding application scenarios.For example, mobile
When handle, server-side can send the spatial information of handle to client, can show in the scene after client parsing virtual
The mobile operation of handle.
Event information, for the click for describing button on handle that after client connects upper server-side, server-side is sent
The information of event.After client receives these data, it is also desirable to parse these data and be then applied in corresponding application scenarios.
For example, cocking key server can send the event information comprising cocking key to client, client receives letter
It is parsed after breath and shows fire in the scene.
S7:Holoens client parses the data packet, obtains the handle data, and apply it in MR scene.
Specifically, the Holoens client parse the data packet, the handle data are obtained, are specifically included:
Holoens client obtains size of data and data type from the data packet；
Data content is read from data packet according to the size of data；
According to the data type for being Data, the data content is parsed, obtains handle data.
Specifically, described obtain the handle data, and apply it in MR scene, specifically include:
Obtain the spatial information in handle data；
According to the spatial information, change position of the corresponding virtual object of the HTC handle in MR scene；Meanwhile it obtaining
Take the event information in handle data, specific event of the event information successively to trigger in one frame of tabular form journal
Information；The corresponding each specific event information of event information described in sequence trigger switch in MR scene.
Wherein, before spatial information applies to MR scene, further includes:
Whether the space coordinates for judging server are consistent with the space coordinates of Holoens client；
If consistent, directly use；
If inconsistent, the spatial information of the space coordinates of corresponding server is converted into corresponding Holoens client
Space coordinates spatial information.Specifically, conversion process includes:
Obtain initial spatial information of the HTC handle in the space coordinates of server in Holoens client, and by its
It is assigned to the corresponding virtual object of HTC handle described in the space coordinates of Holoens client；
Obtain the virtual object correspond to Holoens client space coordinates the first forward direction vector V1 and first position
P1；
It obtains through control HTC handle, the virtual object is moved to the space coordinates for being preset in Holoens client
In a marker be overlapped after, the marker corresponds to the second forward direction vector of the space coordinates of Holoens client
V2 and second position P2；
According to the first forward direction vector V1 and position P1 and the second forward direction vector V2 and position P2, be calculated the first forward direction to
Measure the vertical vector V3 of V1 and the second forward direction vector V2；
According to vertical vector V3, the angle theta of the first forward direction vector V1 and the second forward direction vector V2 is calculated；
According to vertical vector V3 and angle theta, four element Q are calculated；
Spin matrix R is converted by the four elements Q；
The position P1 is converted into the third place P3 according to spin matrix R；
According to the third place P3 and second position P2, positional shift O is calculated；
According to the positional shift O and the third place P3, HTC handle is calculated and corresponds to Holoens client
Initial spatial information after correction in space coordinates；
According to the spin matrix R and positional shift O, acquired spatial information is corrected.
Embodiment two
Referring to figure 4. to Figure 12, the present embodiment further expands embodiment one, provides a kind of Holoens combination
HTCVive realizes the concrete methods of realizing of MR mixed reality interaction.Method includes:
1. server-side
1.1 building servers
In order to successfully obtain the spatial information and event information of HTCVIVE handle, it would be desirable to build a server and use
To collect and handle these data.In the project applied by the present embodiment, we are based on Unity engine and SteamVR component
Building server-side scene is used to obtain the monitoring that the spatial information of handle connects client with foundation with event information.Work as service
After end listens to the connection of client, a new thread can be opened and be used to that the data information being collected into is handled and is packaged,
It is subsequently sent in specified client, and main thread then continues to monitor the connection status of client.Specific flow chart such as Fig. 4 clothes
Shown in end operational flow diagram of being engaged in.
Server-side is constructed there are many mode, we can select different to build mode according to specific application scenarios.Assuming that
Scene applied in the present embodiment is the shooting game of a MR, is developed based on Unity engine and C# script, then server-side
It is built using C# editions Socket sockets；And the integrality in order to guarantee data is connected using TCP connection mode using TCP
Connect the integrality for the data that can guarantee that client receives.
Specific step is as follows:
(1) server end Socket is created, and it is TCP mode that connection type, which is arranged,；
(2) ip and port are bound；
(3) longest connection request troop is set.(could support up a few people's connections)；
(4) it begins listening for.
1.2 processing handle information
Server-side is in addition to needing to handle the connection request from client, it is also necessary to the data of HTCVIVE handle are handled, it will
Handle data record is simultaneously packaged, and finally transmits data to client.It is preferred that data are saved as to the format of JSON, JSON format
It is more succinct compared to traditional XML, facilitate inspection misarrangement；And interminable label symbol is not needed, it is only necessary to which one simple
Bracket can meet, and in this way include the data of identical information, size can be smaller；More important is it by many mainstream speech institutes
It supports, such as C, C++, C#, JAVAJS etc., can very easily carry out the conversion and parsing to data.The data structure of JSON
It is determined by the agreement arranged, the structure of agreement is as shown in following table one.Wherein it is broadly divided into three parts: size of data, data class
Type and data content.
One protocol architecture of table
Size of data is used to the size of minute book time data.Client can be examined after receiving data according to this value
The integrality of data is tested, can also be determined to need that how many data read according to this value.
What the data type that data type is used to identify this transmission is.It is corresponding with number in scene in the present embodiment
According to there are two types of types: Version and Data respectively indicates " version " and " handle data ".When first connection, server-side can be sent out
The data of a Version type are sent to tell client successful connection to client, subsequent transmitted data type after success
It is all Data.Indicate the data of handle.
The main information in this data is contained in data content.Including formatted handle space information and
Event information.It is preferable to use JSON formats to save data, JSON data structure such as two institute of following table that corresponding the present embodiment uses
Show.The position of right-hand man has been respectively included in this structure, has been rotated, present speed, spatial state informations such as acceleration and corresponding
The event informations such as the state of button.Server-side converts them to corresponding JSON format after being collected into these data, then presses
Size of data is added according to agreement, the information such as data type are subsequently sent to given client end.
Two Jsons data structure of table
2. client
2.1 building clients
After server-side makes data, client can be sent the data to.When client terminal start-up, selected according to user
It selects, connection server can be attempted.Judgement can be selected to be to continue with connection according to user when connecting unsuccessful and be also to skip connection.
When successful connection, the monitoring event of meeting trigger the server is preserved the link information of client, then according to these information
Transmit data to client.And when client receives the message from server-side, it can also trigger corresponding readjustment and make phase
The processing answered.Client operational process is as shown in Figure 5.
Client is constructed equally there are many mode, can be selected different to build mode according to specific application scenarios.This reality
Applying the specific of example with scene is using Unity engine developing operation in a MR shooting game in Holoens equipment, due to
The development of games engine that Hololens recommends is Unity, and Unity is mainly developed using C# as scripting language, so client
Also it is built using C# editions Socket sockets.And the integrality in order to guarantee data, equally use TCP connection mode.Make
It can guarantee the integrality for the data that client receives with TCP connection.
Detailed process is as follows:
(1) server end Socket is created, and it is TCP to connect that connection type, which is arranged,.
(2) IP and port are set.
(3) start to connect.
2.2 processing handle information
After client receives the data from server-side, need to parse data according to the prepared protocol architecture of table one.
Preceding 8 data are read first, wherein containing data length N and data type D.Then remaining data are read according to length N
Content S determines how to handle data content S further according to the data type D obtained.At present there are two types of data types:
Link information and handle information.When connection server for the first time, it includes " link information " that server can send one to client
Data, server can send the data comprising " handle information " to client to client in game running later.Work as visitor
When family end receives " link information ", it can show successful connection and make corresponding operation.When receiving " when handle information ", client
Data can be first parsed, then by data application into scene of game.Flow chart of data processing is as shown in Figure 6.
2.3 processing link informations
When client for the first time to connect server when, will receive a data type be " link information " data.Information
Format it is consistent with table one, wherein include length information, data type information and data content information.Client parses data
It after length and data type, that is, may know that this successful connection, subsequent operation can be carried out.
2.4 processing handle event informations
After receiving the data that data type is " link information ", client has known successful connection.It can prepare
Receive the data that data type is " handle information ".And " spatial information " and " event information " are contained in handle data.It can
Find out from table two, event information therein is a list of thing, why is stored as list, be because HTC handle have it is more
A button, player's possible and a variety of operations in same frame.In order to guarantee that these operations of game experiencing require to be saved,
So recording event information using a list.
2.5 processing handle space information
As processing event information, spatial information is also required to after being handled then to use.But unlike that thing
The simple parsing of part information, spatial information is after parsing also it needs to be determined that the space coordinates and Holoens client of server
Space coordinates it is whether consistent, if unanimously, could directly use, if inconsistent, corrected, this is because handle
Spatial information is generated in the scene of server-side, and the space of space corresponding to these data and client is likely to be
It is different.Which results in some cases, handle will appear the position performance of mistake.The left side first as shown in Figure 7 is
Server-side coordinate system, wherein X-axis is towards the right, and Y-axis is towards surface.Axis of the client coordinates system on the right around normal to screen
It is rotated by 90 °, as a result towards just issuing, Y-axis is all built upon under real space its X-axis towards front-right, the two coordinate systems.Its
Middle pentagon is true handle, and rectangle is the virtual handle of server-side, and circle is the virtual handle of client.It at this time if directly will clothes
The handle data application at business end can show as shown in Figure 8 into objective end.
Due to the characteristic of MR game, player can see client's end spaces and real space simultaneously.So when player is true
When true handle being moved true handle along client space Y axis positive direction in the real space, server-side as shown in Figure 8 can be detected
Handle is moved along server-side X-axis in fact, and then server-side will move along the x-axis this spatial information and be transmitted to client, visitor
The virtual handle at family end can also be moved along the X-axis of client.Player is usually desirable to the movement of virtual hand handle under normal circumstances
Mode results in player in the above case said and is intended to make virtual handle mobile toward Y-axis in fact as true handle, still
Reflect to be the mistake moved along X-axis in client's end spaces.As shown in Figure 9 and Figure 10.
As mentioned above due to the space coordinates of server-side and the space coordinates of client and disunity, so directly will
When obtained handle space information assigns the handle in the space Hololens, unless space locating when Hololens starting application
State is as the spatiality of the HTC equipment of server-side, virtual handle position that otherwise we see in the space Hololens
Error is had with the handle position in realistic space, and the position of Hololens starting and the position server-side HTC difference are bigger, this
A error also can be bigger, so we need for the coordinate under obtained service end spaces to be transformed under client's end spaces.Conversion
Middle needs use the operation for asking four elements and spin matrix.It is as shown in figure 11 to correct handle position data flow.
Correct thinking: then starting server-side and client first get handle in the case where servicing end spaces in client
Spatial information, then by its indirect assignment to the virtual handle under Hololensde client's end spaces.Virtual handle institute at this time
The state of performance is the state under server-side.Then in the case where wearing Hololensde again, true handle is held in hand again.So
True handle is overlapped with the label object under the space Hololens as far as possible afterwards (marker be set in advance
An object in the space Hololenss, rotation information are the default value in space, i.e. X, Y, 0) rotational value of Z axis is all.This
When handle under the space HTC will appear corresponding offset certainly, the virtual handle under the space Hololens equally will appear position
It moves.Corrective operations are immediately performed after true handle is overlapped with marker.The spin matrix and positional shift corrected.Then
Subsequent obtained server-side handle space information can be converted directly using this spin matrix and positional shift, obtain
Position under the correct space Hololens.
In corrective operations, a spin matrix and positional shift can be found out.The sky of server-side transmission is received in client
Between after information, then the forward direction vector V1 and position P1 of the virtual handle first under the acquisition space Holoens obtain Holoens
The forward direction vector V2 and position P2 of marker under space.Then V1 is found out according to the multiplication cross (V1XV2) of (following formula 1) two vectors
With the vertical vector V3 of V2 vector, then by (following formula 2) two vector dots, then Arccos finds out V1 and V2 vector
Angle theta finds out a quaternary number Q according to V3 and θ according to (following formula 3) later, then further according to (following formula 4) by quaternary
Number Q is converted into spin matrix R.The position P1 from server-side is transformed into Hololens client P3 using this matrix, so
P3 and P2 is obtained into positional shift O by (following formula 5) again afterwards.Then P3 obtains final position plus positional shift O.After and
As long as continuous server-side was not restarted afterwards, these subsequent data can be done using this spin matrix and positional shift
Corrective operations.
(v3, θ)=[cos (θ/2) (sin (θ/2)) x3 sin (θ/2) y3 sin (θ/2) z]=[w, x, y, z]=Q formula 3；
P2-P3=(xp1, yp2, zp2)-(xp3, yp3, zp3)=(xp1-xp3 yp2-yp3 zp2-zp3)=O formula 5.
The present embodiment has obtained fine practice test in actual items operation, and in MR shooting game project, this is specifically answered
With in scene, player is other than it can experience game using gesture as normal use Hololens, due to joined HTC
The support of handle, handle can deepen the substitution sense of player with the form simulations weapon such as laser gun, laser sword significantly.And due to
Handle, the positions such as movable arm that player can be more flexible can be used, and do not have to the scanning range for worrying to exceed Hololens
Dropout is caused, the flexibility of player has been significantly greatly increased and alleviates the fatiguability problem of player.The effect of final MR scene
As shown in figure 12, laser gun therein is exactly mapping of the HTC handle at the end Holoens, and is debugged for convenience, specific real one
It applies in mode, can also build a log information system, it may be convenient to develop according to the form for building server-side client
Runtime data is easily seen when Hololens game.
The present embodiment cannot only be applied to game, in all Holoens applications, if necessary to the branch of outer handle
This programme can be used by holding, and be supported for application addition HTC outer handle.
Embodiment three
The present embodiment corresponding embodiment one and embodiment two provide a scene for being used in MR shooting game:
1, HTC equipment (helmet and handle) is opened, opens the base station HTC, service is opened on the computer for deploy server
Device, specified the port IP " 127.0.0.1:8080 " simultaneously open monitoring.
2, player wears and opens the Hololens helmet, then opens MR shooting game and applies (client).It chooses whether
Server is connected, directly starts game if being not connected to.If necessary to connect then according to the port of setting " 127.0.0.1:
8080 ", request is initiated the connection to specified server.
3, server end is after receiving the connection request that shooting game application (client) is sent, to corresponding client
Send the message of " successful connection ".Game application (client) confirms successful connection after the message for receiving server return, and
The maps virtual pistol of HTC handle in gaming is initialized, and prepares to parse the event information and sky of the handle from server
Between information.
4, after server has received and sent " connection request " message, during client connection, it can continue to monitor and collect HTC
Then the event information and spatial information of handle send client by protocol packing for these information.For example player is by handle
It is put into front in being held in the hand, and has pulled trigger key.Server will record down the spatial position of handle in this some column movement, and
The event information of cocking key, then these information packages are sent game application (client) by each frame.
5, it after game application (client) receives the message from server, according to corresponding protocol analysis data, obtains
The spatial information of HTC handle has been arrived, and has applied it on the maps virtual pistol of handle in gaming and changes it in sports ground
Then the shooting event information of obtained HTC handle is applied on virtual pistol by the position in scape, in gaming triggering shooting
Operation, in gaming it can be seen that virtual pistol is held in the hand by player takes chest and fire a shot.In whole process game
Hold and the practical wiping of player operation can agree with completely, virtual game picture is superimposed with real world, it is good to player
MR game experiencing.
6, after game over, client disconnects the connection with server, after server is sent to the message failure of client,
Server-side judges failure information and terminates the thread for sending information to the client.
Example IV
The present embodiment corresponding embodiment one provides a kind of computer readable storage medium, is stored thereon with to embodiment three
Computer program, described program can be realized above-described embodiment one or embodiment two or embodiment three when being executed by processor
A kind of Holoens combination HTCVive realizes the step of method of MR mixed reality interaction is included.Specific steps are no longer
It is repeated, please refers to the record of embodiment one to embodiment three in detail.
In conclusion a kind of method of Holoens combination HTCVive realization MR mixed reality interaction provided by the invention,
Storage medium not only improves user and manipulates flexibility；And greatly alleviate the fatigable problem of user's operation；Further
, moreover it is possible to manipulation accuracy of identification is promoted, and promotes the sense of reality that user interacts with MR application, significantly optimization customer experience.
The above description is only an embodiment of the present invention, is not intended to limit the scope of the invention, all to utilize this hair
Equivalents made by bright specification and accompanying drawing content are applied directly or indirectly in relevant technical field, similarly include
In scope of patent protection of the invention.
Claims (10)
1. a kind of method that Holoens combination HTCVive realizes the interaction of MR mixed reality characterized by comprising
After server obtains the handle data that HTC handle includes event information and spatial information, it is wrapped into forming data packet hair
It send to Holoens client；
Holoens client parses the data packet, obtains the handle data, and apply it in MR scene.
2. the method that a kind of Holoens combination HTCVive as described in claim 1 realizes the interaction of MR mixed reality, feature
It is, further includes:
Server monitors the connection request that Holoens client is sent by main thread；
After server receives connection request, a sub thread is created；
Server sends connection data to the corresponding Holoens client of the connection request by the sub thread；
The Holoens client for receiving the connection data is established and the TCP connection of server.
3. the method that a kind of Holoens combination HTCVive as claimed in claim 2 realizes the interaction of MR mixed reality, feature
It is, server sends the data packet to Holoens client by the sub thread.
4. the method that a kind of Holoens combination HTCVive as described in claim 1 realizes the interaction of MR mixed reality, feature
Be, it is described be wrapped into forming data packet be sent to client, specifically:
The handle data are packed into including size of data, data type and data content by server according to preset agreement
Data packet after, send it to Holoens client；The data type is Data；The data content is handle data.
5. the method that a kind of Holoens combination HTCVive as claimed in claim 4 realizes the interaction of MR mixed reality, feature
It is, the Holoens client parses the data packet, the handle data are obtained, specifically:
Holoens client obtains size of data and data type from the data packet；
Data content is read from data packet according to the size of data；
According to the data type for being Data, the data content is parsed, obtains handle data.
6. the method that a kind of Holoens combination HTCVive as described in claim 1 realizes the interaction of MR mixed reality, feature
It is, it is described to obtain the handle data, and apply it in MR scene, specifically:
Obtain the spatial information in handle data；
According to the spatial information, change position of the corresponding virtual object of the HTC handle in MR scene；Meanwhile
Obtain the event information in handle data, tool of the event information successively to trigger in one frame of tabular form journal
Body event information；
The corresponding each specific event information of event information described in sequence trigger switch in MR scene.
7. the method that a kind of Holoens combination HTCVive as described in claim 1 realizes the interaction of MR mixed reality, feature
It is, if the space coordinates of server and the space coordinates of Holoens client are inconsistent, by the sky in handle data
Between before information operating to MR scene, further includes:
The spatial information of the space coordinates of corresponding server is converted to the sky of the space coordinates of corresponding Holoens client
Between information.
8. the method that a kind of Holoens combination HTCVive as claimed in claim 7 realizes the interaction of MR mixed reality, feature
It is, the spatial information of the space coordinates by corresponding server is converted to the space coordinate of corresponding Holoens client
The spatial information of system, specifically:
Initial spatial information of the HTC handle in the space coordinates of server is obtained in Holoens client, and by its assignment
The corresponding virtual object of HTC handle described in space coordinates to Holoens client；
Obtain the first forward direction vector V1 and first position P1 that the virtual object corresponds to the space coordinates of Holoens client；
It obtains through control HTC handle, the virtual object is moved in the space coordinates for being preset in Holoens client
After one marker is overlapped, the marker correspond to the space coordinates of Holoens client the second forward direction vector V2 and
Second position P2；
According to the first forward direction vector V1 and position P1 and the second forward direction vector V2 and position P2, the first forward direction vector V1 is calculated
With the vertical vector V3 of the second forward direction vector V2；
According to vertical vector V3, the angle theta of the first forward direction vector V1 and the second forward direction vector V2 is calculated；
According to vertical vector V3 and angle theta, four element Q are calculated；
Spin matrix R is converted by the four elements Q；
The position P1 is converted into the third place P3 according to spin matrix R；
According to the third place P3 and second position P2, positional shift O is calculated；
According to the positional shift O and the third place P3, the space that HTC handle corresponds to Holoens client is calculated
Initial spatial information after correction in coordinate system；
According to the spin matrix R and positional shift O, acquired spatial information is corrected.
9. the method that a kind of Holoens combination HTCVive as described in claim 1 realizes the interaction of MR mixed reality, feature
It is, the handle data are JSON format.
10. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that described program is being located
When managing device execution, a kind of Holoens combination HTCVive realization MR described in the claims 1-9 any one can be realized
The step of method of mixed reality interaction is included.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810954228.2A CN109085925B (en) | 2018-08-21 | 2018-08-21 | Method and storage medium for realizing MR mixed reality interaction |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810954228.2A CN109085925B (en) | 2018-08-21 | 2018-08-21 | Method and storage medium for realizing MR mixed reality interaction |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109085925A true CN109085925A (en) | 2018-12-25 |
| CN109085925B CN109085925B (en) | 2021-07-27 |
Family
ID=64794072
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810954228.2A Active CN109085925B (en) | 2018-08-21 | 2018-08-21 | Method and storage medium for realizing MR mixed reality interaction |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109085925B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110164233A (en) * | 2019-06-06 | 2019-08-23 | 西安勺子智能科技有限公司 | A kind of Portable individual weapon simulated training system based on MR |
| CN110930516A (en) * | 2018-12-29 | 2020-03-27 | 湖南汉坤建筑安保器材有限公司 | Safety experience control system based on VR all-in-one |
| CN111897437A (en) * | 2020-08-19 | 2020-11-06 | 腾讯科技（深圳）有限公司 | Cross-terminal interaction method and device, electronic equipment and storage medium |
| CN113076088A (en) * | 2021-04-08 | 2021-07-06 | 南京爱奇艺智能科技有限公司 | System for 3DOF handle SDK application development |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101272402A (en) * | 2008-05-14 | 2008-09-24 | 深圳市同洲电子股份有限公司 | Method, system and device for data transmission |
| CN105074783A (en) * | 2013-03-06 | 2015-11-18 | 株式会社巨晶片 | Augmented reality provision system, recording medium, and augmented reality provision method |
| CN105759963A (en) * | 2016-02-15 | 2016-07-13 | 众景视界（北京）科技有限公司 | Method for positioning motion trail of human hand in virtual space based on relative position relation |
| CN206400472U (en) * | 2016-08-24 | 2017-08-11 | 王忠民 | A kind of virtual reality device and its alignment system |
| CN107145223A (en) * | 2017-04-06 | 2017-09-08 | 北京讯腾智慧科技股份有限公司 | Multi-point interaction control system and method based on Unity d engines and the VR helmets |
| US20180039341A1 (en) * | 2016-08-03 | 2018-02-08 | Google Inc. | Methods and systems for determining positional data for three-dimensional interactions inside virtual reality environments |
-
2018
- 2018-08-21 CN CN201810954228.2A patent/CN109085925B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101272402A (en) * | 2008-05-14 | 2008-09-24 | 深圳市同洲电子股份有限公司 | Method, system and device for data transmission |
| CN105074783A (en) * | 2013-03-06 | 2015-11-18 | 株式会社巨晶片 | Augmented reality provision system, recording medium, and augmented reality provision method |
| CN105759963A (en) * | 2016-02-15 | 2016-07-13 | 众景视界（北京）科技有限公司 | Method for positioning motion trail of human hand in virtual space based on relative position relation |
| US20180039341A1 (en) * | 2016-08-03 | 2018-02-08 | Google Inc. | Methods and systems for determining positional data for three-dimensional interactions inside virtual reality environments |
| CN206400472U (en) * | 2016-08-24 | 2017-08-11 | 王忠民 | A kind of virtual reality device and its alignment system |
| CN107145223A (en) * | 2017-04-06 | 2017-09-08 | 北京讯腾智慧科技股份有限公司 | Multi-point interaction control system and method based on Unity d engines and the VR helmets |
Non-Patent Citations (1)
| Title |
|---|
| 搜狐视频: "全息演示微软Windows之间hololens和HTCVive_VR资源网", 《搜狐视频》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110930516A (en) * | 2018-12-29 | 2020-03-27 | 湖南汉坤建筑安保器材有限公司 | Safety experience control system based on VR all-in-one |
| CN110164233A (en) * | 2019-06-06 | 2019-08-23 | 西安勺子智能科技有限公司 | A kind of Portable individual weapon simulated training system based on MR |
| CN111897437A (en) * | 2020-08-19 | 2020-11-06 | 腾讯科技（深圳）有限公司 | Cross-terminal interaction method and device, electronic equipment and storage medium |
| CN113076088A (en) * | 2021-04-08 | 2021-07-06 | 南京爱奇艺智能科技有限公司 | System for 3DOF handle SDK application development |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109085925B (en) | 2021-07-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109085925A (en) | It is a kind of realize MR mixed reality interaction method, storage medium | |
| US20210201554A1 (en) | Method and apparatus for sport-specific training with captured body motions | |
| US11771987B2 (en) | Limbo spectating in a video game | |
| US6767287B1 (en) | Computer system and method for implementing a virtual reality environment for a multi-player game | |
| US7904577B2 (en) | Data transmission protocol and visual display for a networked computer system | |
| US9345967B2 (en) | Method, device, and system for interacting with a virtual character in smart terminal | |
| CN104778121B (en) | The test method of games, apparatus and system | |
| KR20170109496A (en) | Synchronized video with in game telemetry | |
| CN107479699A (en) | Virtual reality exchange method, apparatus and system | |
| CN104136086B (en) | Information processing system, method and equipment | |
| CN113350779A (en) | Game virtual character action control method and device, storage medium and electronic equipment | |
| CN112131112A (en) | Operation information acquisition method and device, storage medium and electronic equipment | |
| CN108654088A (en) | program debugging method and device, storage medium and electronic device | |
| CN110152292B (en) | Display control method and device for jumping characters in game, storage medium and electronic equipment | |
| CN113961064B (en) | Device interaction control method, device, equipment and storage medium based on workstation | |
| McNamara et al. | Investigating low-cost virtual reality technologies in the context of an immersive maintenance training application | |
| CN109873876A (en) | A kind of method of the interaction and computational load equilibrium assignment of distributed virtual environment | |
| CN115865916A (en) | Multi-user cooperation method and system for mobile virtual reality based on edge computing | |
| CN116236785A (en) | Method, device, equipment and medium for processing information in game | |
| US20230129718A1 (en) | Biometric feedback captured during viewing of displayed content | |
| US11966278B2 (en) | System and method for logging visible errors in a videogame | |
| US20230125930A1 (en) | Techniques for combining geo-dependent and geo-independent experiences in a virtual environment | |
| Fumić et al. | Bridging the Physical and Virtual Worlds: A Hand Tracking Gesture Recognition System for XR Applications | |
| CN117101131A (en) | Interaction control method and device in game and electronic equipment | |
| Bondarenko et al. | Latency Visualization Emulation Scheme for Interactive Cloud XR Gaming |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |