CN115834867A - Local multi-device rapid spatial anchor point synchronization method and system for mixed reality - Google Patents
Local multi-device rapid spatial anchor point synchronization method and system for mixed reality Download PDFInfo
- Publication number
- CN115834867A CN115834867A CN202211397487.2A CN202211397487A CN115834867A CN 115834867 A CN115834867 A CN 115834867A CN 202211397487 A CN202211397487 A CN 202211397487A CN 115834867 A CN115834867 A CN 115834867A
- Authority
- CN
- China
- Prior art keywords
- host
- local
- coordinate system
- timestamp
- guest
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D30/00—Reducing energy consumption in communication networks
- Y02D30/70—Reducing energy consumption in communication networks in wireless communication networks
Abstract
The invention discloses a local multi-device fast spatial anchor point synchronization method and a system for mixed reality, which comprises the following steps: any one device located in a local space is used as a host to create a local network, and other devices are used as guests to join the local network created by the host; the guest machine and the host machine exchange respective local time stamps first, and then time offset between the local time stamp of the guest machine and the local time stamp of the host machine is calculated, so that time synchronization between the guest machine and the host machine is realized; on the basis of time synchronization, one guest machine in the local network performs space coordinate system synchronization with the host machine through an augmented reality picture tracking technology, and other guest machines can perform space coordinate system synchronization with the host machine or the synchronized guest machines, so that space coordinate system synchronization among all devices in the local network is realized. The synchronization method provided by the invention can use multi-user mixed reality application under the condition of no external operator network, and realizes quick and quick space coordinate system synchronization.
Description
Technical Field
The invention belongs to the technical field of mixed reality, and particularly relates to a local multi-device fast spatial anchor point synchronization method and system for mixed reality.
Background
Mixed Reality (MR), augmented Reality (AR), or other Display device is a technology that renders virtual content that can be associated with the real world by using an HMD (Head Mounted Display), a smartphone, or other Display device, so that a user can simultaneously see and interact with the real world and the virtual content superimposed thereon. At present, most of the applications in the mixed reality technology are single experience, namely, virtual contents rendered by the device can only be seen by a user wearing the device. Even more than two users in the same real space see different virtual content, so that more than two users cannot interact with the same virtual content at the same time. Moreover, because the content rendered by the head display device can only be seen by the wearer, the experience of mixed reality can only be observed at the first visual angle, and the content played by the wearer can not be seen at the third visual angle by a third party without the head display, so that the propagation of the experience of mixed reality is difficult.
In order to realize virtual content interaction among different devices, different platforms and devices have different space scanning operation methods, if the mixed reality devices cannot synchronize the space, the mixed reality devices are difficult to work cooperatively, and the operation of one mixed reality device is difficult to reflect in other mixed reality devices in real time. There is therefore a need for a simple and convenient co-operating algorithm for synchronizing the spatial coordinates of different devices and platforms, when synchronization of the spatial coordinate systems of the devices is involved.
In order to solve the problem of lack of online interaction among multiple devices, the chinese patent invention with the publication number "CN107945116B" discloses a coordinate axis synchronization method for realizing AR indoor positioning among multiple devices, the technical scheme includes the following steps: establishing a local coordinate system of each device on each device participating in online by using the position of the device and the angle of a gyroscope; selecting any one device as a master reference device and other devices as slave devices, taking the coordinate system of the master reference device as a master reference coordinate system, and correcting the coordinate systems of the other slave devices; the slave device obtains the position information of the master reference device and converts it into slave device coordinates based on the master reference coordinate system. The multi-equipment coordinate axis synchronization method provided by the technical scheme can form a uniform coordinate system among all equipment, so that more convenient and uniform online interaction is carried out, and the virtual and real combination can be realized in a large range. However, in a mixed reality use scenario with a high real-time requirement, such as an augmented reality multi-player battle game, due to delay of an ISP (Internet Service Provider) network or fast transformation of a virtual space, accuracy of coordinate synchronization is insufficient, and user experience is poor.
In order to improve the accuracy of the synchronization of the spatial coordinate systems of the mixed reality equipment, the Chinese patent with the publication number of 'CN 109766001B' discloses a method for unifying the coordinate systems of different MR equipment, wherein the method for unifying the coordinate systems of the different MR equipment comprises the following steps: taking any point in a preset space as a reference point, and acquiring the relative position and rotation information of each calibration plate in the preset space relative to the reference point; binding each MR device with a corresponding calibration plate; establishing a global coordinate system with the reference point as an origin; unifying the relative position and rotation information corresponding to the original coordinate system of each MR device into a global coordinate system. According to the technical scheme, the calibration plate and the reference points are arranged in the preset space, the MR equipment is bound with the calibration plate, the position of each MR equipment in a global coordinate system established by taking the reference calibration plate as an origin is determined through the relative positions of the calibration plate and the reference points and the rotation information, the MR equipment corresponds to the same global coordinate system, and interaction of different MR equipment in the same scene is realized. The synchronization process of the technical scheme is relatively complicated, so that the synchronization speed of the space coordinate system is relatively low.
Disclosure of Invention
A local multi-device fast spatial anchor point synchronization method for mixed reality comprises the following steps:
s1: the method comprises the steps that local network connection is established among a plurality of devices located in a local space so as to realize data transmission among the devices, any one of the devices serves as a host to establish a local network, and other devices serve as guests to join the local network established by the host.
S2: and carrying out time synchronization on the host and the guest machine in the local network, wherein the guest machine and the host machine exchange respective local time stamps to each other firstly, then calculating the time offset between the local time stamp of the guest machine and the local time stamp of the host machine, and adjusting and realizing the time synchronization between the guest machine and the host machine according to the time offset.
S3: the method comprises the steps of synchronizing a host in a local network with a passenger plane by a spatial anchor point, using a specific picture displayed on a display system of the host as an anchor point, scanning the specific picture by an augmented reality picture tracking technology, obtaining a second position and rotation of the picture anchor point under a spatial coordinate system of the passenger plane, and requesting a first position and rotation of the anchor point under the spatial coordinate system of the host under the same timestamp from the host, obtaining two position and rotation data by the passenger plane as a pair of poses, scanning the anchor point for multiple times by the passenger plane to obtain multiple pairs of poses, calculating the multiple pairs of poses by a least square method to obtain a third position and rotation of an origin of a spatial coordinate of the host under the spatial coordinate system of the passenger plane, resetting the spatial coordinate system of the passenger plane to the third position and rotation to enable the position and rotation of the origin of the spatial coordinate systems of the host and the passenger plane to be the same, completing synchronization of the spatial coordinate systems among devices, and enabling other passenger planes to be synchronized with the spatial coordinate system in the local network or synchronized with the spatial coordinate system in the local network.
Preferably, the step S1 specifically includes:
s1-1: the host establishes a local network and broadcasts on the local network in a wireless communication mode.
S1-2: and the passenger plane searches the broadcast of the host through the wireless communication mode.
S1-3: and the guest machine discovers the broadcast and sends a connection request to the host machine by using the wireless communication mode.
S1-4: and the host receives the connection request, agrees to the connection request and establishes network connection between the guest machine and the host.
Preferably, the step S2 specifically includes:
s2-1: a guest sends the guest's local timestamp one to the host.
S2-2: and the host sends the time point of receiving the first local timestamp as a second local timestamp to the guest.
S2-3: and the first guest machine takes the time point of receiving the local timestamp I as a local timestamp III, and calculates and stores the timestamp offset of the first guest machine and the host machine according to the local timestamp I, the local timestamp II and the local timestamp III.
S2-4: and repeating the step S2-1 to the step S2-3 until the number of the timestamp offsets reaches a set value to obtain a plurality of timestamp offsets.
S2-5: and calculating the standard deviation of the plurality of timestamp offsets, if the standard deviation is not less than a preset value, abandoning the earliest calculated timestamp offset, repeating the steps from S2-1 to S2-3 until the standard deviation is less than the preset value, and calculating the average value of the plurality of timestamp offsets to serve as a final timestamp offset result.
S2-6: the other guests calculate the final timestamp offset result between each of the other guests and the host according to steps S2-1 through S2-5.
Preferably, the formula of calculating the timestamp offset in step S2-3 is as follows:
wherein, T0 is a local timestamp one, T0' is a local timestamp two, and T1 is a local timestamp three.
Preferably, said one guest creates a queue holding said plurality of timestamp offsets.
Preferably, the step S3 specifically includes:
s3-1: the host machine utilizes an accelerometer, a gyroscope and an imaging system to calculate the position and rotation of the specific picture under a local coordinate system of the host machine in real time according to the invariance of the offset vectors of the specific picture and the imaging system of the host machine.
S3-2: and a passenger plane in the local network scans the specific picture by using an imaging system through an augmented reality picture tracking technology, the augmented reality picture tracking technology identifies a second position and rotation of the specific picture in the scene, sends the second position and rotation to the host, and receives the first position and rotation sent by the host, wherein the first position and rotation and the second position and rotation have the same timestamp.
S3-3: the method comprises the steps that the first position and rotation and the second position and rotation under different timestamps are obtained by the passenger plane, the optimal displacement vector and the optimal rotation are obtained through least square calculation, the position and the rotation of the origin of the local space coordinate system of the host relative to the origin of the local space coordinate system of the passenger plane are obtained, the origin of the local space coordinate system of the passenger plane is reset to the origin of the local space coordinate system of the host, and the synchronization of the space coordinate systems of the host and the passenger plane is completed. And for other guests in the local network, resetting the origin of the local spatial coordinate system of the guest to the origin of the local spatial coordinate system of the host or the origin of the local spatial coordinate system of any synchronized guest, and synchronizing the spatial coordinate systems of the devices in the local network.
S3-4: rendering a virtual object with the size consistent with that of the host equipment in the mixed reality virtual space of the guest machine, synchronizing the position and rotation of the virtual object with the host machine in real time, and if the virtual object is not completely overlapped with the host machine in reality, repeating the steps S3-1 to S3-3 by the guest machine until the virtual object is completely overlapped with the host machine in reality.
Preferably, the expression of the optimal displacement vector t is:
wherein the content of the first and second substances,
representing the weighted center values of all anchor points under the host spatial coordinate system,
representing the center value of the weights of all anchor points under the one guest space coordinate system,
is a rotation matrix rotated by an angle phi along the spatial Y-axis.
Preferably, the calculation method of the optimal rotation phi is as follows:
wherein, P' a，i，x 、P′ a，i，z Is the difference value P 'of the next anchor point of the host space coordinate system and the weight center thereof' b，i，x 、P′ b，i，z Is the difference, R ', of a next anchor point in the spatial coordinate system of the passenger aircraft and the weight center thereof' i，1，x 、R′ i，1，z 、R′ i，3，x 、R′ i，3，z Is the rotational difference between the spatial coordinate systems of the host and a guest, c is an adjustable constant greater than 0, and by adjusting the value of c, the formula can be biased toward smaller rotational errors or smaller displacement errors.
A local multi-device fast spatial anchor synchronization system for mixed reality, the system comprising a memory having instructions stored thereon that enable the processor to perform: the local multi-device fast spatial anchor point synchronization method for mixed reality is described.
A machine-readable storage medium having instructions stored thereon for causing a machine to perform: the local multi-device fast spatial anchor point synchronization method for mixed reality is described.
Compared with the prior art, the invention has the following technical effects:
1. the synchronization method provided by the invention firstly establishes the local network in the local space, which is beneficial to overcoming the dependence on the external network and reducing the delay of the network so as to improve the precision and speed of space synchronization; then, time synchronization is carried out on all the devices in the local network, and the precision of space synchronization is further improved; on the basis of establishing local network and time synchronization, the spatial anchor points are determined, the optimal positions and the optimal rotation are calculated by using a least square method according to the positions and the rotations of the anchor points in respective spatial coordinate systems, the spatial synchronization among the devices is completed, and the requirements of mixed reality application on the real-time performance and the accuracy of the spatial coordinate synchronization are met.
2. The synchronization method provided by the invention selects any equipment in the local space as the host, establishes the local network through a local wireless communication mode, and adds other equipment into the local network as the passenger plane, thereby overcoming the dependence on the external network environment, and leading a plurality of pieces of equipment in the local space to be capable of using multi-user mixed reality application under the condition of poor external network signals or no external network signals.
3. The synchronization method provided by the invention utilizes the specific picture displayed by the equipment display system as the anchor point. The two-dimensional code picture is a good choice because the two-dimensional code picture is complex enough and has enough characteristic points, which is beneficial for the scanning equipment to quickly identify, and the characteristic of the two-dimensional code picture with information can send extra information to the scanning equipment by the scanned equipment under the condition of not transmitting through the network.
4. The synchronization method provided by the invention can be simultaneously applied to handheld mixed reality equipment (mobile phones and tablet computers) and mixed reality head displays (HMDs) driven by the mobile phones, and any number of the two kinds of equipment can establish a network and synchronize spatial anchor points through the method, so that multi-user mixed reality experience is carried out.
5. The synchronization method provided by the invention uses an augmented reality picture tracking technology, and carries out rapid space coordinate synchronization between two devices by scanning a specific picture which is displayed on a host screen and is used as an anchor point by a passenger plane, or carries out rapid space coordinate synchronization between two devices by scanning a specific picture which is displayed on a synchronized passenger plane screen and is used as an anchor point by the passenger plane, thereby realizing the synchronization of a space coordinate system under the condition of not manually inputting information in advance and having high synchronization speed.
6. The synchronization method provided by the invention supports the connection and space coordinate synchronization of any number of mixed reality devices in the local space, and the space coordinate synchronization of the subsequently added devices can be carried out in a chain picture tracking mode under the condition that the number of the devices exceeds two. The chained picture tracking here means that, assuming that the host a and the guest B have already established a network connection and completed synchronization, the newly added guest C can perform spatial coordinate synchronization not only by scanning a specific picture displayed by the host a but also by scanning a picture displayed by the guest B. Similarly, when the passenger plane D is added, the passenger plane D can complete the spatial coordinate synchronization with all the devices by scanning a specific picture on any one of the devices a, B, and C, so as to provide convenient synchronization between spatial coordinate systems.
7. In the space synchronization stage, the synchronization method provided by the invention uses the least square method to calculate and obtain the optimal rotation and optimal displacement value between a pair of space coordinate systems, can simply and conveniently obtain unknown data, enables the square sum of the errors between the obtained data and actual data to be minimum, and ensures the accuracy of space synchronization between different space coordinate systems.
Drawings
FIG. 1 is a flow chart of a local multi-device fast spatial anchor point synchronization method for mixed reality according to the present invention;
FIG. 2 is a flow chart of the networking between local devices of the method for synchronizing local multi-device fast spatial anchor points for mixed reality according to the present invention;
FIG. 3 is a flow chart of time synchronization between local devices for a mixed reality local multi-device fast spatial anchor synchronization method according to the present invention;
fig. 4 is a flow chart of the spatial synchronization between local devices according to the local multi-device fast spatial anchor synchronization method for mixed reality of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be described in detail and completely with reference to the accompanying drawings.
Please refer to fig. 1, which is a flowchart of the present embodiment.
The synchronization method described in this embodiment includes three steps: 1. establishing a local network, wherein one device in the local space is used as a host to establish the local network, and other devices are used as guests to join the local network; 2. synchronizing local time stamps of multiple devices in the local network, and performing time synchronization between the host and the guest by calculating the offset of the local time stamps; 3. the host and the passenger plane or the passenger plane and the passenger plane share the position information and the rotation information of the same picture anchor point through a picture tracking technology, and the spatial coordinate system is synchronized by combining time information.
Spatial Anchors (Spatial Anchors) represent physical points that exist in a virtual world. For mixed reality applications, the hologram may be attached to a spatial anchor point. The spatial anchor can be stored and persisted and later queried by the device that created it or any other supported device. This enables backup and sharing of anchors. For example, at the device end, two people may view a chessboard at the same location (desktop) in the real world. Wherever in physical space the board is moved, it will be fixed to a point.
Fig. 2 is a flowchart of networking between local devices according to this embodiment.
In the case where there are more than one (two or more) devices in the local area, any device may create a local network as a host, and other devices may join the local network created by the host as guests. First, any one of the devices a in the local space creates a local network as a host, and continuously broadcasts its presence to all devices in the vicinity. Then, another device B in the local space starts to attempt to join the local network, the device B starts to search whether a host having established the local network is broadcasting itself in the nearby space, and the device B sends a connection request to the host a after searching for the existence of the host a. Finally, upon receiving the connection request from device B, host a approves the connection request, thereby joining device B as guest B to the local network. Any number of subsequent devices may join the local network created by host a as a guest by repeating the above process.
The local network of the embodiment is based on Bluetooth (Bluetooth) or wireless local area network (Wi-Fi), can perform direct point-to-point connection under the condition that no mobile provider network (4G or 5G) and router provide the Internet, and can avoid delayed interference of the Internet. When the equipment is connected, the equipment can be automatically connected without any operation of a user as long as the related application software is opened on the mobile phone or the tablet computer. Therefore, under the condition that external network signals are poor or no, a plurality of devices in the local space can use multi-user mixed reality application.
Please refer to fig. 3, which is a flowchart illustrating time synchronization between local devices according to the present embodiment.
After the local network is established and all devices join the local network, the devices in the local network will perform time synchronization, i.e. calculate the local timestamp offset between the two devices. Each device has its own local timestamp that is used to time any local operation occurs. There are many ways to implement a timestamp in a device, such as calculating the time in seconds that elapses between the time the device is powered on and the time an operation occurs. No matter which implementation of the timestamp is used, the timestamp keeps the correct consistency on the local device, and the unit of the timestamp is the same among multiple devices, so that the implementation of the technology is not influenced. The purpose of calculating the timestamp offset between the two devices is to align the position of the two devices within the same time with the rotation in the third section. All subsequent devices accessing the local network can perform timestamp synchronization by repeating the same operation.
The specific process of time synchronization of the devices in the local network is as follows: at a certain moment, the local timestamp of guest B is T0, guest B sends this timestamp T0 as a parameter to host a and requests the timestamp of host a. When host a receives the timestamp request of guest B, its local timestamp is T1, and host a sends back received timestamp T0 and its own timestamp T1 to guest B at the same time. When receiving the timestamp sent back by the host, guest B locally has three timestamps, which are T0 when guest B previously sent a timestamp request to the host, T1 when host a received a timestamp request, and T0' when guest B received a timestamp sent back by the host. The time offset between guest B and host a can be calculated by the following equation:
at this time, the guest B has a single time offset data, and it is difficult to obtain an accurate value by calculating the time stamp offset only once in consideration of the influence of the network delay. Thus, in this embodiment, to ensure the accuracy and stability of the timestamp offsets, the above steps are repeated, and the timestamp offsets obtained each time are stored in a queue Q created by guest B. When the number of timestamp offsets stored in the queue Q reaches a preset number value, the standard deviation of the entire queue Q is calculated. If the standard variance of the queue is too large (larger than the preset standard variance value), the earliest calculated timestamp offset data in the queue is abandoned (the dequeuing operation of the queue), and then the processes of acquiring the timestamps T0, T1 and T0' and calculating the time offset are continuously repeated once, so that a new time offset data is acquired and queued. If the queue variance is less than the preset variance value, this indicates that the timestamp offset has already stabilized. The average of the entire queue Q can be calculated as the timestamp offset between the final guest B and host a.
Please refer to fig. 4, which is a flowchart illustrating spatial synchronization between local devices according to the present embodiment.
After the Guest B establishes local network connection with the host A and synchronizes the local timestamps, the host A displays a specific picture on the display system as an anchor point, starts recording the picture local pose (pose including position and rotation) within a period of time and attaches timestamps, and stores the pose with the timestamp in the queue Q1. The guest B uses an augmented reality picture tracking technique to scan and attempt to identify a particular picture displayed on the host a display system, each time the guest B finds a picture anchor point of the host a through an augmented reality picture tracking algorithm, the guest B immediately sends the local pose with a timestamp of this anchor point to the host a.
The specific picture as the anchor point is displayed on the display system of the host A, the host A is an augmented reality device, the position and the rotation of the imaging system under the local coordinate system of the host A can be calculated in real time through the accelerometer, the gyroscope and the imaging system of the host A, and the difference between the central position of the picture and the position of the imaging system is fixed, so the host A can calculate the position and the rotation of the anchor point of the picture in real time. The guest B continuously searches for a specific picture displayed on the host from the pictures taken by its imaging system through an augmented reality picture tracking technique (this picture is preset in advance, so the guest B knows what picture is displayed on the host a). The augmented reality image tracking technique can accurately identify the position and rotation of a specific image in a scene, so that the passenger plane B can also know the position and rotation of the image anchor point.
The above-described augmented reality picture tracking technology for scanning a specific picture may use the ARKit technology of apple inc, or other imaging technology supporting the MarkerTracking method.
The specific pictures are preset in advance for a program, and preferably, pictures which are complex enough and have enough characteristic points are used, and a two-dimensional code picture is a very good choice. In addition to the advantage of easy identification, the use of the two-dimensional code image may also provide additional information from host a to guest B without going through the network. Besides the two-dimensional code picture, any picture with enough characteristic points can be used as the picture anchor point.
While the guest B scans a specific picture displayed on the display system of the host a, the two devices share their spatial positions, and the positions and rotations of the two devices in the local coordinate system are transmitted to each other through the local network. Each time guest B successfully scans a particular picture displayed on host a display system as an anchor point, both devices know the location and rotation of the anchor point in their respective device's local spatial coordinate system. Through the synchronization of the local network and the timestamp, the passenger plane B can obtain a plurality of pairs of positions and rotations of the anchor point under the space coordinate systems of the two devices at the same time, the position and the rotation of the origin of the local space coordinate system of the host A relative to the origin of the local space coordinate system of the passenger plane B are calculated through a least square method, at the moment, the origin of the local coordinate system of the passenger plane B is reset to the origin of the host A, and the origins and the directions of the space coordinate systems of the two devices are the same, so that the synchronization of the space coordinate systems of the two devices is realized.
The least squares method of this example targets as follows:
R a,i is the rotation of the next anchor point in the space coordinate system of the host A, P a,i Is the position of the next anchor point in the space coordinate system of the host A, R b，i Is the rotation of the next anchor point in the spatial coordinate system of passenger plane B, P b，i Is the position of the next anchor point in the spatial coordinate system of passenger plane BThe device is placed in a water tank,
is the pose of a queue of picture anchor points under the local space coordinate system of the host A,
is a queue of poses of picture anchor points under the local space coordinate system of the passenger plane B, and the queue
And queue
The anchor points in (1) correspond one-to-one in real world time.
Because two mixed reality devices can find the same Y-axis orientation by using a built-in gyroscope according to gravity, it is necessary to find a one-dimensional rotation along the Y-axis direction of the coordinate system:
and a three-dimensional vector t that shifts the origin of the coordinate system for passenger aircraft B to the origin of the coordinate system for passenger aircraft A. By solving for one rotation angle phi and the vector t, the following equation takes the minimum value (minimum error):
f in the above formula can be expressed as:
where w represents the weight of each pair of anchor points, w i Is greater than 0; c represents the weight of penalty, c > 0; c is a constant, in this embodiment, the position and rotation data with error are processed simultaneously, and the value of c is adjustedTo bias the formula more toward smaller rotational errors or smaller displacement errors.
Then, the optimal displacement vector is calculated, assuming
It is fixed and the best displacement vector can be found by deriving t, as follows:
the expression for the best displacement vector t can be derived:
wherein
And
are respectively
Wherein the content of the first and second substances,
representing the weighted center values of all anchor points in the host a coordinate system,
representing the weight center value of all anchor points under the coordinate system of the guest B. Under the condition of no change of rotation phi, the weight centers of A and B can be calculated by calculatingAn optimal displacement vector t is calculated.
Finally, the rotation is calculated:
P′ a，i is the difference value P 'of the next anchor point of the host A space coordinate system and the weight center thereof' b，i Is the difference between the next anchor point in the spatial coordinate system of passenger aircraft B and its weight center.
Substituting the optimal displacement vector t into the target formula F can be derived from the above expression which has been extrapolated to t in the case where the rotation phi has been determined:
since the trace (trace) of the matrix AB is the same as that of the matrix BA, i.e., tr (AB) = tr (BA),
target rotation may be calculated knowing a pair of anchor rotations
Wherein R' i For the target rotation (the difference in rotation between the two coordinate systems),
as an inverse matrix of the rotation matrix of the anchor point in the coordinate system of the passenger aircraft, R a，i For rotation of the anchor point under the host coordinate system) and then obtaining the optimal rotation by means of derivation phi:
the optimal rotation phi obtained by the above calculation is:
any number of devices subsequently joining the network can be synchronized with the spatial coordinates of host a in this way. After the synchronization is successful, a virtual object with the size consistent with that of the equipment of the host computer A is rendered in the mixed reality virtual space of the Guest B, and the position and the rotation of the virtual object are synchronized with that of the host computer A in real time. If the spatial synchronization accuracy of the two devices is high enough, the virtual object and the real host A should be completely overlapped in the view angle of the passenger plane B; if the error between the two is too large, it indicates that the synchronization is not successful, and the guest B will repeat the spatial coordinate system synchronization process until the host A successfully realizes accurate spatial coordinate system synchronization.
In another embodiment of the invention, a device subsequently joining the local network completes the synchronization of the spatial coordinate system by scanning a particular picture displayed on the guest display system of the synchronized spatial coordinate system in the local network. Guest B attaches to the local network created by host A and performs spatial coordinate system synchronization with host A by scanning for specific pictures on the host A display system. And a new subsequent guest machine C is added into the network, except for scanning the picture on the host machine A, the guest machine C can also carry out space coordinate system synchronization with the guest machine B in a mode of scanning a specific picture on a display system of the guest machine B, and as the guest machine B and the host machine A complete the space coordinate system synchronization, the three devices A, B and C complete the space synchronization successfully at the moment. Similarly, when a new guest D joins, guest D can synchronize the spatial coordinate system with all devices in the network by spatial synchronization with any of devices a, B, C by scanning a particular picture. The chain-type spatial coordinate synchronization is realized through the mode, so that any device in the local network can perform spatial coordinate system synchronization with other devices nearby.
A local multi-device fast spatial anchor synchronization system for mixed reality, the system comprising a memory and a processor, the memory having stored thereon instructions that enable the processor to perform: the invention relates to a local multi-device fast spatial anchor point synchronization method for mixed reality.
A machine-readable storage medium having instructions stored thereon for causing a machine to perform: the invention relates to a local multi-device fast spatial anchor point synchronization method for mixed reality.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various changes and modifications can be made without departing from the inventive concept of the present invention, and these changes and modifications are all within the scope of the present invention.
Claims (10)
1. A local multi-device fast spatial anchor point synchronization method for mixed reality is characterized by comprising the following steps:
s1: establishing local network connection by a plurality of devices positioned in a local space to realize data transmission among the devices, establishing a local network by taking any one of the devices as a host, and adding other devices into the local network established by the host as guests;
s2: time synchronization is carried out on a host machine and a guest machine in the local network, the guest machine and the host machine exchange respective local time stamps to each other, then time offset between the local time stamp of the guest machine and the local time stamp of the host machine is calculated, and the time synchronization between the guest machine and the host machine is adjusted and realized according to the time offset;
s3: the method comprises the steps of synchronizing a host in a local network with a passenger plane by a spatial anchor point, using a specific picture displayed on a display system of the host as an anchor point, scanning the specific picture by an augmented reality picture tracking technology, obtaining a second position and rotation of the picture anchor point under a spatial coordinate system of the passenger plane, and requesting a first position and rotation of the anchor point under the spatial coordinate system of the host under the same timestamp from the host, obtaining two position and rotation data by the passenger plane as a pair of poses, scanning the anchor point for multiple times by the passenger plane to obtain multiple pairs of poses, calculating the multiple pairs of poses by a least square method to obtain a third position and rotation of an origin of a spatial coordinate of the host under the spatial coordinate system of the passenger plane, resetting the spatial coordinate system of the passenger plane to the third position and rotation to enable the position and rotation of the origin of the spatial coordinate systems of the host and the passenger plane to be the same, completing synchronization of the spatial coordinate systems among devices, and enabling other passenger planes to be synchronized with the spatial coordinate system in the local network or synchronized with the spatial coordinate system in the local network.
2. The local multi-device fast spatial anchor synchronization method for mixed reality according to claim 1, wherein the step S1 specifically comprises:
s1-1: the host establishes a local network and broadcasts on the local network in a wireless communication mode;
s1-2: the passenger plane searches the broadcast of the host through the wireless communication mode;
s1-3: the guest machine finds the broadcast and sends a connection request to the host machine by using the wireless communication mode;
s1-4: and the host receives the connection request, agrees to the connection request and establishes network connection between the guest machine and the host.
3. The local multi-device fast spatial anchor point synchronization method for mixed reality according to claim 1, wherein the step S2 specifically comprises:
s2-1: a guest sends a local timestamp one of the guest to the host;
s2-2: the host sends the time point of receiving the local timestamp I as a local timestamp II to the guest machine;
s2-3: the first passenger plane takes the time point of receiving the local timestamp two as a local timestamp three, and time stamp offset between the first passenger plane and the host is calculated and stored according to the local timestamp one, the local timestamp two and the local timestamp three;
s2-4: repeating the step S2-1 to the step S2-3 until the number of the timestamp offsets reaches a set value to obtain a plurality of timestamp offsets;
s2-5: calculating the standard deviation of the plurality of timestamp offsets, if the standard deviation is not smaller than a preset value, abandoning the earliest calculated timestamp offset, repeating the steps S2-1 to S2-3 until the standard deviation is smaller than the preset value, and calculating the average value of the plurality of timestamp offsets to serve as a final timestamp offset result;
s2-6: the other guests calculate the final timestamp offset result between each of the other guests and the host according to steps S2-1 through S2-5.
4. The local multi-device fast spatial anchor synchronization method for mixed reality according to claim 3, wherein the formula of calculating the timestamp offset in step S2-3 is:
wherein, T0 is a local timestamp one, T0' is a local timestamp two, and T1 is a local timestamp three.
5. The local multi-device fast spatial anchor synchronization method for mixed reality of claim 3, wherein the one guest creation queue holds the plurality of timestamp offsets.
6. The local multi-device fast spatial anchor point synchronization method for mixed reality according to claim 1, wherein the step S3 specifically comprises:
s3-1: displaying a specific picture on a display system of the host, wherein the specific picture is an anchor point in space, and the host calculates the position and rotation of the specific picture under a local coordinate system of the host in real time by using an accelerometer, a gyroscope and an imaging system according to the invariance of the offset vectors of the specific picture and the imaging system of the host;
s3-2: a passenger plane in the local network scans the specific picture by using an imaging system through an augmented reality picture tracking technology, the augmented reality picture tracking technology identifies a second position and rotation of the specific picture in the scene, sends the second position and rotation to the host, and receives the first position and rotation sent by the host, wherein the first position and rotation and the second position and rotation have the same timestamp;
s3-3: the method comprises the steps that the passenger plane obtains a plurality of pairs of first positions and rotations and second positions and rotations under different timestamps, an optimal displacement vector and optimal rotation are obtained through least square calculation, the position and rotation of the origin of a local space coordinate system of a host relative to the origin of the local space coordinate system of the passenger plane are obtained, the origin of the local space coordinate system of the passenger plane is reset to the origin of the local space coordinate system of the host, and the synchronization of the space coordinate systems of the host and the passenger plane is completed; for other guests in the local network, the local space coordinate system origin of the guest is reset to the local space coordinate system origin of the host or the local space coordinate system origin of any synchronized guest, so that the space coordinate systems of the devices in the local network are synchronized;
s3-4: rendering a virtual object with the size consistent with that of the host equipment in the mixed reality virtual space of the guest machine, synchronizing the position and rotation of the virtual object with the host machine in real time, and if the virtual object is not completely overlapped with the host machine in reality, repeating the steps S3-1 to S3-3 by the guest machine until the virtual object is completely overlapped with the host machine in reality.
7. The local multi-device fast spatial anchor synchronization method for mixed reality according to claim 6, wherein the expression of the optimal displacement vector t is:
wherein the content of the first and second substances,
representing the weight center value of all anchor points under the host spatial coordinate system,
representing the center value of the weights of all anchor points under the one guest space coordinate system,
is a rotation matrix rotated by an angle phi along the spatial Y-axis.
8. The local multi-device fast spatial anchor synchronization method for mixed reality according to claim 6, wherein the optimal rotation φ is calculated by:
wherein, P' a,i,x 、P′ a,i,z Is the difference value P 'of the next anchor point of the host space coordinate system and the weight center thereof' b,i,x 、P′ b,i,z Is the difference, R ', of a next anchor point in the spatial coordinate system of the passenger aircraft and the weight center thereof' i,1,x 、R′ i,1,z 、R′ i,3,x 、R′ i,3,z Is the rotational difference between the spatial coordinate systems of the host and a guest, c is an adjustable constant greater than 0, and by adjusting the value of c, the formula can be biased toward smaller rotational errors or smaller displacement errors.
9. A local multi-device fast spatial anchor synchronization system for mixed reality, the system comprising a memory and a processor, the memory having stored thereon instructions that enable the processor to perform: the local multi-device fast spatial anchor synchronization method for mixed reality according to any one of claims 1-8.
10. A machine-readable storage medium having instructions stored thereon for causing a machine to perform: the local multi-device fast spatial anchor synchronization method for mixed reality according to any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US18/498,637 US20240154711A1 (en) | 2022-10-31 | 2023-10-31 | Local multi-device fast spatial anchor point synchronization method for mixed reality and system |
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2022113510789 | 2022-10-31 | ||
| CN202211351078 | 2022-10-31 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115834867A true CN115834867A (en) | 2023-03-21 |
Family
ID=85527332
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211397487.2A Pending CN115834867A (en) | 2022-10-31 | 2022-11-09 | Local multi-device rapid spatial anchor point synchronization method and system for mixed reality |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN115834867A (en) |
| WO (1) | WO2024092890A1 (en) |
-
2022
- 2022-11-09 CN CN202211397487.2A patent/CN115834867A/en active Pending
- 2022-11-19 WO PCT/CN2022/133039 patent/WO2024092890A1/en unknown
Also Published As
| Publication number | Publication date |
|---|---|
| WO2024092890A1 (en) | 2024-05-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10764626B2 (en) | Method and apparatus for presenting and controlling panoramic image, and storage medium | |
| TWI675351B (en) | User location location method and device based on augmented reality | |
| US20210233272A1 (en) | Data processing method and device used in virtual scenario | |
| US11090561B2 (en) | Aligning location for a shared augmented reality experience | |
| CN107330978B (en) | Augmented reality modeling experience system and method based on position mapping | |
| KR101720132B1 (en) | Method and system for sharing same augmented reality among multiple users based on positioning information | |
| CN102939139A (en) | Calibration of portable devices in shared virtual space | |
| WO2017027338A1 (en) | Apparatus and method for supporting interactive augmented reality functionalities | |
| JP7209474B2 (en) | Information processing program, information processing method and information processing system | |
| WO2018175335A1 (en) | Method and system for discovering and positioning content into augmented reality space | |
| US20220264251A1 (en) | System for combining sensor data to determine a relative position | |
| CN110728755A (en) | Method and system for roaming among scenes, model topology creation and scene switching | |
| CN103457965B (en) | Virtual machine system and remote display method | |
| US11914614B2 (en) | System and method for generating a collection of approximately coordinated region maps | |
| CN115834867A (en) | Local multi-device rapid spatial anchor point synchronization method and system for mixed reality | |
| US11388205B2 (en) | Systems and methods for establishing a shared augmented reality session | |
| CN112675541A (en) | AR information sharing method and device, electronic equipment and storage medium | |
| US20240154711A1 (en) | Local multi-device fast spatial anchor point synchronization method for mixed reality and system | |
| WO2006043319A1 (en) | Terminal and server | |
| CN110837297B (en) | Information processing method and AR equipment | |
| US11238658B2 (en) | AR space image projecting system, AR space image projecting method, and user terminal | |
| Negulescu et al. | Open project: a lightweight framework for remote sharing of mobile applications | |
| CN114373016A (en) | Method for positioning implementation point in augmented reality technical scene | |
| KR20220125536A (en) | System and operating method for providing mutual interaction service between virtual reality users and augmented reality users | |
| Jian et al. | Hybrid cloud computing for user location-aware augmented reality construction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |