WO2025035576A1 - 一种增强现实的方法、装置、存储介质及电子设备 - Google Patents

一种增强现实的方法、装置、存储介质及电子设备 Download PDF

Info

Publication number
WO2025035576A1
WO2025035576A1 PCT/CN2023/125666 CN2023125666W WO2025035576A1 WO 2025035576 A1 WO2025035576 A1 WO 2025035576A1 CN 2023125666 W CN2023125666 W CN 2023125666W WO 2025035576 A1 WO2025035576 A1 WO 2025035576A1
Authority
WO
WIPO (PCT)
Prior art keywords
augmented reality
data
user
interface
scene
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/CN2023/125666
Other languages
English (en)
French (fr)
Inventor
魏一雄
赵志峰
潘宇
陈高伟
李睿
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhejiang Lab
Original Assignee
Zhejiang Lab
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhejiang Lab filed Critical Zhejiang Lab
Publication of WO2025035576A1 publication Critical patent/WO2025035576A1/zh
Anticipated expiration legal-status Critical
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/451—Execution arrangements for user interfaces

Definitions

  • the present disclosure relates to the field of computer technology, and in particular to an augmented reality method, device, storage medium and electronic device.
  • Augmented/mixed reality technology is to seamlessly superimpose computer-simulated digital information data with the real scene environment, break through the limitations of space/time, and achieve high immersion, real-time information guidance, and virtual-real interaction without a sense of separation. Its essence is reflected in "what you see is what you get, what you touch is what you see”.
  • the information displayed/interacted with the help of the screen can be replaced with three-dimensional information that is strongly related to the actual environment objects, and presented intuitively in front of the user. This means that the user can use any plane in front of him as a screen and place any virtual object in any visual space (placed like real objects, and even simulating physical properties). Break the barriers between the virtual and real space and directly complete the communication between the virtual and real space. That is, through gesture recognition, depth recognition, artificial intelligence and other technologies, collect and understand the user's intentions, there is no separation between the digital virtual space and the real environment, and the information exchange is completed with the help of the user himself.
  • the present disclosure provides an augmented reality method, device, storage medium and electronic device to partially solve the above-mentioned problems existing in the prior art.
  • the present disclosure provides an augmented reality device, wherein the device is provided with a data-driven interactive interface module, a universal extensible service module, an identification and tracking module, a multi-person collaboration module, and a hybrid network communication module;
  • the data-driven interactive interface module is used to instantiate, for each interface element included in the interactive interface for interacting with the user, the class corresponding to the interface element to obtain a first instantiated object, and instantiate the element data class pre-bound to the interface element to obtain a second instantiated object, and for each specified variable included in the first instantiated object, bind the specified variable to a specified variable with the same name as the specified variable among the specified variables included in the second instantiated object, so that when the value of any specified variable bound to the first instantiated object in the second instantiated object changes, the presentation of the interface element corresponding to the specified variable bound to the any specified variable in the first instantiated object in the interactive interface changes synchronously;
  • the universal extensible service module is used to pre-save the container corresponding to the service executed by the augmented reality device, and when executing the target service, call the component in the container corresponding to the target service to execute the target service based on the called component; create a new service to be expanded through the preset components, and manage the creation status of the service to be expanded;
  • the recognition and tracking module is used to perform image recognition on the image data collected by the augmented reality device to determine the position and posture of the target object in the augmented reality interaction scene where the user is located;
  • the multi-person collaboration module is used to respond to a data synchronization instruction and send a data synchronization instruction generated by a local device to other augmented reality devices, wherein the data synchronization instruction is used to control a target object in the augmented reality interaction scene or update the state of the target object;
  • the hybrid network communication module is used to determine a transmission strategy according to different types of data to be transmitted generated in the augmented reality interaction scene, and transmit different types of data to be transmitted according to different transmission strategies.
  • the data-driven interactive interface module is also used to update the elements in the interface in real time according to the update status of the back-end data bound to the interface elements in the display interface of the augmented reality device.
  • the present disclosure also provides a method for augmenting reality, comprising:
  • the element variables of the interface element are bound to the data variables of the element data required for rendering the interface element, so as to realize the binding of the interface element with the element data corresponding to the interface element, so that the interface elements affected by the user's interactive actions and the element data required for rendering the interface elements are bound in the augmented reality interactive scene through the defined designated object class, wherein the designated object class includes the object class corresponding to the interface element and the object class corresponding to the interface element data;
  • the augmented reality interaction scene to be displayed by the user in the augmented reality device is determined, and the business scene data required in the augmented reality interaction scene and the common data used in different augmented reality interaction scenes are called to display the virtual interface object to be displayed by the user in the display interface of the augmented reality device, and the position and posture of the target object in the augmented reality interaction scene where the user is located are determined;
  • the posture and the synchronization instruction corresponding to the interactive action performed by the user are sent to other augmented reality devices, so that the other augmented reality devices synchronize the augmented reality interaction scene presented by the augmented reality device according to the posture and the synchronization instruction.
  • the method further includes: updating the elements in the interface in real time according to the update status of the data bound to the interface elements in the display interface of the augmented reality device in the back end.
  • Determining the position and posture of a target object in an augmented reality interaction scene where the user is located includes:
  • the posture of a target object in the augmented reality interaction scene where the user is located is determined according to the initial posture and the image data.
  • the scene data is transmitted to other augmented reality devices.
  • the scene data is transmitted to other augmented reality devices, and the method further includes:
  • the scene data is transmitted to other augmented reality devices according to the data transmission strategy determined by the user.
  • the method further comprises:
  • the service to be expanded is determined, the service to be expanded is instantiated through a specified container, and the executable service in the augmented reality device is updated according to the instantiated service to be expanded.
  • the present disclosure provides an augmented reality device, comprising:
  • a creation module used to create an augmented reality interaction scene based on an augmented reality interaction scene creation request received from a user
  • a binding module is used to preset element variables for each interface element and data variables for each element data, and for each interface element, bind the element variable of the interface element with the data variable of the element data required for rendering the interface element, so as to realize the binding of the interface element with the element data corresponding to the interface element, so that the interface elements affected by the user's interactive actions and the element data required for rendering the interface elements are bound in the augmented reality interaction scene through the defined specified object class, wherein the specified object class contains the object corresponding to the interface element The object class corresponding to the class and interface element data;
  • a determination and display module which is used to determine the augmented reality interaction scene that the user wants to display in the augmented reality device when an interactive action performed by the user is monitored, and call the business scene data required in the augmented reality interaction scene and the common data used in different augmented reality interaction scenes to display the virtual interface object that the user wants to display in the display interface of the augmented reality device, and determine the position and posture of the target object in the augmented reality interaction scene where the user is located;
  • a synchronization module is used to send synchronization instructions corresponding to the posture and the interactive action performed by the user to other augmented reality devices, so that the other augmented reality devices synchronize the augmented reality interaction scene presented by the augmented reality device according to the posture and the synchronization instructions.
  • the device further comprises:
  • the updating module is used to update the elements in the display interface of the augmented reality device in real time according to the update status of the data bound to the interface elements in the back end.
  • the determination and display module is used to collect image data of the augmented reality scene; identify the posture of the target object in the image represented by the image data as an initial posture according to a preset recognition model; and determine the posture of the target object in the augmented reality interaction scene in which the user is located based on the initial posture and the image data.
  • the synchronization module is used to determine scene data corresponding to the posture and the synchronization instruction corresponding to the interactive action performed by the user in the augmented reality interaction scene; and transmit the scene data to other augmented reality devices.
  • the synchronization module is also used to obtain a data transmission strategy determined by a user; and transmit the scene data to other augmented reality devices according to the data transmission strategy determined by the user.
  • the device further comprises:
  • the business expansion module is used to obtain the new business demand instruction sent by the user; in response to the new business demand instruction, determine the business to be expanded, instantiate the business to be expanded through the specified container, and update the executable business in the augmented reality device according to the instantiated business to be expanded.
  • the present disclosure provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned augmented reality method is implemented.
  • the present disclosure provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned augmented reality method when executing the program.
  • the augmented reality method creates an augmented reality interaction scene based on an augmented reality interaction scene creation request received from a user; by presetting element variables for each interface element and presetting data variables for each element data, for each interface element, the element variable of the interface element is bound to the data variable of the element data required for rendering the interface element, so as to realize the binding of the interface element with the element data corresponding to the interface element, so that the interface element affected by the user's interactive action and the element data required for rendering the interface element are bound in the augmented reality interaction scene through the defined specified object class, wherein the specified object class includes the object class corresponding to the interface element, the interface element data Corresponding object class; when monitoring the interactive action performed by the user, determining the augmented reality interactive scene that the user wants to display in the augmented reality device, and calling the business scenario data required in the augmented reality interactive scene and the common data used in different augmented reality interactive scenes, so as to display the virtual interface object that the user wants to display in the display interface of the augmented reality device, and determine
  • the user using the augmented reality device can use the augmented reality device to perform the interaction between virtual and reality, and the operations performed by the user when performing the interactive service can be synchronized to other users, so that the user can share the interaction between virtual and reality with other users.
  • FIG1 is a schematic diagram of the structure of an augmented reality device provided in the present disclosure.
  • FIG2 is a schematic diagram of a flow chart of an augmented reality method provided in the present disclosure
  • FIG3 is a schematic diagram of a process of creating a service to be expanded of an augmented reality device provided in the present disclosure
  • FIG4 is a schematic diagram of a flow chart of information interaction provided in the present disclosure.
  • FIG5 is a schematic diagram of a process of real-time sharing of interactive data of an augmented reality device provided in the present disclosure
  • FIG6 is a schematic diagram of a process of transmitting data by an augmented reality device provided in the present disclosure
  • FIG7 is a schematic diagram of an augmented reality device structure provided by the present disclosure.
  • FIG8 is a schematic structural diagram of an electronic device corresponding to FIG1 provided by the present disclosure.
  • the augmented reality device may include: a data-driven interactive interface module 101, a general extensible business module 102, an identification and tracking module 103, a multi-person collaboration module 104, and a hybrid network communication module 105.
  • People can use the display interface of the augmented reality device in the embodiment of the present disclosure to observe virtual objects in the real environment, and perform operations on the target objects together with other users to interact, thereby establishing a connection between the virtual space and the real space.
  • the augmented reality device in the embodiment of the present disclosure can be applied to augmented reality interaction scenarios such as augmented reality guidance scenarios and augmented reality game scenarios.
  • a pipeline maintenance guidance scenario in an augmented reality guidance scenario user A can see the pipeline that needs maintenance in front of him through an augmented reality device such as AR (Augmented Reality) glasses held by him, and send the image of the pipeline in front of user A to user B through the AR glasses.
  • User B can mark a virtual arrow in the video through a mobile terminal or augmented reality device such as a mobile phone or computer held by user B that can receive images and videos to guide user A to perform pipeline maintenance.
  • User A can receive the virtual arrow marked by user B through the AR glasses held by user A himself, and user A can observe the virtual arrow marked by user B appearing in the image of the pipeline in front of him through the AR glasses held by user A himself, and perform relevant operations according to the instructions of the arrow.
  • the augmented reality function of the augmented reality device can be implemented through the following modules.
  • the data-driven interactive interface module 101 is used to instantiate the class corresponding to each interface element contained in the interactive interface for interacting with the user to obtain a first instantiated object, and instantiate the element data class pre-bound to the interface element to obtain a second instantiated object, and for each specified variable contained in the first instantiated object, bind the specified variable to a specified variable with the same name as the specified variable among the specified variables contained in the second instantiated object, so that when the value of any specified variable bound to the first instantiated object in the second instantiated object changes, the presentation of the interface element corresponding to the specified variable bound to the any specified variable in the first instantiated object in the interactive interface changes synchronously.
  • the class corresponding to the above interface element can be an abstract interface element parent class
  • the element data class bound to the interface element can be an abstract element data parent class.
  • the interface element corresponding to the interactive action for example, a virtual button
  • the abstract interface element parent class is concretized as the interface element affected by the interactive action to obtain a specific interface element class
  • the abstract element data parent class is concretized as the background data of the interface element affected by the interactive action (i.e., the element data required for rendering the interface element) to obtain a specific element data class.
  • the interface element affected by the user's interactive action and the element data required for rendering the interface element in the augmented reality interaction scene are bound.
  • the general extensible business module 102 is used to pre-save the container corresponding to the business executed by the augmented reality device, and when executing the target business, call the component in the container corresponding to the target business to execute the target business based on the called component; create a new business to be expanded through the preset components, and manage the creation status of the business to be expanded.
  • the augmented reality device may first obtain a new business demand instruction sent by a user or a new business demand instruction sent by a server, and then respond to the new business demand instruction to determine the business to be expanded, instantiate the business to be expanded through a specified container, and update the executable business in the augmented reality device according to the instantiated business to be expanded.
  • the recognition and tracking module 103 is used to perform image recognition on the image data collected by the augmented reality device to determine the position and posture of the target object in the augmented reality interaction scene where the user is located.
  • the augmented reality device can specifically identify the target object based on the image (including color images and depth images) captured by the camera installed in the device through a preset recognition model, and identify the initial posture of the target object in the image (the posture includes position and direction), and based on the image, the initial posture and the parameters of the internal and external matrices of the camera installed in the device, determine the actual posture of the target object in the augmented reality interaction scene where the user is located.
  • the multi-person collaboration module 104 is used to respond to data synchronization instructions and send data synchronization instructions generated by the local device to other augmented reality devices.
  • the data synchronization instructions are used to control the target object in the augmented reality interaction scene or update the state of the target object.
  • the augmented reality device can complete real-time information interaction between users based on the multi-person collaboration module.
  • the hybrid network communication module 105 is used to determine a transmission strategy for different types of data to be transmitted generated in the augmented reality interaction scene, and transmit different types of data to be transmitted according to different transmission strategies.
  • the augmented reality device can allocate corresponding matching transmission strategies to the data to be transmitted based on the hybrid network communication module to achieve efficient data transmission.
  • the data-driven interactive interface module 101 is further used to update the elements in the interface in real time according to the update status of the data bound to the interface elements in the display interface of the augmented reality device at the back end.
  • the augmented reality device may further include: a starter 106, a spatial positioning module 107, a data storage module 108, and a state management module 109.
  • the starter 106 is used to start the augmented reality device
  • the spatial positioning module 107 is used to determine the spatial positioning information of the augmented reality device
  • the data storage module 108 is used to store the data generated during the execution of the service by the augmented display device
  • the state management module 109 is used to manage the state of each of the above modules.
  • FIG2 is a flow chart of an augmented reality method provided in the present disclosure, which specifically includes the following steps S201 - S204 .
  • S201 Creating an augmented reality interaction scene based on an augmented reality interaction scene creation request received from a user.
  • the method shown in FIG. 2 may be executed by the augmented reality device shown in FIG. 1 above.
  • the augmented reality device can create an augmented reality interaction scene corresponding to the augmented reality interaction scene creation request sent by the user.
  • the augmented reality interaction scene creation request sent by the user may refer to the user touching a control in the augmented reality device interface that indicates a request for an augmented reality interaction scene, or it may refer to the user uttering a voice that indicates a request for an augmented reality interaction scene.
  • no specific limitation is made on the triggering conditions of the above requests.
  • the augmented reality interaction scene created by the augmented reality device refers to a scene consistent with the business corresponding to the augmented reality interaction scene creation request issued by the user.
  • the above-mentioned augmented reality interaction scene can be a scene observed by the user through the augmented reality device.
  • the AR glasses can create an augmented reality interaction scene corresponding to the business in response to the request.
  • the augmented reality interaction scene includes the above-mentioned real environment and a virtual arrow created by the AR glasses to guide the user to complete the game. The arrow can point to the cup lid that best matches the cup.
  • the user can interact with other users in the augmented reality interactive scene created by the augmented reality device.
  • the element variables of the interface element are bound to the data variables of the element data required for rendering the interface element, so as to realize the binding of the interface element with the element data corresponding to the interface element, so that the interface elements affected by the user's interactive actions and the element data required for rendering the interface elements in the augmented reality interactive scene are bound through the defined designated object class, wherein the designated object class includes the object class corresponding to the interface element and the object class corresponding to the interface element data.
  • the augmented reality device will preset element variables for each interface element in the display interface of the augmented reality device, and for each of the above-mentioned interface elements, bind the element variable of the interface element with the data variable of the element data required to render the interface element, so as to realize the binding of the interface element with the element data corresponding to the interface element, and through the pre-defined designated object class, in the augmented reality interaction scene, the interface elements that will be affected by the user's interactive actions and the element data required to render the interface elements (that is, the data corresponding to the interface elements in the background) are bound.
  • the specified object class mentioned here may include: an abstract interface element parent class (ie, the class corresponding to the above interface element) and an abstract element data parent class (ie, the element data class bound to the above interface element).
  • the abstract interface element parent class contains multiple different interface elements
  • the abstract element data parent class contains the data corresponding to multiple different interface elements in the background.
  • Interface elements refer to the elements displayed on the display interface of the augmented reality device (such as buttons, charts, 3D models, etc. in the interactive interface).
  • Element data data corresponding to the interface elements in the background refers to the data representation corresponding to the above interface elements in the background (such as the trigger signal data of the button, the value data of the chart, the posture data of the 3D model, etc.).
  • the augmented reality device will pre-set corresponding binding element tools, such as variable A, for the interface elements affected by the user's interactive actions, and will also pre-set corresponding binding element attributes, such as variable B, for the background data of the interface elements affected by the user's interactive actions.
  • the augmented reality device will also establish a corresponding relationship between the binding element tool and the binding element attribute.
  • the interface element affected by the interaction can be determined.
  • the abstract interface element parent class is concretized into the interface element affected by the interaction, that is, a specific interface element class is obtained
  • the abstract element data parent class is concretized into the background data of the interface element affected by the user's interaction, that is, a specific element data class is obtained.
  • the specific interface element class can be bound to a preset binding element tool, and the specific element data class can be bound to a preset binding element attribute.
  • the interface element affected by the interactive action can be determined as a binding element tool, and based on the correspondence between the binding element tool and the binding element attribute, the binding element attribute corresponding to the binding element tool is determined to be the background data corresponding to the interface element affected by the interactive action, that is, the specific element data class corresponding to the binding element tool.
  • the interface element affected by the interaction action is the triangle button, that is, the specific interface element class is the triangle button. Since the interface element class is bound to the binding element tool variable A, the triangle button corresponding to variable A also performs the upward translation interaction action. Since the binding element attribute corresponding to the binding element tool variable A is variable B, the coordinates in the data corresponding to variable B will also be translated upward accordingly. Since the binding element attribute variable B is bound to the specific element data class corresponding to the triangle button, the coordinates in the data corresponding to the specific element data class corresponding to the triangle button will also be translated upward accordingly.
  • the elements in the interface can also be updated in real time according to the update status of the data bound to the interface elements in the display interface of the augmented reality device in the back end.
  • the specific update process will not be repeated in this disclosure.
  • the interface elements displayed in the user's augmented reality device are dynamically bound to the backend data corresponding to the interface elements.
  • S203 When an interactive action performed by the user is monitored, the augmented reality interaction scene that the user wants to display in the augmented reality device is determined, and the business scenario data required in the augmented reality interaction scene and the common data used in different augmented reality interaction scenes are called to display the virtual interface object that the user wants to display in the display interface of the augmented reality device, and determine the posture of the target object in the augmented reality interaction scene where the user is located.
  • the business scenario data required in the augmented reality interaction scenario refers to the data required for a specific business scenario, which may include data corresponding to rendering a specific augmented reality business scenario, and may also include data corresponding to different states of the same virtual object presented in different augmented reality interaction scenarios.
  • the interactive action here is used to represent the action issued by the user based on the augmented reality device to execute the business selected by the user.
  • the augmented reality device can call the business scenario data required in the augmented reality interaction scene (that is, the data corresponding to the different states of the same virtual object in different augmented reality interaction scenes) and the common data used in different augmented reality interaction scenes (that is, the data corresponding to the same state of the same virtual object in different augmented reality interaction scenes) to specifically display the virtual interface object that the user wants to display in the display interface of the augmented reality device, and determine the position and posture of the target object in the augmented reality interaction scene where the user is located.
  • the business scenario data required in the augmented reality interaction scene that is, the data corresponding to the different states of the same virtual object in different augmented reality interaction scenes
  • the common data used in different augmented reality interaction scenes that is, the data corresponding to the same state of the same virtual object in different augmented reality interaction scenes
  • the data required to create the hair is the business scenario data mentioned above
  • the data required to create other parts of the long-haired girl is the business scenario data mentioned above.
  • the data required for each part is the general data mentioned above.
  • the augmented reality device can determine the virtual interface object to be displayed in the display interface of the augmented reality device.
  • the augmented display device can determine the augmented reality interaction scene that the user wants to display in the augmented reality device, call the business scenario data required in the augmented reality interaction scene and the common data used in different augmented reality interaction scenes, and call the function in the preset object container that can be used to create the virtual interface object in the augmented reality interaction scene to create the virtual interface object, so as to display the virtual interface object that the user wants to display in the display interface of the augmented reality device.
  • the augmented reality device since the user's use of the augmented reality device to perform business involves the user's operation on the target object displayed on the display interface of the augmented reality device, it is necessary to track the target object so that the user can successfully complete the operation on the target object or virtual interface object displayed on the display interface of the augmented reality device. At this time, the augmented reality device needs to determine the position and posture of the target object in the augmented reality interaction scene where the user is located.
  • the target object here refers to the object that needs to be operated corresponding to the business executed by the user.
  • the target object at this time is the cup and a cup lid that matches the cup
  • the virtual arrow guiding the user to complete the game is the virtual interface object to be displayed in the display interface of the augmented reality device.
  • the augmented reality device may specifically collect an image of the augmented reality scene created for the user by the augmented reality device in front of the user through a camera installed in the device.
  • the image here may include an RGB (Red, Grey, Blue) image of the image, that is, a color image, and a depth image of the image.
  • the image data corresponding to the image may also be determined, and the image data here may include: the image data of the image and the depth data of the image, and the position and posture of the target object in the pixel coordinate system of the image is identified as the initial position and posture according to a preset recognition model, and the preset recognition model here may also identify what the target object is specifically, and the above-mentioned position and posture include position and direction.
  • the augmented reality device can determine the actual posture of the target object in the augmented reality interaction scene where the user is located based on the initial posture, the RGB image and depth image corresponding to the image, the above-mentioned image data and the parameters of the internal and external matrices of the camera installed in the device.
  • a series of spatial coordinate transformations are performed to determine the actual position and posture of the target object in the augmented reality interaction scene where the user is located. That is, firstly, a one-to-one matching at the pixel level is performed based on the depth image and the RGB image to convert the pixel coordinate system where the initial position and posture are located into the image coordinate system, and at the same time, the initial position and posture are also converted into the initial position and posture in the image coordinate system.
  • the augmented reality device can convert the image coordinate system into a camera coordinate system with the camera under the user's perspective (the position of the camera under the user's perspective and the camera installed in the device are unified) as the zero position, and convert the initial position and posture in the image coordinate system into the initial position and posture in the camera coordinate system.
  • the camera coordinate system can also be converted into the world coordinate system based on the parameters of the internal and external matrices of the camera installed in the device, and the initial position and posture in the camera coordinate system can be converted into the initial position and posture in the world coordinate system.
  • the initial position and posture in the world coordinate system here is the actual position and posture of the target object in the augmented reality interaction scene where the user is located mentioned above.
  • the preset recognition model here can be constructed and trained according to the following ideas:
  • a target recognition model framework First, set a predefined scale range and center position for each object, and specify a feature pyramid network (FPN) for each object.
  • FPN feature pyramid network
  • the Feature Pyramid Network solves this problem by constructing multiple feature maps of different resolutions. It uses a convolutional neural network (CNN) with shared weights to extract features from the input image. Then, for each different scale, it resizes the original image by increasing or decreasing the stride. At each scale, FPN generates a set of feature maps from high to low, which are called "pyramids", where the bottom layer is the feature map obtained by upsampling the original image, and the top layer is a coarse feature map obtained by applying pooling operations and convolution operations.
  • CNN convolutional neural network
  • the network consists of two sub-network modules, the first is a top-down architecture (the bottom-up network starts from the original image, and gradually reduces the image size through multiple convolutional layers and pooling layers to obtain a series of feature maps from coarse to fine.
  • the bottom-up network is mainly responsible for extracting low-level features, such as edges and textures, etc.
  • the second is a bottom-up architecture (the top-down network starts from the top of the pyramid, and gradually restores the size of the feature map through upsampling or deconvolution operations, while adding lateral connections at each layer.
  • the top-down network is laterally connected to the corresponding layers of the bottom-up network so that the top-down network can obtain high-level feature information from the bottom-up network.
  • the top-down network is mainly responsible for extracting high-level features, such as semantic information and the overall shape of objects).
  • Each sub-network uses deep residual network (ResNet) blocks to build a deep network, which contain jump connections across layers.
  • the jump connection allows some layers to be skipped between the input and output layers, so that the network can capture higher-level features and is easier to train.
  • the jump connection can also pass the gradient back to the earlier layer, alleviating the gradient vanishing problem.
  • the RGB image is input into the bottom-up subnetwork, and then the feature map is transferred to the next subnetwork.
  • each feature map of each layer is combined to predict the object category and the corresponding candidate bounding box. Subsequently, based on the target recognition model framework, a deep learning recognition model can be obtained by collecting a large number of images of the target object for training.
  • the camera in the user's AR glasses can capture the image (including RGB image and depth image) corresponding to the augmented reality interaction scene seen from the user's perspective.
  • the augmented reality device can determine the image data corresponding to the image, which may include the image data of the RGB image captured by the camera when the user performs the "multi-person cooperative cup lid twisting" service and the depth data of the depth image matching the image.
  • the augmented reality device can identify the position of the target object cup lid in the pixel coordinate system of the image as the initial position according to a preset recognition model.
  • the actual position of the target object in the augmented reality interaction scene where the user is located can be determined by performing a series of spatial coordinate transformations.
  • the service that the user needs to add is taken as the service to be expanded, and the service to be expanded is instantiated.
  • the instantiated service is retrieved into the preset IOC (Inversion of Control) container, and the behavior actions of the service to be expanded in the interactive interface are managed, and the instantiated service to be expanded is called to update the executable service in the augmented reality device.
  • IOC Inversion of Control
  • FIG3 is a schematic diagram of a process of creating a service to be expanded of an augmented reality device provided in the present disclosure
  • the button when the user touches a button in the display interface of the enhanced display device, the button is used to indicate that a new service A and a new service B are to be added to the enhanced display device, and service A and service B are the services to be expanded, the element data corresponding to the button is activated, and the preset interface of the control component in the enhanced display device will respond to the changes in the preset element data.
  • the business context component used to manage different types of business can obtain the business to be expanded determined by the user through the control component after IOC dependency injection, and the business context component can derive the business context component of business A and the business context component of business B based on the business to be expanded determined by the user.
  • the business instance component used for specific business instantiation can rely on the business context component of the above-mentioned business A and the business context component of the business B, derive the business instance component of the business A and the business instance component of the business B through the business component, and determine the tasks A1 to be instantiated, A2 to be instantiated and A3 to be instantiated contained in the business A, and determine the tasks B1 to be instantiated and B2 to be instantiated contained in the business B.
  • all the above-mentioned tasks to be instantiated can be injected into the IOC container by means of dependency injection, and all the above-mentioned tasks to be instantiated are instantiated in the IOC container to realize the instantiation of the above-mentioned business to be expanded, that is, at this time, the IOC container will contain the instantiated business A and business B, and the instantiation mentioned above can be the specific implementation of the functions corresponding to the above-mentioned tasks.
  • the preset first control component will call the business A and business B from the preset IOC container to realize the update of the executable business in the augmented reality device.
  • S204 Send the posture and the synchronization instruction corresponding to the interactive action performed by the user to other augmented reality devices, so that the other augmented reality devices synchronize the augmented reality interaction scene presented by the augmented reality device according to the posture and the synchronization instruction.
  • users When users use augmented reality devices to perform business, they may be involved in information interaction with other users, that is, multiple users may perform operations on the same target object or virtual interface object, or users may perform operations or other interactive operations on virtual interface objects created by each other.
  • the augmented reality device in order to ensure the user experience of the augmented reality device, so that the interactive interface of each user's augmented reality device can promptly display the effects of these operations to all the users who perform the business together, it is necessary for the augmented reality device to share the effects of these operations in real time among each user.
  • the online and offline status of each user can be represented by the user's virtual character image in the augmented reality interaction scene, that is, if the user is online, then the user's virtual character image will appear in the augmented reality interaction scene, otherwise, the user's virtual character image will not appear in the augmented reality interaction scene.
  • the virtual character of the first user needs to disappear, which is the synchronization instruction mentioned above that corresponds to the operation and needs to be sent synchronously to other augmented reality devices.
  • the virtual character of the first user in each user's augmented reality device will disappear.
  • each of the five users has a virtual ball, and the five users take turns throwing the virtual ball into a box in front of them in reality. If the first user performs an interactive action of throwing a virtual ball into the box, the augmented reality device of the first user will send the position of the virtual ball and the synchronization instruction corresponding to the interactive action to other augmented reality devices, so that the other augmented reality devices synchronize the augmented reality interaction scene presented by the augmented reality device according to the position and the synchronization instruction.
  • the augmented reality device may also respond to the data update instruction sent by the server to update the scene or button or other components of the display interface of the augmented reality device
  • the synchronization instruction can be an instruction to update the above content.
  • the server needs to simulate the scene of the sun rising and setting in nature in the display interface of the augmented reality device
  • the augmented reality device can respond to the synchronization instruction to update the position, color and other information of the sun in the above display interface, and the sun here can be the specified object mentioned above that needs to be updated.
  • FIG. 4 is a schematic diagram of a flow chart of information interaction provided in the present disclosure.
  • the collaborative component in FIG4 is responsible for the management of the server, the client, and multiple network connections.
  • the server is the intermediate node of multiple clients in the information interaction, responsible for managing the client's addition, deletion, behavior response, data transmission, and other behaviors.
  • the server here can be either an augmented display device or a server in reality.
  • the client is an augmented display device, which is used to manage the current user's behavior transmission and the synchronization of other synchronization instruction data locally on the client.
  • the "network connection" management component in FIG4 includes two derived components: the management component of the network connection server (corresponding to the network connection with the server) and the client management component of the network connection server (corresponding to the network connection with the client).
  • each role needs to be uniquely marked to facilitate distinction in data synchronization, so a collaborative identity component is required to determine a unique identity for each synchronization instruction.
  • the behavior management component is managed by the above-mentioned server and client, and is associated with the "network connection" management component.
  • the behavior management component is responsible for the behavior management of all roles.
  • the behavior management component is managed by the collaborative identity identification component.
  • the behavior management component can clearly determine the sender and receiver of the synchronization instruction based on the information in the collaborative identity identification component to achieve stable information interaction.
  • the transformation component is used for interface management of network communication to achieve information interaction and partial information connection for subsequent data transmission.
  • FIG5 is a schematic diagram of a process of real-time sharing of interactive data of an augmented reality device provided in the present disclosure.
  • the client in Figure 5 refers to the augmented reality device held by the user, and the server refers to the server or an augmented reality device designated as the server.
  • the client is used to manage the current user's behavior transmission and the local synchronization of other client collaborative information data.
  • the server is the intermediate node of multiple clients in multi-person collaboration, responsible for managing the client's addition, deletion, behavior response, data transmission and other behaviors.
  • Other clients in Figure 5 refer to augmented reality devices held by other users other than the user that have established information connections with the server.
  • the collaboration component is used to manage the server and client.
  • the client held by the user can send a connection request carrying a creation client object to the server.
  • the client object here can be used to represent the virtual interface object that the user wants to display in the display interface of the augmented reality device corresponding to the client.
  • the server can trigger the instruction to create the client object in the server through a preset callback function.
  • the cooperative component can create the client object in the augmented reality interaction scene on the server according to the callback function and mark the created client object with the client.
  • the server can mark the client as information that requires collaboration (i.e., synchronization variables) and serialize the synchronization variables (i.e., convert the synchronization variables into binary data).
  • the server can send information about the client object carrying the serialized synchronization variables to other clients through the collaboration component.
  • the collaboration component can send the client object and the client's tag (i.e., the subject affiliation information in Figure 5) to the client.
  • the client can grant the server the authority to create the client object in the client's augmented reality scene and present the client object in the client's augmented reality scene.
  • other clients can parse the information and deserialize the information of the client object carrying the serialized synchronization variables (i.e., convert the binary data into the form of synchronization variables) to obtain the information of the client and the client object created by the client, and synchronously create the client object in the augmented reality scene created by other clients.
  • the augmented reality device can obtain the synchronization instructions issued by each user for the cup cover in turn. For example, there are 3 users (user A, user B, user C) participating in the game. User A first issues a synchronization instruction to move the cup cover 3 cm to the left.
  • the augmented reality device needs to send the synchronization instruction to other augmented reality devices, so that other augmented reality devices can synchronize the augmented reality interaction scenes presented by these other augmented reality devices according to the posture of the cup cover after user A moves it. That is, after user A moves the cup cover 3 cm to the left, after the above synchronization, the posture of the cup cover recognized by the augmented reality devices of users B and C is also 3 cm to the left based on the original posture of the cup cover.
  • the augmented reality device can synchronize the scene data to the augmented reality devices held by other users according to the preset data transmission strategy and data transmission method corresponding to the scene data when transmitting data.
  • the data transmission strategy here can also be specified by the user. For example, the user can specify that the augmented reality device transmits video at a transmission speed of 30 frames per second.
  • FIG6 is a schematic diagram of a flow chart of data transmission by an augmented reality device provided in the present disclosure.
  • the hybrid network communication module is mainly composed of five logical layers: physical layer, transport layer, policy layer, data layer, and multi-channel communication control.
  • the data layer contains data that needs to be communicated, such as application basic data, interaction data, status data, scene data, virtual-reality fusion data, collection data, analysis and evaluation data, and other augmented reality application data.
  • the data layer is mainly used to classify and label the data that needs to be communicated.
  • the labels are divided into data strategy S and data transmission method T.
  • the strategy layer is divided into three types: high-frequency real-time transmission strategy, low-frequency online transmission strategy, and signal-triggered transmission strategy.
  • the high-frequency real-time transmission strategy is aimed at communication needs with high real-time requirements for data transmission.
  • the transmission frequency adopts a predefined unified frequency, and the amount of data transmitted is limited and optimized by the specific network conditions.
  • the low-frequency online transmission strategy is aimed at scenarios with low requirements for data update frequency. It can transmit large-capacity data, and the transmission frequency adopts a predefined unified frequency.
  • the signal-triggered transmission strategy is aimed at the business needs of users. It can be triggered by internal processes or external inputs, and there is no specific limit on the amount of data transmitted.
  • the transport layer is divided into three transmission modes: lightweight communication, asynchronous large-capacity communication, and streaming data communication.
  • Lightweight communication is suitable for small data communication needs and is completed based on the basic Socket communication method.
  • Streaming data communication is suitable for real-time audio and video communication needs, with high real-time requirements, and is usually implemented based on Web Real-Time Communication (WebRTC).
  • WebRTC Web Real-Time Communication
  • Asynchronous large-capacity communication is suitable for communication situations with large data volumes and requirements for transmission speed and transmission quality.
  • the multi-channel communication control layer is used to judge, integrate, make decisions and verify the strategies and specific methods of data transmission. It mainly includes the construction of a converged network (situations where multiple data strategies or transmission methods need to be integrated at the same time in business needs), protocol adaptation (protocol compatibility of different data transmission interfaces), data classification and label addition, data serialization and deserialization, communication session construction (such as building a chat dialog box), concurrent thread management for improving data transmission efficiency and stability, and data information verification (used to verify the security of transmitted data).
  • the user using the augmented reality device can use the augmented reality device to perform the interaction between virtual and reality, and the operations performed by the user when performing the interactive service can be synchronized to other users, so that the user can share the interaction between virtual and reality with other users.
  • the present disclosure also provides a corresponding augmented reality apparatus, as shown in FIG. 7 .
  • FIG7 is a schematic diagram of an augmented reality device provided by the present disclosure, comprising:
  • the creation module 701 is used to create an augmented reality interaction scene creation request sent by the user. Augmented reality interactive scenes;
  • the binding module 702 is used to bind the element variable of each interface element with the data variable of the element data required for rendering the interface element according to the element variable preset for the interface element and the data variable preset for the element data, so as to realize the binding of the interface element with the element data corresponding to the interface element, so that the interface element affected by the user's interactive action and the element data required for rendering the interface element are bound in the augmented reality interactive scene through the defined designated object class, wherein the designated object class includes the object class corresponding to the interface element and the object class corresponding to the interface element data;
  • the determination and display module 703 is used to determine the augmented reality interaction scene that the user wants to display in the augmented reality device when the interactive action performed by the user is monitored, and call the business scene data required in the augmented reality interaction scene and the common data used in different augmented reality interaction scenes to display the virtual interface object that the user wants to display in the display interface of the augmented reality device, and determine the position and posture of the target object in the augmented reality interaction scene where the user is located;
  • the synchronization module 704 is used to send the synchronization instructions corresponding to the posture and the interactive action performed by the user to other augmented reality devices, so that the other augmented reality devices synchronize the augmented reality interaction scene presented by the augmented reality device according to the posture and the synchronization instructions.
  • the device further comprises:
  • the updating module 705 is used to update the elements in the display interface of the augmented reality device in real time according to the update status of the data bound to the interface elements in the back end.
  • the determination and display module 703 is used to collect image data of the augmented reality scene; identify the posture of the target object in the image represented by the image data as an initial posture according to a preset recognition model; and determine the posture of the target object in the augmented reality interaction scene in which the user is located based on the initial posture and the image data.
  • the synchronization module 704 is used to determine the scene data corresponding to the synchronization instruction corresponding to the posture and the interactive action performed by the user in the augmented reality interaction scene; and transmit the scene data to other augmented reality devices.
  • the synchronization module 704 is further used to obtain a data transmission strategy determined by a user; and transmit the scene data to other augmented reality devices according to the data transmission strategy determined by the user.
  • the device further comprises:
  • the business expansion module 706 is used to obtain the new business demand instruction sent by the user; in response to the new business demand instruction, determine the business to be expanded, instantiate the business to be expanded through the specified container, and update the executable business in the augmented reality device according to the instantiated business to be expanded.
  • the present disclosure also provides a computer-readable storage medium, which stores a computer program.
  • the computer program can be used to execute an augmented reality method provided in FIG. 1 above.
  • the present disclosure also provides a schematic structural diagram of an electronic device corresponding to FIG1 as shown in FIG8.
  • the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include hardware required for other services.
  • the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the augmented reality method described in FIG2 above.
  • a programmable logic device such as a field programmable gate array (FPGA)
  • FPGA field programmable gate array
  • HDL There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc.
  • ABEL Advanced Boolean Expression Language
  • AHDL Altera Hardware Description Language
  • HDCal JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc.
  • VHDL Very-High-Speed Integrated Circuit Hardware Description Language
  • Verilog Verilog
  • the controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer readable medium storing a computer readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory.
  • a computer readable program code e.g., software or firmware
  • the controller may be implemented in the form of a logic gate, a switch, an application specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the means for implementing various functions included therein may also be considered as a structure within the hardware component. Or even, the means for implementing various functions may be considered as both a software module for implementing the method and a structure within the hardware component.
  • a typical implementation device is a computer.
  • the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
  • the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
  • computer-usable storage media including but not limited to disk storage, CD-ROM, optical storage, etc.
  • These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.
  • These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.
  • a computing device includes one or more processors (CPU), input/output interfaces, network interfaces, and memory.
  • processors CPU
  • input/output interfaces network interfaces
  • memory volatile and non-volatile memory
  • Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and/or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
  • RAM random access memory
  • ROM read-only memory
  • flash RAM flash memory
  • Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information.
  • Information can be computer readable instructions, data structures, program modules or other data.
  • Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.
  • computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
  • the embodiments of the present disclosure may be provided as methods, systems or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
  • computer-usable storage media including but not limited to disk storage, CD-ROM, optical storage, etc.
  • program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types.
  • program modules may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network.
  • program modules may be located in local and remote computer storage media, including storage devices.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Processing Or Creating Images (AREA)

Abstract

本公开公开了一种增强现实的方法、装置、存储介质及电子设备，通过为增强现实设备部署包含多个可以互相进行协作的模块，使得使用增强现实设备的用户能够使用增强现实设备执行完成虚拟与现实的交互，并且所述用户在执行所述交互业务时所做出的操作可同步给其他用户，以使得该用户能够与其他用户共享所述虚拟与现实的交互。

Description

一种增强现实的方法、装置、存储介质及电子设备 技术领域
本公开涉及计算机技术领域，尤其涉及一种增强现实的方法、装置、存储介质及电子设备。
背景技术
增强/混合现实技术是将计算机模拟的数字化信息数据与现实场景环境无缝叠加，突破空间/时间限制，实现高沉浸、无割裂感的实时信息指引、虚实互动。其本质体现在“所见即得，所触即现”。借助此项技术，可以将借助屏幕显示/交互的信息替换成与实际环境对象强关联的三维信息，直观呈现于使用者眼前。也就意味着，使用者可以将面前的任何平面作为屏幕，在任意视野空间放置任意虚拟物品(和真实物品一样摆放，甚至模拟物理属性)。打破空间虚实壁垒，直接完成虚实空间交流。即通过手势识别、深度识别、人工智能等技术，采集、理解用户意图，数字虚拟空间与真实环境之间不存在割裂，借助使用者自身完成信息交流。
由于增强/混合现实的设备和系统应用还处于发展阶段，特别是针对通用应用的框架设计还没有统一的规范和方法。因此本公开从应用系统的架构、模块设计、多人协同等角度出发，提出一种通用的应用系统框架，可以满足未来增强/现实环境的实际应用需求。
发明内容
本公开提供一种增强现实的方法、装置、存储介质及电子设备，以部分的解决现有技术存在的上述问题。
本公开采用下述技术方案：
本公开提供了一种增强现实的设备，所述设备中设有数据驱动交互界面模块、通用可扩展业务模块、识别追踪模块、多人协同模块、混合网络通信模块；
所述数据驱动交互界面模块，用于针对与用户进行交互的交互界面中包含的每个界面元素，对该界面元素对应的类进行实例化得到第一实例化对象，并对预先与该界面元素绑定的元素数据类进行实例化，得到第二实例化对象，并针对所述第一实例化对象中包含的每个指定变量，将该指定变量与所述第二实例化对象中包含的指定变量中与该指定变量名称相同的指定变量相绑定，以在所述第二实例化对象中与所述第一实例化对象绑定的任意一个指定变量的值改变时，所述第一实例化对象中与所述任意一个指定变量相绑定的指定变量对应的界面元素在所述交互界面中的呈现同步改变；
所述通用可扩展业务模块，用于预先保存所述增强现实设备所执行的业务对应的容器，并在执行目标业务时，调用所述目标业务对应容器中的组件，以基于调用出的组件，执行所述目标业务；通过预设的各个组件，创建新的待扩展业务，并管理所述待扩展业务的创建状态；
所述识别追踪模块，用于对所述增强现实设备采集到的图像数据进行图像识别，以确定出所述用户所处的增强现实交互场景中目标对象的位姿；
所述多人协同模块，用于响应数据同步指令，并向其他增强现实设备发送本地设备所产生的数据同步指令，所述数据同步指令用于对所述增强现实交互场景中的目标对象进行控制或对目标对象的状态进行更新；
所述混合网络通信模块，用于根据所述增强现实交互场景中产生不同类型的待传输数据确定传输策略，并分别按照不同的传输策略，将不同类型的待传输数据进行传输。
可选的，所述数据驱动交互界面模块还用于根据后端的与所述增强现实设备的显示界面中的界面元素所绑定的数据的更新情况来实时更新所述界面中的元素。
本公开还提供了一种增强现实的方法，包括：
基于接收到的用户所发送的增强现实交互场景创建请求，创建增强现实交互场景；
根据为界面元素预设的元素变量，和为元素数据预设的数据变量，针对每个界面元素，将该界面元素的元素变量与渲染该界面元素所需的元素数据的数据变量相绑定，实现该界面元素与该界面元素对应的元素数据的绑定，以使通过定义的指定对象类，在增强现实交互场景中对用户的交互动作所影响的界面元素以及渲染界面元素所需的元素数据进行绑定，其中，所述指定对象类包含有界面元素对应的对象类、界面元素数据对应的对象类；
当监测到所述用户执行的交互动作时，确定所述用户在所述增强现实设备中所要展示的增强现实交互场景，并调用所述增强现实交互场景中所需的业务场景数据以及在不同增强现实交互场景中所使用的通用数据，以在所述增强现实设备的显示界面中展示用户所要展示的虚拟界面对象，并确定出所述用户所处的增强现实交互场景中目标对象的位姿；
将所述位姿以及所述用户执行的交互动作所对应的同步指令发送给其他增强现实设备，以使得其他增强现实设备根据所述位姿以及同步指令来同步该增强现实设备所呈现的增强现实交互场景。
可选地，所述方法还包括：根据后端的与所述增强现实设备的显示界面中的界面元素所绑定的数据的更新情况来实时更新所述界面中的元素。
可选地，
确定出所述用户所处的增强现实交互场景中目标对象的位姿，包括：
采集所述增强现实场景的图像数据；
根据预设的识别模型识别所述目标对象在所述图像数据所表示的图像中的位姿作为初始位姿；
根据所述初始位姿以及所述图像数据，确定出所述用户所处的增强现实交互场景中目标对象的位姿。
可选地，
将所述位姿以及所述用户执行的交互动作所对应的同步指令发送给其他增强现实设备，包括：
确定所述位姿以及所述用户执行的交互动作所对应的同步指令在所述增强现实交互场景中所对应的场景数据；
将所述场景数据传输给其他增强现实设备。
可选地，将所述场景数据传输给其他增强现实设备，所述方法还包括：
获取用户所确定的数据传输策略；
根据所述用户所确定的数据传输策略，将所述场景数据传输给其他增强现实设备。
可选地，所述方法还包括：
获取用户发送的新增业务需求指令；
响应于所述新增业务需求指令，确定待扩展业务，通过指定容器，将待扩展业务实例化，根据所述实例化的待扩展业务，更新所述增强现实设备中的可执行业务。
本公开提供了一种增强现实的装置，包括：
创建模块，用于基于接收到的用户所发送的增强现实交互场景创建请求，创建增强现实交互场景；
绑定模块，用于通过为各个界面元素预设有元素变量，为各个元素数据预设有数据变量，针对每个界面元素，将该界面元素的元素变量与渲染该界面元素所需的元素数据的数据变量相绑定，实现该界面元素与该界面元素对应的元素数据的绑定，以使通过定义的指定对象类，在增强现实交互场景中对用户的交互动作所影响的界面元素以及渲染界面元素所需的元素数据进行绑定，其中，所述指定对象类包含有界面元素对应的对象 类、界面元素数据对应的对象类；
确定与展示模块，用于当监测到用户执行的交互动作时，确定用户在所述增强现实设备中所要展示的增强现实交互场景，并调用所述增强现实交互场景中所需的业务场景数据以及在不同增强现实交互场景中所使用的通用数据，以在所述增强现实设备的显示界面中展示用户所要展示的虚拟界面对象，并确定出所述用户所处的增强现实交互场景中目标对象的位姿；
同步模块，用于将所述位姿以及所述用户执行的交互动作所对应的同步指令发送给其他增强现实设备，以使得其他增强现实设备根据所述位姿以及同步指令来同步该增强现实设备所呈现的增强现实交互场景。
可选的，所述装置还包括：
更新模块，用于根据后端的与所述增强现实设备的显示界面中的界面元素所绑定的数据的更新情况来实时更新所述界面中的元素。
可选的，所述确定与展示模块用于，采集所述增强现实场景的图像数据；根据预设的识别模型识别所述目标对象在所述图像数据所表示的图像中的位姿作为初始位姿；根据所述初始位姿以及所述图像数据，确定出所述用户所处的增强现实交互场景中目标对象的位姿。
可选的，所述同步模块用于，确定所述位姿以及所述用户执行的交互动作所对应的同步指令在所述增强现实交互场景中所对应的场景数据；将所述场景数据传输给其他增强现实设备。
可选的，所述同步模块还用于，获取用户所确定的数据传输策略；根据所述用户所确定的数据传输策略，将所述场景数据传输给其他增强现实设备。
可选的，所述装置还包括：
业务拓展模块，用于获取用户发送的新增业务需求指令；响应于所述新增业务需求指令，确定待扩展业务，通过指定容器，将待扩展业务实例化，根据所述实例化的待扩展业务，更新所述增强现实设备中的可执行业务。
本公开提供了一种计算机可读存储介质，所述存储介质存储有计算机程序，所述计算机程序被处理器执行时实现上述增强现实的方法。
本公开提供了一种电子设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述程序时实现上述增强现实的方法。
本公开采用的上述至少一个技术方案能够达到以下有益效果：
本公开提供的增强现实的方法，基于接收到的用户所发送的增强现实交互场景创建请求，创建增强现实交互场景；通过为各个界面元素预设有元素变量，为各个元素数据预设有数据变量，针对每个界面元素，将该界面元素的元素变量与渲染该界面元素所需的元素数据的数据变量相绑定，实现该界面元素与该界面元素对应的元素数据的绑定，以使通过定义的指定对象类，在增强现实交互场景中对用户的交互动作所影响的界面元素以及渲染界面元素所需的元素数据进行绑定，其中，所述指定对象类包含有界面元素对应的对象类、界面元素数据对应的对象类；当监测到用户执行的交互动作时，确定用户在所述增强现实设备中所要展示的增强现实交互场景，并调用所述增强现实交互场景中所需的业务场景数据以及在不同增强现实交互场景中所使用的通用数据，以在所述增强现实设备的显示界面中展示用户所要展示的虚拟界面对象，并确定出所述用户所处的增强现实交互场景中目标对象的位姿；将所述位姿以及所述用户执行的交互动作所对应的同步指令发送给其他增强现实设备，以使得其他增强现实设备根据所述位姿以及同步指令来同步该增强现实设备所呈现的增强现实交互场景。
从上述方法中可以看出，通过为增强现实设备部署包含多个可以互相进行协作的模块，使得使用增强现实设备的用户能够使用增强现实设备执行完成虚拟与现实的交互，并且所述用户在执行所述交互业务时所做出的操作可同步给其他用户，以使得该用户能够与其他用户共享所述虚拟与现实的交互。
附图说明
此处所说明的附图用来提供对本公开的进一步理解，构成本公开的一部分，本公开的示意性实施例及其说明用于解释本公开，并不构成对本公开的不当限定。
图1为本公开中提供的一种增强现实设备的结构示意图；
图2为本公开中提供的一种增强现实方法的流程示意图；
图3为本公开中提供的一种增强现实设备的待扩展业务的创建过程的示意图；
图4为本公开中提供的一种信息交互的流程示意图；
图5为本公开中提供的一种增强现实设备交互数据的实时共享的流程示意图；
图6为本公开中提供的一种增强现实设备传输数据的流程示意图；
图7为本公开提供的一种增强现实的装置结构的示意图；
图8为本公开提供的对应于图1的电子设备的结构示意图。
具体实施方式
为使本公开的目的、技术方案和优点更加清楚，下面将结合本公开具体实施例及相应的附图对本公开技术方案进行清楚、完整地描述。所描述的实施例仅是本公开一部分实施例，而不是全部的实施例。基于本公开中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本公开保护的范围。
以下结合附图，详细说明本公开各实施例提供的技术方案。
图1为本公开中提供的一种增强现实设备的结构示意图，该增强现实设备可包括：数据驱动交互界面模块101、通用可扩展业务模块102、识别追踪模块103、多人协同模块104、混合网络通信模块105。
人们可以使用本公开实施例中的增强现实设备的展示界面，观察到真实环境中的虚拟物体，并同其他用户一起对目标对象执行操作，进行互动，建立起了虚拟空间与现实空间之间的连接。
本公开实施例中的增强现实设备可应用增强现实指导场景、增强现实游戏场景等增强现实交互场景。
例如，在增强现实指导场景中的管道维修指导场景中，A用户可以通过自己所持有的如AR(Augmented Reality)眼镜等的增强现实设备看到眼前需要维修的管道，并通过该AR眼镜将A用户眼前的包含管道的图像发送给B用户，B用户可以通过B用户所持有的类似手机、电脑等可以接收图像视频的移动终端或者增强现实设备来在该视频中标记出虚拟箭头来指导A用户进行管道维修，A用户可以通过A用户自身持有的该AR眼镜来接收B用户所标记的虚拟箭头，并且A用户可以通过该A用户自身持有的该AR眼镜观察到眼前的包含管道的图像中出现的上述B用户所标记的虚拟箭头，并根据该箭头的指示进行相关操作。
在该增强现实设备运行过程中，可通过以下模块实现增强现实设备的增强现实功能。
数据驱动交互界面模块101，用于针对与用户进行交互的交互界面中包含的每个界面元素，对该界面元素对应的类进行实例化得到第一实例化对象，并对预先与该界面元素绑定的元素数据类进行实例化，得到第二实例化对象，并针对所述第一实例化对象中包含的每个指定变量，将该指定变量与所述第二实例化对象中包含的指定变量中与该指定变量名称相同的指定变量相绑定，以在所述第二实例化对象中与所述第一实例化对象绑定的任意一个指定变量的值改变时，所述第一实例化对象中与所述任意一个指定变量相绑定的指定变量对应的界面元素在所述交互界面中的呈现同步改变。
上述界面元素对应的类可以是抽象界面元素父类，而与该界面元素绑定的元素数据类可以是抽象元素数据父类，根据增强现实交互场景中用户的交互动作确定出该交互动作所对应的界面元素(例如，某个虚拟按钮)，将抽象界面元素父类具象化为该交互动作所影响的界面元素得到具体界面元素类，并且，将抽象元素数据父类具象化为该交互动作所影响的界面元素在后台的数据(即渲染该界面元素所需的元素数据)得到具体元素数据类。通过将该具体界面元素类与预设的绑定元素工具相绑定，将该具体元素数据类与预设的绑定元素属性相绑定，建立该绑定元素工具与该绑定元素属性之间的对应关系，通过改变该绑定元素属性以及该绑定元素工具，来调节该具体界面元素类以及具体元素数据类，以实现在增强现实交互场景中对用户的交互动作所影响的界面元素以及渲染界面元素所需的元素数据进行绑定。
通用可扩展业务模块102，用于预先保存所述增强现实设备所执行的业务对应的容器，并在执行目标业务时，调用所述目标业务对应容器中的组件，以基于调用出的组件，执行目标业务；通过预设的各个组件，创建新的待扩展业务，并管理所述待扩展业务的创建状态。
具体可以是增强现实设备首先获取用户发送的新增业务需求指令或者服务器发送的新增业务需求指令，之后再响应于该新增业务需求指令，确定待扩展业务，通过指定容器，将待扩展业务实例化，根据该实例化的待扩展业务，更新该增强现实设备中的可执行业务。
识别追踪模块103，用于对所述增强现实设备采集到的图像数据进行图像识别，以确定出所述用户所处的增强现实交互场景中目标对象的位姿。
该增强现实设备具体可以根据该设备所装有的摄像机采集到的图像(包括彩色图像以及深度图像)，通过预置的识别模型识别出目标对象具体是什么，以及识别出目标对象在该图像中的初始位姿(位姿包括位置和方向)，并且基于该图像、初始位姿以及该设备中所装有的摄像机的内外矩阵的参数，来确定出该用户所处的增强现实交互场景中目标对象的实际位姿。
多人协同模块104，用于响应数据同步指令，并向其他增强现实设备发送本地设备所产生的数据同步指令，所述数据同步指令用于对所述增强现实交互场景中的目标对象进行控制或对目标对象的状态进行更新。
该增强现实设备可以基于该多人协同模块，完成进行各个用户之间的信息实时信息交互。
混合网络通信模块105，用于将所述增强现实交互场景中产生不同类型的待传输数据确定传输策略，并分别按照不同的传输策略，将不同类型的待传输数据进行传输。
该增强现实设备可以基于该混合网络通信模块，为待传输数据分配对应的匹配的传输策略，以实现高效的数据传输。
并且，所述数据驱动交互界面模块101，还用于根据后端的与所述增强现实设备的显示界面中的界面元素所绑定的数据的更新情况来实时更新所述界面中的元素。
此外，增强现实设备还可以包括：启动器106、空间定位模块107、数据存储模块108、状态管理模块109。启动器106，用于启动该增强现实设备，空间定位模块107，用于确定该增强现实设备的空间定位信息，数据存储模块108，用于存储所述增强显示设备执行业务过程中所产生的数据，状态管理模块109，用于管理上述各个模块的状态。
基于上述如图1所示的增强现实设备，本公开还提供了相应的增强现实方法，如图2所示，图2为本公开中提供的一种增强现实方法的流程示意图，具体包括以下步骤S201-S204。
S201：基于接收到的用户所发送的增强现实交互场景创建请求，创建增强现实交互场景。
本公开实施例中，可由上述图1所示的增强现实设备执行图2所示的方法。
一旦接收到用户所发送的增强现实交互场景创建请求，增强现实设备则可以创建出与用户所发送的增强现实交互场景创建请求相对应的增强现实交互场景。
这里用户所发送的增强现实交互场景创建请求可以指用户触控了该增强现实设备界面中的某个表示请求增强现实交互场景的控件，也可以指用户发出了表示请求增强现实交互场景的语音，在本公开实施例中不对上述请求的触发条件做出具体限定。
而这里增强现实设备所创建的增强现实交互场景指的是：与用户所发出的增强现实交互场景创建请求对应的业务一致的场景。而上述增强现实交互场景可以是该用户通过该增强现实设备所观测到的场景。
例如，用户所处的真实环境中有一张真实的桌子，该桌子上有一个真实的杯子以及5个真实的杯盖，当用户向AR眼镜这种增强现实设备发送的增强现实交互场景创建请求表示用户所要执行的业务是“多人合作拧杯盖的游戏”，该AR眼镜则可以响应于该请求创建出该业务所对应的增强现实交互场景，该增强现实交互场景中包含有上述真实环境以及该AR眼镜所创建出的指导用户完成游戏的虚拟的箭头，该箭头可以指向与该杯子最匹配的杯盖。
后续，用户可以在该增强现实设备所创建的增强现实交互场景中与其他用户进行互动。
S202：根据为界面元素预设的元素变量，和为元素数据预设的数据变量，针对每个界面元素，将该界面元素的元素变量与渲染该界面元素所需的元素数据的数据变量相绑定，实现该界面元素与该界面元素对应的元素数据的绑定，以使通过定义的指定对象类，在增强现实交互场景中对用户的交互动作所影响的界面元素以及渲染界面元素所需的元素数据进行绑定，其中，所述指定对象类包含有界面元素对应的对象类、界面元素数据对应的对象类。
增强现实设备会为该增强现实设备的显示界面中的各个界面元素预设元素变量，并且针对上述每个界面元素，将该界面元素的元素变量与渲染该界面元素所需的元素数据的数据变量相绑定，以实现该界面元素与该界面元素对应的元素数据的绑定，并且通过预先定义的指定对象类，在增强现实交互场景中将被该用户的交互动作所影响的界面元素以及渲染该界面元素所需的元素数据(即界面元素在后台所对应的数据)进行绑定。
这里提到的指定对象类可以包括：抽象界面元素父类(即上述界面元素对应的类)、抽象元素数据父类(即上述与该界面元素绑定的元素数据类)。
抽象界面元素父类中包含有多个不同的界面元素，抽象元素数据父类中包含有多个不同的界面元素在后台所对应的数据。
界面元素指的是在增强现实设备的展示界面所展示出的元素(如交互界面中的按钮、图表、三维模型等元素)。元素数据(界面元素在后台所对应的数据)指的是上述界面元素在后台所对应的数据表示(如该按钮的触发信号数据、该图表的值数据，该三维模型的位姿数据等)。
增强现实设备会预先为被该用户的交互动作所影响的界面元素预设有相应的绑定元素工具，例如变量A，并且预先为被该用户的交互动作所影响的界面元素在后台的数据也预设有相应的绑定元素属性，例如变量B，并且，增强现实设备还建立该绑定元素工具与该绑定元素属性之间的对应关系。
具体来说，可以是增强现实设备一旦监测到该用户在增强现实交互场景中所发出的 交互动作，则可以确定出被该交互动作所影响的界面元素，以及确定出被该用户的交互动作所影响的界面元素在后台的数据。并且，将该抽象界面元素父类具体化为被该交互动作所影响的界面元素，即得到具体界面元素类，将该抽象元素数据父类具体化为该被该用户的交互动作所影响的界面元素在后台的数据，即得到具体元素数据类。
之后，还可以将该具体界面元素类与预设的绑定元素工具相绑定，将该具体元素数据类与预设的绑定元素属性相绑定。如此，在增强现实设备监测到该用户在增强现实交互场景中所发出的交互动作之后，则可以将被该交互动作影响后的界面元素确定为绑定元素工具，并且根据绑定元素工具与绑定元素属性之间的对应关系，确定与该绑定元素工具对应的绑定元素属性为被该交互动作影响后的界面元素所对应的后台数据，即该绑定元素工具所对应的具体元素数据类。
例如，当用户在增强现实交互场景中将虚拟的某个三角按钮做出了向上平移的交互动作，那么被该交互动作影响后的界面元素则是该三角按钮，即具体界面元素类是该三角按钮，由于与该界面元素类相绑定的是绑定元素工具变量A，那么此时变量A对应的该三角按钮也进行了该向上平移的交互动作，由于该绑定元素工具变量A对应的绑定元素属性为变量B，那么此时变量B所对应的数据中的坐标也会相应的向上平移，而由于绑定元素属性变量B是与该三角按钮对应的具体元素数据类相绑定的，那么此时该三角按钮对应的具体元素数据类所对应的数据中的坐标也会相应的向上平移。
此外，还可以根据后端的与该增强现实设备的显示界面中的界面元素所绑定的数据的更新情况来实时更新该界面中的元素，具体更新过程，本公开中不再赘述。
如此，则实现了将用户的增强型现实设备中所展示的界面元素与该界面元素所对应的后端数据进行动态绑定。
S203：当监测到用户执行的交互动作时，确定用户在所述增强现实设备中所要展示的增强现实交互场景，并调用所述增强现实交互场景中所需的业务场景数据以及在不同增强现实交互场景中所使用的通用数据，以在所述增强现实设备的显示界面中展示用户所要展示的虚拟界面对象，并确定出所述用户所处的增强现实交互场景中目标对象的位姿。
其中，这里的增强现实交互场景中所需的业务场景数据指的是特定的业务场景所需要的数据，可以包括渲染某个具体的增强现实业务场景对应的数据，也可以包括同一个虚拟物体在不同的增强现实交互场景中所呈现出来的不同的状态的部分对应的数据。
这里的交互动作用于表示用户基于该增强现实设备所发出的执行用户所选定的业务的动作。
用户可以触控用于创建虚拟物体的控件，这个触控的动作则可以作为交互动作。而由于同一个虚拟物体在不同的增强现实交互场景中所呈现出来的状态可能会在某一部分有所不同，但是其他部分是相同的。所以，为了提高虚拟物体的创建效率，增强现实设备可以调用该增强现实交互场景中所需的业务场景数据(即同一个虚拟物体在不同的增强现实交互场景中所呈现出来的不同的状态的部分对应的数据)以及在不同增强现实交互场景中所使用的通用数据(即同一个虚拟物体在不同的增强现实交互场景中所呈现出来的相同的状态的部分对应的数据)，以针对性的在该增强现实设备的显示界面中展示用户所要展示的虚拟界面对象，并确定出该用户所处的增强现实交互场景中目标对象的位姿。
例如，在创建长发女孩的虚拟物体时，由于长发女孩的头发在陆地上向下垂，而在太空中会漂浮，但是同一个长发女孩这个虚拟物体在这两种增强现实交互场景中的除了头发以外的其他部位的表现是一致的，那么在太空这种增强现实交互场景中创建该长发女孩时，创建头发所需要的数据是上述提及的业务场景数据，而创建该长发女孩的其他 部位所需要的数据则是上述提及的通用数据。
具体来说，一旦增强现实设备监测到该用户执行了交互动作，该增强现实设备则可以确定在该增强现实设备的显示界面中所要展示的虚拟界面对象，之后，该增强显示设备可以结合确定用户在该增强现实设备中所要展示的增强现实交互场景，调用该增强现实交互场景中所需的业务场景数据以及在不同增强现实交互场景中所使用的通用数据，并调用预设的对象容器中可以用于创建在该增强现实交互场景下的该虚拟界面对象的函数来创建出该虚拟界面对象，来在该增强现实设备的显示界面中展示用户所要展示的虚拟界面对象。
此外，由于用户在使用增强现实设备执行业务时，会涉及到用户对增强现实设备显示界面所显示的目标对象的操作，那么此时则有必要对该目标对象进行追踪，以使得用户可以顺利完成对增强现实设备显示界面所显示的目标对象或者虚拟界面对象的操作，那么此时增强现实设备则需要确定出用户所处的增强现实交互场景中目标对象的位姿。
这里的目标对象表示用户所执行的业务所对应的需要操作的对象。
继续沿用上例，在用户通过增强现实设备执行“多人合作拧杯盖的游戏”的业务时，由于用户所选择的业务是为杯子找到匹配的杯盖并将该杯盖拧到该杯子上面，那么此时的目标对象则是该杯子以及与该杯子所匹配的某一个杯盖，而这里的指导用户完成游戏的虚拟的箭头则是增强现实设备的显示界面中所要展示的虚拟界面对象。
而在增强现实设备确定该用户所处的增强现实交互场景中目标对象的位姿时，具体可以是增强现实设备通过该设备中所装有的摄像机来采集该用户面前的该增强现实设备为该用户创建的增强现实场景的图像。其中，这里的图像可以包含该图像的RGB(Red,Grean,Blue)图像，即彩色图像，以及该图像的深度图像。并且，还可以确定该图像所对应的图像数据，这里的图像数据可以包括：该图像的图像数据以及该图像的深度数据，以及根据预置的识别模型来识别目标对象在该图像中的像素坐标系下的位姿作为初始位姿，并且，这里的预置的识别模型还可以识别出该目标对象具体是什么，上述提到的位姿包括位置和方向。
之后，该增强现实设备可以根据该初始位姿，基于该图像对应的RGB图像以及深度图像，上述提到的图像数据以及该设备中所装有的摄像机的内外矩阵的参数，来确定出该用户所处的增强现实交互场景中目标对象的实际位姿。
具体的，是通过进行一系列的空间坐标变换，以确定出该用户所处的增强现实交互场景中目标对象的实际位姿。即，首先基于该深度图像和RGB图像进行像素级的一一匹配，以将该初始位姿所位于的像素坐标系转换为图像坐标系，同时初始位姿也转换为图像坐标系下的初始位姿，此后，该增强现实设备可以将该图像坐标系转换为以该用户视角下的相机(该用户视角下的相机与该设备中所装有的摄像机位置统一)为零位的相机坐标系，并且将该图像坐标系下的初始位姿转换为该相机坐标系下的初始位姿。之后，还可以基于该设备中所装有的摄像机的内外矩阵的参数，来将相机坐标系转换为世界坐标系，并且将上述该相机坐标系下的初始位姿转换为世界坐标系下的初始位姿，这里的世界坐标系下的初始位姿则是上述提到的该用户所处的增强现实交互场景中目标对象的实际位姿。
其中，这里预置的识别模型具体可以是按照以下思路所构建与训练的：
为实现基于RGB图像的目标对象识别，可以先构建目标识别模型框架：首先，为每个对象设置一个预定义的比例范围和中心位置，并为每个对象指定特征金字塔网络(feature pyramid networks，FPN)，当进行计算机视觉任务时，例如目标检测和图像分类，需要考虑目标在不同尺度下的大小和形状。这是因为相同类型的物体可能在图像中以不同的大小和比例出现。因此，对于模型来说，必须能够识别和定位不同尺度的目标， 而不仅仅是某一特定尺度上的对象。
特征金字塔网络(FPN)通过构建多个不同分辨率的特征图来解决这个问题。它使用一个共享权重的卷积神经网络(CNN)来从输入图像中提取特征。然后，对于每个不同的尺度，它都会通过增加或减小步幅来调整原始图像的大小。在每个尺度上，FPN都会生成一组由高到低的特征图，这些特征图被称为“金字塔”(pyramid)，其中，最底层是原始图像上采样得到的特征图，而最高层是通过应用池化操作和卷积操作得到的粗略特征图。
这些特征映射然后级联起来，以产生更高级别的特征表示。这种级联过程使得模型能够捕获不同尺度下的目标信息，并且在深度上更加丰富，从而提高了模型的性能。对于每个位置，预测四个值：网格左上角的两个偏移量以及预测框的高度和宽度(像素表示)。为了更准确地检测对象类型，融合了上述FPN从不同网络层获得的多尺度特征。该网络由两个子网模块组成，第一个是自上而下的架构(自下而上网络从原始图像开始，通过多个卷积层和池化层逐步缩小图像尺寸，得到一系列由粗到细的特征图。这些特征图在通道数上逐渐增加，捕获了越来越高级别的特征。自下而上网络主要负责提取低级别的特征，例如边缘和纹理等)，第二个是自下而上的架构(自上而下网络则是从金字塔的顶部开始，通过上采样或反卷积操作逐步恢复特征图的大小，同时在每一层都添加横向连接。横向连接了自上而下网络与自下而上网络的相应层，以便自上而下网络可以获取来自自下而上网络的高级别特征信息。自上而下网络主要负责提取高级别的特征，例如语义信息和物体的整体形状)。每个子网络都使用深度残差网络(Residual Network，ResNet)块构建深层网络，这些残差块包含跨越层的跳跃连接。跳跃连接允许在从输入到输出之间的层中跳过一些层，使得网络可以捕获更高级别的特征，并且更容易训练。同时，跳跃连接还可以将梯度传递回较早的层，缓解了梯度消失问题。RGB图像输入自下而上的子网络，然后将特征图传输到下一个子网络。在自上而下的子网络中，每一层的每个特征图都被组合起来以预测对象类别和相应的候选边界框。后续可以基于目标识别模型框架，通过采集大量目标对象的图像进行训练，得到深度学习识别模型。
继续沿用上例，该用户的AR眼镜中的摄像机可以采集到用户的视角所看到的该增强现实交互场景所对应的图像(包括RGB图像以及深度图像)，那么此时增强现实设备可以确定出该图像所对应的图像数据，该图像数据可以包含用户在执行“多人合作拧杯盖”业务时该摄像机所采集到的RGB图像的图像数据以及与该图像所匹配的深度图像的深度数据，并且，增强现实设备可以根据预置的识别模型识别出该目标对象杯盖在该图像的像素坐标系下的位姿作为初始位姿，之后，可以根据该初始位姿，基于该图像对应的RGB图像、深度图像、上述提到的图像数据以及该设备中所装有的摄像机的内外矩阵的参数，通过进行一系列的空间坐标变换，确定出该用户所处的增强现实交互场景中目标对象所在的实际位姿。
此外，当监测到用户需要在该增强显示设备中新增业务，例如当监测到用户想在该增强显示设备中使用语音识别业务，但是该增强显示设备中没有语音识别业务，那么此时将用户需要新增的业务作为待扩展业务，并将待扩展业务实例化，之后，将得到的实例化业务调取到预设的IOC(Inversion of Control)容器中,并管理该待扩展业务在所述交互界面中的行为动作，以及调用该实例化后的该待扩展业务，以更新该增强现实设备中的可执行业务。
图3为本公开中提供的一种增强现实设备的待扩展业务的创建过程的示意图；
如图3所示，当用户触碰了增强显示设备的显示界面中的某一个按钮，该按钮用于表示要在该增强显示设备中新增一个业务A以及新增一个业务B，业务A以及业务B即是待扩展业务，则激活了该按钮对应的元素数据，该增强显示设备中的控制组件的预设接口会响应于预设的元素数据的变化。
之后，用于管理不同类型的业务的业务上下文组件可以通过由控制组件在进行IOC依赖注入后返回获取到用户所确定的待扩展业务，并且业务上下文组件可以根据上述用户所确定的待扩展业务派生出A业务的业务上下文组件以及B业务的业务上下文组件。
用于具体进行业务实例化的业务实例组件可以依赖于上述A业务的业务上下文组件以及B业务的业务上下文组件，通过业务组件派生出A业务的业务实例组件以及B业务的业务实例组件，以及确定出A业务中包含的待实例化任务A1,待实例化任务A2以及待实例化任务A3，并且确定出B业务中包含的待实例化任务B1,待实例化任务B2。之后，可以将上述所有待实例化的任务利用依赖注入的方式注入到IOC容器中，并且在IOC容器将上述所有待实例化的任务进行实例化，以实现上述待扩展业务的实例化，即，此时IOC容器中会包含已经实例化的A业务以及B业务，上述提到的实例化可以是将上述任务对应的函数进行具体的实现。此后，预设的第一控制组件会从预设的IOC容器中调取该业务A以及业务B，以实现更新该增强现实设备中的可执行业务。
S204：将所述位姿以及所述用户执行的交互动作所对应的同步指令发送给其他增强现实设备，以使得其他增强现实设备根据所述位姿以及同步指令来同步该增强现实设备所呈现的增强现实交互场景。
用户在使用增强现实设备执行业务时，可能会涉及到与其他用户之间的信息交互，即可能会涉及到多个用户对同一个目标对象或者虚拟界面对象执行操作，或者用户对彼此所创建的虚拟界面对象所执行的操作或者其他交互操作。
那么此时为了保证用户对增强现实设备的使用体验，以顺利的使得各个用户的增强现实设备中的交互界面中可以及时的将这些操作所带来的效果展示给所有上述一起执行该业务的用户，增强现实设备有必要对这些操作所带来的效果进行各个用户之间的实时共享。
例如，当共有5个用户每个人持有一个增强现实设备，在每个增强现实设备的增强现实交互场景中，每个用户的上线下线的状态可以通过该用户在该增强现实交互场景中的虚拟人物形象来表示，即，若该用户在线，那么该增强现实交互场景中则会出现该用户的虚拟人物形象，否则，该增强现实交互场景中不会出现该用户的虚拟人物形象。
那么当5个用户中从5个用户都在线变化为第一个用户下线，即第一个用户通过该用户持有的增强现实设备执行了下线的交互操作，此时，这第一个用户的虚拟人物形象需要消失则是上述提到的与该操作对应的需要同步发送给其他增强现实设备的同步指令，同步完成之后，每个用户的增强现实设备中这第一个用户的虚拟人物形象都会消失。
又例如，当共有5个用户每个人持有一个增强现实设备，都进入了合作投球的游戏，5个用户每人都有1个虚拟小球，5人轮流向现实的眼前的盒子中投入虚拟小球，若第一个用户执行了向该盒子中投进了1个虚拟小球的交互动作，那么该第一个用户的增强现实设备则会将这个虚拟小球的位姿以及这个交互动作所对应的同步指令发送给其他增强现实设备，以使得其他增强现实设备根据该位姿以及同步指令来同步该增强现实设备所呈现的增强现实交互场景。
此外，由于增强现实设备也可能会响应与服务器所发送的数据更新指令，对该增强现实设备中的显示界面的场景或者按钮或者其他组件进行更新，那么同步指令则可以是有关上述内容的更新的指令。例如，当服务器需要在该增强现实设备中的显示界面中模拟大自然中太阳的东升西落的场景，那么一旦该增强现实设备接收到关于更新上述显示界面中太阳的位置、色彩等信息的同步指令，则可以响应于该同步指令，更新上述显示界面中太阳的位置、色彩等信息，这里的太阳则可以是上述提到的需要进行更新的指定对象。
上述提到的增强现实设备与其他增强现实设备之间的信息同步以及信息交互或者 增强现实设备与服务器之间的信息同步以及信息交互的过程具体可以如图4所示。
图4为本公开中提供的一种信息交互的流程示意图。
如图4所示，图4中的协同组件负责对于服务器及客户端以及多个网络连接的管理。服务端是信息交互中多个客户端的中间节点，负责管理客户端的添加、删除、行为响应、数据传递等行为，这里的服务端既可以是增强显示设备，也可以是现实中的服务器。客户端是增强显示设备，客户端用于管理当前用户的行为传输，以及其他同步指令数据在该客户端本地的同步。图4中的“网络连接”的管理组件，包含两个派生组件：分别是网络连接的服务器的管理组件(对应着与服务器的网络连接)以及网络连接的服务器的客户端管理组件(对应着与客户端的网络连接)，在信息交互过程中，需要对每个角色进行唯一性标记，便于在数据同步中进行区分，所有就需要协同身份标识组件来为各个同步指令确定唯一的身份标识。此外，行为管理组件由上述服务器及客户端进行管理，并与“网络连接”的管理组件进行关联，行为管理组件负责所有角色的行为管理，行为管理组件通过由协同身份标识组件管理，行为管理组件可以根据协同身份标识组件中的信息，来明确确定同步指令的发出者以及接收者，以实现稳定的信息交互，变换组件用于网络通信的接口管理，实现信息交互以及后续数据传输的部分的信息连接。
上述提到的增强现实设备交互数据的实时共享的过程具体可以如图5所示。
图5为本公开中提供的一种增强现实设备交互数据的实时共享的流程示意图。
图5中的客户端指该用户所持有的增强现实设备，服务端指的是服务器或者被指定为服务端的某个增强现实设备。客户端用于管理当前用户的行为传输，以及其他客户协同信息数据在本地的同步，服务端是多人协同中多个客户端的中间节点，负责管理客户端的添加、删除、行为响应、数据传递等行为。图5中的其他客户端指的是除该用户之外的其他用户所持有的已经与服务端建立了信息连接的增强现实设备，合作组件器用于管理于服务端及客户端的管理。
用户所持有的客户端可以向服务端发送携带有创建客户端对象的连接请求，这里的客户端对象可以用于表示用户想在该客户端所对应的增强现实设备的显示界面中所要展示的虚拟界面对象。
服务端一旦收到该数据共享请求，则可以通过预设的回调函数来触发在服务端中创建该客户端对象的指令，合作组件器可以根据该回调函数，在服务端上的增强现实交互场景中创建该客户端对象，并对创建出的客户端对象做出该客户端的标记。
之后，服务端可以将该客户端标记为需要协同的信息(即同步变量)，并将该同步变量进行序列化(即将该同步变量转换为二进制数据的形式)，服务端可以通过合作组件器将携带有序列化后的同步变量的客户端对象的信息发送给其他客户端。
同时，合作组件器可以将客户端对象以及该客户端的标记(即图5中的主体隶属信息)发送给该客户端，之后，该客户端可以为该服务端赋予在该客户端的增强现实场景中创建该客户端对象的权限，并在该客户端的增强现实场景中呈现出该客户端对象。
其他客户端在接收到该携带有序列化后的同步变量的客户端对象的信息之后，可以对该信息进行解析，将该携带有序列化后的同步变量的客户端对象的信息进行反序列化(即将该二进制数据转换为同步变量的形式)，以获取该客户端的信息以及该客户端所创建的客户端对象，并在其他客户端所创建的增强现实场景中同步创建该客户端对象。
而若该用户在该客户端中对客户端对象执行了操作，那么同理，通过上述步骤，其他客户端也可以接收到该用户对客户端对象所执行的操作，并在其他客户端的增强现实交互场景中同步更新该客户端的状态，以实现上述同步共享的目的。
继续沿用上例中的“多人合作拧杯盖”业务，用户在执行上例中的“多人合作拧杯 盖”时，则可能会涉及到多个用户依次对目标对象杯盖进行一些操作，那么增强现实设备则可以依次获取到每个用户所对应的对于该杯盖所发出的同步指令，例如该游戏共有3个用户(用户A、用户B、用户C)参加，用户A首先发出将该杯盖向左挪动3厘米的同步指令，那么此时增强现实设备的需要将该同步指令发送给其他增强现实设备，以使得其他增强现实设备根据用户A挪动后的杯盖的位姿来同步这些其他增强现实设备所呈现的增强现实交互场景，即用户A在将该杯盖向左挪动3厘米后，在经过上述同步之后，用户B、用户C的增强现实设备所识别出的该杯盖的位姿也是在原来的杯盖位姿基础上向左挪动3厘米。
而在具体的同步过程中，增强现实设备在传输数据时可以根据预设的该场景数据所对应的数据传输策略以及数据传输方式来将该场景数据同步发送给其他用户所持有的增强现实设备。其中，这里的数据传输策略也可以时由用户所指定的，例如，用户可以指定增强现实设备在传输视频时传输速度为30帧/秒。
上述具体的数据传输过程可以如图6所示，图6为本公开中提供的一种增强现实设备传输数据的流程示意图。
从图6可以看出，增强现实设备传输数据这一步骤是通过混合网络通信模块完成的，混合网络通信模块主要由物理层、传输层、策略层、数据层、多通道通信管控五层逻辑构成。
数据层包含有需要进行通信的数据，例如应用基础数据、交互数据、状态数据、场景数据、虚实融合数据、采集数据、分析评估数据等增强现实应用数据，数据层主要用于对需要进行通信的数据进行分类、标签，标签分为数据策略S以及数据传输方式T。
策略层分为高频实时传输策略、低频在线传输策略、信号触发传输策略三种；其中高频实时传输策略面向传输数据实时性要求较高的通信需求，传输的频率采用预定义统一频率，传输的数据量由具体网络情况进行限制优化。低频在线传输策略面向数据更新频次要求较低的场景需求，可以进行大容量数据的传递，传输的频率采用预定义统一频率。信号触发传输策略面向的是用户的业务需求，可以通过内部流程或外部输入进行触发，传输的数据量不做具体限制。
传输层分为轻量级通信、异步大容量通信以及流数据通信三种传输方式。轻量级通信适用于小数据量数据通信需求，基于基本的Socket通信方式完成。流数据通信适用于实时的音视频通信需求，实时性要求较高，通常基于网页实时通信(Web Real-Time Communication，WebRTC)实现。异步大容量通信适用于数据量较大，且对传输速度、传输质量有要求的通信情形。
多通道通信管控层用于对于传输数据的策略以及具体方式进行判断、融合、决策及校验。主要包含融合网络构建(业务需求中同时需要多种数据策略或传输方式融合的情况)、协议适配(不同数据传输接口的协议兼容)、数据分类及标签添加、数据序列化及反序列化、通信会话构建(例如构建聊天对话框)、用于提高数据传输效率以及稳定性的并发线程管理以及数据信息校验(用于校验传输的数据的安全性)等内容。
从上述方法中可以看出，通过为增强现实设备部署包含多个可以互相进行协作的模块，使得使用增强现实设备的用户能够使用增强现实设备执行完成虚拟与现实的交互，并且该用户在执行该交互业务时所做出的操作可同步给其他用户，以使得该用户能够与其他用户共享该虚拟与现实的交互。
以上为本公开的增强现实设备以及一个或多个实施的方法，基于同样的思路，本公开还提供了相应的增强现实的装置，如图7所示。
图7为本公开提供的一种增强现实的装置的示意图，包括：
创建模块701，用于基于接收到的用户所发送的增强现实交互场景创建请求，创建 增强现实交互场景；
绑定模块702，用于根据为界面元素预设的元素变量，和为元素数据预设的数据变量，针对每个界面元素，将该界面元素的元素变量与渲染该界面元素所需的元素数据的数据变量相绑定，实现该界面元素与该界面元素对应的元素数据的绑定，以使通过定义的指定对象类，在增强现实交互场景中对用户的交互动作所影响的界面元素以及渲染界面元素所需的元素数据进行绑定，其中，所述指定对象类包含有界面元素对应的对象类、界面元素数据对应的对象类；
确定与展示模块703，用于当监测到所述用户执行的交互动作时，确定所述用户在所述增强现实设备中所要展示的增强现实交互场景，并调用所述增强现实交互场景中所需的业务场景数据以及在不同增强现实交互场景中所使用的通用数据，以在所述增强现实设备的显示界面中展示用户所要展示的虚拟界面对象，并确定出所述用户所处的增强现实交互场景中目标对象的位姿；
同步模块704，用于将所述位姿以及所述用户执行的交互动作所对应的同步指令发送给其他增强现实设备，以使得其他增强现实设备根据所述位姿以及同步指令来同步该增强现实设备所呈现的增强现实交互场景。
可选地，所述装置还包括：
更新模块705，用于根据后端的与所述增强现实设备的显示界面中的界面元素所绑定的数据的更新情况来实时更新所述界面中的元素。
可选地，所述确定与展示模块703用于，采集所述增强现实场景的图像数据；根据预设的识别模型识别所述目标对象在所述图像数据所表示的图像中的位姿作为初始位姿；根据所述初始位姿以及所述图像数据，确定出所述用户所处的增强现实交互场景中目标对象的位姿。
可选地，所述同步模块704用于，确定所述位姿以及所述用户执行的交互动作所对应的同步指令在所述增强现实交互场景中所对应的场景数据；将所述场景数据传输给其他增强现实设备。
可选地，所述同步模块704还用于，获取用户所确定的数据传输策略；根据所述用户所确定的数据传输策略，将所述场景数据传输给其他增强现实设备。
可选地，所述装置还包括：
业务拓展模块706，用于获取用户发送的新增业务需求指令；响应于所述新增业务需求指令，确定待扩展业务，通过指定容器，将待扩展业务实例化，根据所述实例化的待扩展业务，更新所述增强现实设备中的可执行业务。
本公开还提供了一种计算机可读存储介质，该存储介质存储有计算机程序，计算机程序可用于执行上述图1提供的一种增强现实的方法。
本公开还提供了图8所示的一种对应于图1的电子设备的示意结构图。如图8所示，在硬件层面，该电子设备包括处理器、内部总线、网络接口、内存以及非易失性存储器，当然还可能包括其他业务所需要的硬件。处理器从非易失性存储器中读取对应的计算机程序到内存中然后运行，以实现上述图2所述的增强现实的方法。
当然，除了软件实现方式之外，本公开并不排除其他实现方式，比如逻辑器件抑或软硬件结合的方式等等，也就是说以下处理流程的执行主体并不限定于各个逻辑单元，也可以是硬件或逻辑器件。
在20世纪90年代，对于一个技术的改进可以很明显地区分是硬件上的改进(例如，对二极管、晶体管、开关等电路结构的改进)还是软件上的改进(对于方法流程的改进)。 然而，随着技术的发展，当今的很多方法流程的改进已经可以视为硬件电路结构的直接改进。设计人员几乎都通过将改进的方法流程编程到硬件电路中来得到相应的硬件电路结构。因此，不能说一个方法流程的改进就不能用硬件实体模块来实现。例如，可编程逻辑器件(Programmable Logic Device,PLD)(例如现场可编程门阵列(Field Programmable Gate Array，FPGA))就是这样一种集成电路，其逻辑功能由用户对器件编程来确定。由设计人员自行编程来把一个数字系统“集成”在一片PLD上，而不需要请芯片制造厂商来设计和制作专用的集成电路芯片。而且，如今，取代手工地制作集成电路芯片，这种编程也多半改用“逻辑编译器(logic compiler)”软件来实现，它与程序开发撰写时所用的软件编译器相类似，而要编译之前的原始代码也得用特定的编程语言来撰写，此称之为硬件描述语言(Hardware Description Language，HDL)，而HDL也并非仅有一种，而是有许多种，如ABEL(Advanced Boolean Expression Language)、AHDL(Altera Hardware Description Language)、Confluence、CUPL(Cornell University Programming Language)、HDCal、JHDL(Java Hardware Description Language)、Lava、Lola、MyHDL、PALASM、RHDL(Ruby Hardware Description Language)等，目前最普遍使用的是VHDL(Very-High-Speed Integrated Circuit Hardware Description Language)与Verilog。本领域技术人员也应该清楚，只需要将方法流程用上述几种硬件描述语言稍作逻辑编程并编程到集成电路中，就可以很容易得到实现该逻辑方法流程的硬件电路。
控制器可以按任何适当的方式实现，例如，控制器可以采取例如微处理器或处理器以及存储可由该(微)处理器执行的计算机可读程序代码(例如软件或固件)的计算机可读介质、逻辑门、开关、专用集成电路(Application Specific Integrated Circuit，ASIC)、可编程逻辑控制器和嵌入微控制器的形式，控制器的例子包括但不限于以下微控制器：ARC 625D、Atmel AT91SAM、Microchip PIC18F26K20以及Silicone Labs C8051F320，存储器控制器还可以被实现为存储器的控制逻辑的一部分。本领域技术人员也知道，除了以纯计算机可读程序代码方式实现控制器以外，完全可以通过将方法步骤进行逻辑编程来使得控制器以逻辑门、开关、专用集成电路、可编程逻辑控制器和嵌入微控制器等的形式来实现相同功能。因此这种控制器可以被认为是一种硬件部件，而对其内包括的用于实现各种功能的装置也可以视为硬件部件内的结构。或者甚至，可以将用于实现各种功能的装置视为既可以是实现方法的软件模块又可以是硬件部件内的结构。
上述实施例阐明的系统、装置、模块或单元，可以由计算机芯片或实体实现，或者由具有某种功能的产品来实现。一种典型的实现设备为计算机。具体的，计算机例如可以为个人计算机、膝上型计算机、蜂窝电话、相机电话、智能电话、个人数字助理、媒体播放器、导航设备、电子邮件设备、游戏控制台、平板计算机、可穿戴设备或者这些设备中的任何设备的组合。
为了描述的方便，描述以上装置时以功能分为各种单元分别描述。当然，在实施本公开时可以把各单元的功能在同一个或多个软件和/或硬件中实现。
本领域内的技术人员应明白，本公开的实施例可提供为方法、系统、或计算机程序产品。因此，本公开可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本公开可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。
本公开是参照根据本公开实施例的方法、设备(系统)、和计算机程序产品的流程图和/或方框图来描述的。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中 指定的功能的装置。
这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能。
这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步骤。
在一个典型的配置中，计算设备包括一个或多个处理器(CPU)、输入/输出接口、网络接口和内存。
内存可能包括计算机可读介质中的非永久性存储器，随机存取存储器(RAM)和/或非易失性内存等形式，如只读存储器(ROM)或闪存(flash RAM)。内存是计算机可读介质的示例。
计算机可读介质包括永久性和非永久性、可移动和非可移动媒体可以由任何方法或技术来实现信息存储。信息可以是计算机可读指令、数据结构、程序的模块或其他数据。计算机的存储介质的例子包括，但不限于相变内存(PRAM)、静态随机存取存储器(SRAM)、动态随机存取存储器(DRAM)、其他类型的随机存取存储器(RAM)、只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、快闪记忆体或其他内存技术、只读光盘只读存储器(CD-ROM)、数字多功能光盘(DVD)或其他光学存储、磁盒式磁带，磁带磁磁盘存储或其他磁性存储设备或任何其他非传输介质，可用于存储可以被计算设备访问的信息。按照本文中的界定，计算机可读介质不包括暂存电脑可读媒体(transitory media)，如调制的数据信号和载波。
还需要说明的是，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、商品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、商品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括所述要素的过程、方法、商品或者设备中还存在另外的相同要素。
本领域技术人员应明白，本公开的实施例可提供为方法、系统或计算机程序产品。因此，本公开可采用完全硬件实施例、完全软件实施例或结合软件和硬件方面的实施例的形式。而且，本公开可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。
本公开可以在由计算机执行的计算机可执行指令的一般上下文中描述，例如程序模块。一般地，程序模块包括执行特定任务或实现特定抽象数据类型的例程、程序、对象、组件、数据结构等等。也可以在分布式计算环境中实践本公开，在这些分布式计算环境中，由通过通信网络而被连接的远程处理设备来执行任务。在分布式计算环境中，程序模块可以位于包括存储设备在内的本地和远程计算机存储介质中。
本公开中的各个实施例均采用递进的方式描述，各个实施例之间相同相似的部分互相参见即可，每个实施例重点说明的都是与其他实施例的不同之处。尤其，对于系统实施例而言，由于其基本相似于方法实施例，所以描述的比较简单，相关之处参见方法实施例的部分说明即可。
以上所述仅为本公开的实施例而已，并不用于限制本公开。对于本领域技术人员来说，本公开可以有各种更改和变化。凡在本公开的精神和原理之内所作的任何修改、等 同替换、改进等，均应包含在本公开的权利要求范围之内。

Claims (15)

  1. 一种增强现实设备，其特征在于，所述设备中设有数据驱动交互界面模块、通用可扩展业务模块、识别追踪模块、多人协同模块、混合网络通信模块；
    所述数据驱动交互界面模块，用于针对与用户进行交互的交互界面中包含的每个界面元素，对该界面元素对应的类进行实例化得到第一实例化对象，并对预先与该界面元素绑定的元素数据类进行实例化得到第二实例化对象，并针对所述第一实例化对象中包含的每个指定变量，将该指定变量与所述第二实例化对象中包含的指定变量中与该指定变量名称相同的指定变量相绑定，以在所述第二实例化对象中与所述第一实例化对象绑定的任意一个指定变量的值改变时，所述第一实例化对象中与所述任意一个指定变量相绑定的指定变量对应的界面元素在所述交互界面中的呈现同步改变；
    所述通用可扩展业务模块，用于预先保存所述增强现实设备所执行的业务对应的容器，并在执行目标业务时，调用所述目标业务对应容器中的组件，以基于调用出的组件，执行所述目标业务；通过预设的各个组件，创建新的待扩展业务，并管理所述待扩展业务的创建状态；
    所述识别追踪模块，用于对所述增强现实设备采集到的图像数据进行图像识别，以确定出所述用户所处的增强现实交互场景中目标对象的位姿；
    所述多人协同模块，用于响应数据同步指令，并向其他增强现实设备发送本地设备所产生的数据同步指令，所述数据同步指令用于对所述增强现实交互场景中的目标对象进行控制或对目标对象的状态进行更新；
    所述混合网络通信模块，用于根据所述增强现实交互场景中产生不同类型的待传输数据确定传输策略，并分别按照不同的传输策略，将不同类型的待传输数据进行传输。
  2. 如权利要求1所述的增强现实设备，其特征在于，所述数据驱动交互界面模块还用于根据后端的与所述增强现实设备的显示界面中的界面元素所绑定的数据的更新情况来实时更新所述界面中的元素。
  3. 一种增强现实方法，其特征在于，所述方法应用于如权利要求1或2所述的增强现实设备，所述方法包括：
    基于接收到的用户所发送的增强现实交互场景创建请求，创建增强现实交互场景；
    根据为界面元素预设的元素变量，和为元素数据预设的数据变量，针对每个界面元素，将该界面元素的元素变量与渲染该界面元素所需的元素数据的数据变量相绑定，实现该界面元素与该界面元素对应的元素数据的绑定，以使通过定义的指定对象类，在增强现实交互场景中对用户的交互动作所影响的界面元素以及渲染界面元素所需的元素数据进行绑定，其中，所述指定对象类包含有界面元素对应的对象类、界面元素数据对应的对象类；
    当监测到所述用户执行的交互动作时，确定所述用户在所述增强现实设备中所要展示的增强现实交互场景，并调用所述增强现实交互场景中所需的业务场景数据以及在不同增强现实交互场景中所使用的通用数据，以在所述增强现实设备的显示界面中展示用户所要展示的虚拟界面对象，并确定出所述用户所处的增强现实交互场景中目标对象的位姿；
    将所述位姿以及所述用户执行的交互动作所对应的同步指令发送给其他增强现实设备，以使得其他增强现实设备根据所述位姿以及同步指令来同步该增强现实设备所呈现的增强现实交互场景。
  4. 如权利要求3所述的方法，其特征在于，所述方法还包括：根据后端的与所述增强现实设备的显示界面中的界面元素所绑定的数据的更新情况来实时更新所述界面中的元素。
  5. 如权利要求3所述的方法，其特征在于，确定出所述用户所处的增强现实交互场景中目标对象的位姿，包括：
    采集所述增强现实场景的图像数据；
    根据预设的识别模型识别所述目标对象在所述图像数据所表示的图像中的位姿作 为初始位姿；
    根据所述初始位姿以及所述图像数据，确定出所述用户所处的增强现实交互场景中目标对象的位姿。
  6. 如权利要求3所述的方法，其特征在于，将所述位姿以及所述用户执行的交互动作所对应的同步指令发送给其他增强现实设备，包括：
    确定所述位姿以及所述用户执行的交互动作所对应的同步指令在所述增强现实交互场景中所对应的场景数据；
    将所述场景数据传输给其他增强现实设备。
  7. 如权利要求6所述的方法，其特征在于，将所述场景数据传输给其他增强现实设备，所述方法还包括：
    获取用户所确定的数据传输策略；
    根据所述用户所确定的数据传输策略，将所述场景数据传输给其他增强现实设备。
  8. 如权利要求3所述的方法，其特征在于，所述方法还包括：
    获取用户发送的新增业务需求指令；
    响应于所述新增业务需求指令，确定待扩展业务，通过指定容器，将待扩展业务实例化，根据所述实例化的待扩展业务，更新所述增强现实设备中的可执行业务。
  9. 一种增强现实的装置，其特征在于，所述装置用于基于如权利要求1或2所述的增强现实设备实现增强现实方法，包括：
    创建模块，用于基于接收到的用户所发送的增强现实交互场景创建请求，创建增强现实交互场景；
    绑定模块，用于根据为界面元素预设的元素变量，和为元素数据预设的数据变量，针对每个界面元素，将该界面元素的元素变量与渲染该界面元素所需的元素数据的数据变量相绑定，实现该界面元素与该界面元素对应的元素数据的绑定，以使通过定义的指定对象类，在增强现实交互场景中对用户的交互动作所影响的界面元素以及渲染界面元素所需的元素数据进行绑定，其中，所述指定对象类包含有界面元素对应的对象类、界面元素数据对应的对象类；
    确定与展示模块，用于当监测到所述用户执行的交互动作时，确定所述用户在所述增强现实设备中所要展示的增强现实交互场景，并调用所述增强现实交互场景中所需的业务场景数据以及在不同增强现实交互场景中所使用的通用数据，以在所述增强现实设备的显示界面中展示用户所要展示的虚拟界面对象，并确定出所述用户所处的增强现实交互场景中目标对象的位姿；
    同步模块，用于将所述位姿以及所述用户执行的交互动作所对应的同步指令发送给其他增强现实设备，以使得其他增强现实设备根据所述位姿以及同步指令来同步该增强现实设备所呈现的增强现实交互场景。
  10. 如权利要求9所述的装置，其特征在于，所述装置还包括：
    更新模块，用于根据后端的与所述增强现实设备的显示界面中的界面元素所绑定的数据的更新情况来实时更新所述界面中的元素。
  11. 如权利要求9所述的装置，其特征在于，所述确定与展示模块用于，采集所述增强现实场景的图像数据；根据预设的识别模型识别所述目标对象在所述图像数据所表示的图像中的位姿作为初始位姿；根据所述初始位姿以及所述图像数据，确定出所述用户所处的增强现实交互场景中目标对象的位姿。
  12. 如权利要求9所述的装置，其特征在于，所述同步模块用于，确定所述位姿以及所述用户执行的交互动作所对应的同步指令在所述增强现实交互场景中所对应的场景数据；将所述场景数据传输给其他增强现实设备。
  13. 如权利要求12所述的装置，其特征在于，所述同步模块还用于，获取用户所确定的数据传输策略；根据所述用户所确定的数据传输策略，将所述场景数据传输给其他增强现实设备。
  14. 如权利要求9所述的装置，其特征在于，所述装置还包括：
    业务拓展模块，用于获取用户发送的新增业务需求指令；响应于所述新增业务需求指令，确定待扩展业务，通过指定容器，将待扩展业务实例化，根据所述实例化的待扩展业务，更新所述增强现实设备中的可执行业务。
  15. 一种计算机可读存储介质，其特征在于，所述存储介质存储有计算机程序，所述计算机程序被处理器执行时实现上述权利要求3～8任一项所述的方法。
PCT/CN2023/125666 2023-08-11 2023-10-20 一种增强现实的方法、装置、存储介质及电子设备 Pending WO2025035576A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202311010114.X 2023-08-11
CN202311010114.XA CN116755587B (zh) 2023-08-11 2023-08-11 一种增强现实的方法、装置、存储介质及电子设备

Publications (1)

Publication Number Publication Date
WO2025035576A1 true WO2025035576A1 (zh) 2025-02-20

Family

ID=87951759

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/125666 Pending WO2025035576A1 (zh) 2023-08-11 2023-10-20 一种增强现实的方法、装置、存储介质及电子设备

Country Status (2)

Country Link
CN (1) CN116755587B (zh)
WO (1) WO2025035576A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116755587B (zh) * 2023-08-11 2023-12-19 之江实验室 一种增强现实的方法、装置、存储介质及电子设备

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130178257A1 (en) * 2012-01-06 2013-07-11 Augaroo, Inc. System and method for interacting with virtual objects in augmented realities
CN106981000A (zh) * 2016-10-13 2017-07-25 阿里巴巴集团控股有限公司 基于增强现实的多人线下交互、点餐方法及系统
CN107111996A (zh) * 2014-11-11 2017-08-29 本特图像实验室有限责任公司 实时共享的增强现实体验
CN112346594A (zh) * 2020-10-27 2021-02-09 支付宝(杭州)信息技术有限公司 基于增强现实的互动方法及装置
CN114693890A (zh) * 2020-12-31 2022-07-01 华为技术有限公司 一种增强现实交互方法及电子设备
CN116755587A (zh) * 2023-08-11 2023-09-15 之江实验室 一种增强现实的方法、装置、存储介质及电子设备

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102945564A (zh) * 2012-10-16 2013-02-27 上海大学 基于视频透视式增强现实的真三维建模系统和方法
CN115359567A (zh) * 2014-06-14 2022-11-18 奇跃公司 用于产生虚拟和增强现实的方法和系统
CN104102545B (zh) * 2014-07-04 2017-12-01 北京理工大学 移动增强现实浏览器三维资源配置与加载的优化方法
CN107608649A (zh) * 2017-11-02 2018-01-19 泉州创景视迅数字科技有限公司 一种ar增强现实智能图像识别展示内容系统及使用方法
US10572016B2 (en) * 2018-03-06 2020-02-25 Microsoft Technology Licensing, Llc Spatialized haptic device force feedback
US20200125322A1 (en) * 2018-10-22 2020-04-23 Navitaire Llc Systems and methods for customization of augmented reality user interface
US11579757B2 (en) * 2020-09-30 2023-02-14 Snap Inc. Analyzing augmented reality content item usage data
CN114845020B (zh) * 2022-04-20 2023-02-24 深圳禾木信息技术有限公司 一种可扩展网络摄像机的控制方法及设备
CN115562497B (zh) * 2022-11-04 2024-04-05 浙江舜为科技有限公司 增强现实信息交互方法、增强现实设备和存储介质
CN116489149B (zh) * 2023-03-06 2025-09-23 太原理工大学 面向工业应用的云边端协同混合现实协作环境构建方法
CN116433835A (zh) * 2023-03-15 2023-07-14 杭州易现先进科技有限公司 三维增强现实操作系统的构造方法、装置和介质

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130178257A1 (en) * 2012-01-06 2013-07-11 Augaroo, Inc. System and method for interacting with virtual objects in augmented realities
CN107111996A (zh) * 2014-11-11 2017-08-29 本特图像实验室有限责任公司 实时共享的增强现实体验
CN106981000A (zh) * 2016-10-13 2017-07-25 阿里巴巴集团控股有限公司 基于增强现实的多人线下交互、点餐方法及系统
CN112346594A (zh) * 2020-10-27 2021-02-09 支付宝(杭州)信息技术有限公司 基于增强现实的互动方法及装置
CN114693890A (zh) * 2020-12-31 2022-07-01 华为技术有限公司 一种增强现实交互方法及电子设备
CN116755587A (zh) * 2023-08-11 2023-09-15 之江实验室 一种增强现实的方法、装置、存储介质及电子设备

Also Published As

Publication number Publication date
CN116755587A (zh) 2023-09-15
CN116755587B (zh) 2023-12-19

Similar Documents

Publication Publication Date Title
US10740978B2 (en) Surface aware lens
CN108520552A (zh) 图像处理方法、装置、存储介质及电子设备
CA3185810A1 (en) Systems and methods for augmented or mixed reality writing
US20120235899A1 (en) Apparatus, system, and method for controlling virtual object
KR20150079640A (ko) 실시간 통신 및 경험 공유 세션 동안 서비스를 업데이트하는 기법
KR102719382B1 (ko) 이미지 데이터 인코딩 방법 및 장치, 표시 방법 및 장치, 전자기기
CN109992111B (zh) 增强现实扩展方法和电子设备
CN114564101A (zh) 一种三维界面的控制方法和终端
WO2024066977A1 (zh) 基于掌部的人机交互方法、装置、设备、介质及程序产品
CN117710527B (zh) 基于人工智能大模型的图像处理方法、装置及产品
CN116755587B (zh) 一种增强现实的方法、装置、存储介质及电子设备
CN112379815A (zh) 图像截取方法、装置和存储介质及电子设备
CN115065863B (zh) 视频生成方法、装置、电子设备及存储介质
WO2023231553A1 (zh) 虚拟场景的道具交互方法、装置、电子设备、计算机可读存储介质及计算机程序产品
EP4170588B1 (en) Video photographing method and apparatus, and device and storage medium
CN118071893A (zh) 一种图形界面处理方法以及装置
CN113766295B (zh) 一种播放处理方法、装置、设备及存储介质
CN115713701A (zh) 一种基于HoloLens 2自处理的装配零部件识别方法
US20250329093A1 (en) Generation method and apparatus for image with three-dimensional effect, and electronic device and storage medium
CN117132479B (zh) 摩尔纹的消除方法、电子设备及可读存储介质
WO2024164652A1 (zh) 一种服务器、显示设备和虚拟数字人交互方法
CN120130087A (zh) 用于生成三维音频的多模态机器学习
CN116977504A (zh) 一种动画制作方法、装置、电子设备和存储介质
WO2023129162A1 (en) Real-time lightweight video tracking, processing, and rendering
CN112121418A (zh) 图像处理方法及装置、交互控制方法及装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23948998

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE