KR20190134513A - Two-way real-time 3d interactive operations of real-time 3d virtual objects within a real-time 3d virtual world representing the real world - Google Patents

Abstract

According to the present invention, described are a system enabling two-way interactive operations of real-time 3D virtual replicas and real objects and a method thereof. The system includes: a persistent virtual world system comprising a data structure which has at least one real-time 3D virtual replica of a real object represented therein and is stored and computed on a server; at least one corresponding real object connected to the real-time 3D virtual replica via a network through the persistent virtual world system stored and computed on the server; and at least one user device connected to the real object via the network through the virtual world system stored and computed on the server. Virtually selecting and effecting changes on the real-time 3D virtual replica results in a real-time corresponding effect on the real object. Likewise, effecting one or more changes on the real object results in a real-time corresponding effect on the real-time 3D virtual replica.

Description

TWO-WAY REAL-TIME 3D INTERACTIVE OPERATIONS OF REAL-TIME 3D VIRTUAL OBJECTS WITHIN A REAL-TIME 3D VIRTUAL WORLD REPRESENTING THE REAL WORLD }

Aspects of the present disclosure generally relate to computer systems, and more particularly to systems and methods that enable real-time 3D virtual replicas and two-way interactive operations of real objects.

Industries such as manufacturing, the military and automotive now benefit from control technology that utilizes control technology to enable monitoring along with passive and active management of the various objects involved. For example, various objects in a manufacturing process are usually managed remotely via a computer at a workstation. The workstation has recently been upgraded from a stationary computer to a mobile device that provides a more user-friendly and flexible human machine interface (HMI).

Nevertheless, most objects of interest may be displayed and manageable at the workstation, but interaction with such objects may still not be performed in a natural manner. For example, the user experience UX may include several buttons that allow the user to manipulate the object, which tends to be cumbersome when a large number of control buttons are included. In addition, changes to the actual object of interest may not be fully synchronized to include most of the details that may be needed for effective monitoring and management. Cooperation between the various elements is generally limited to a few objects, and in many cases requires a great deal of human interaction. In addition, digital reality technologies such as augmented reality and virtual reality, which provide opportunities for applications requiring control technology, are not fully utilized to facilitate efficient management.

Thus, there is a need to develop systems and methods that enable more synchronized communication and interaction between and with real objects.

This description is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This description is not intended to identify key features of the claimed subject matter, nor is it intended to assist in determining the scope of the claimed subject matter.

The present disclosure provides systems and methods that enable natural control and real-time 3D-based interactions between real and real objects through interfaces through accurate virtual replicas of real objects, and through virtual direct control of real objects. Provides synchronized control of replication. In the present disclosure description, the term “real object” means any physical object, device or machine that can be connected to a network and can be remotely controlled, modified or set up in some physical way. In some embodiments, the real object receives sensor information from a plurality of sources. In some embodiments, real objects may be communicatively connected with each other or other devices over a network in an Internet of Things (IOT) deployment, where such devices are referred to as IoT devices. Accurate virtual replicas may refer to real-time 3D virtual replicas of real objects, and may include real coordinates, including real positions and orientations, through the same or nearly identical physical properties and shared data synchronized with real objects. For example, direct manipulation of real-time 3D virtual replicas may be useful for remotely managing a plurality of industrial machines or vehicles in real time. Similarly, direct manipulation of industrial machines, for example, can be useful when a process manager or machine operator needs a model that is constantly updated in real time that displays and manages all objects in the system. Pairs of real objects and real-time 3D virtual replicas may be referred to herein as virtual tweens, virtual-real pairs, or real-virtual pairs. Further applications of the disclosed systems and methods may be enabled by real-time 3D virtual replicas provided through digital reality that may change the user's perception with respect to the real world, such as augmented reality, virtual reality or mixed reality. The systems and methods of the present disclosure may be suitable for managing operations at different levels of complexity, such as factories or homes, neighborhoods, cities, countries, and more. In other embodiments described in the detailed description of the invention and the following detailed description, further uses and advantages of the present disclosure may become apparent.

A system that enables bidirectional real-time 3D interactive manipulation of real objects and 3D real time virtual replicas of the present disclosure includes a data structure in which one real time 3D virtual replica of at least one corresponding real object is represented and stored on a server. A permanent virtual world system calculated; At least one corresponding real object communicatively and permanently connected to at least one real-time 3D virtual replica via a network via a permanent virtual world system stored and calculated on a server; And at least one user device communicatively and permanently connected to the one or more real objects via the network via a permanent virtual world system stored and calculated on a server. At least one real time 3D virtual replica is synchronized with at least one corresponding real object via a plurality of sensing mechanisms providing a plurality of data points shared between the at least one real object and the at least one real time 3D virtual replica. In addition, the virtual physical properties and virtual world coordinates of the at least one real-time 3D virtual replica correspond to the physical properties and the real coordinates of the corresponding one or more real objects. The plurality of sensing mechanisms may be a combination of IoT sensing mechanisms that provide a plurality of data points shared between virtual-physical pairs.

Validating one or more changes in at least one real-time 3D virtual replica or one of the corresponding real objects through a suitable interface allows the server to process manipulation commands in real-time or non-real-time to enable real-world management through the virtual world. . The server can then send these processed instructions to each target real object or each real time 3D virtual replica. More specifically, virtually selecting a real-time 3D virtual replica and then performing one or more changes to the real-time 3D virtual replica through at least one user device generates a corresponding effect on the corresponding real object in real time. Similarly, applying more than one change to a real object results in a real-time response to real-time 3D virtual copies.

According to one aspect of the present disclosure, the direction of manipulation of real objects and real-time 3D virtual replicas is bidirectional. For example, the manipulation may be generated as data and instructions generated from the user's interaction with the at least one real-time 3D virtual replica to control a corresponding real object or from interaction with the at least one object operator, At least one real object directly affects the corresponding real-time 3D virtual replica. However, as a result of artificial intelligence, group analysis, simulation, and context calculation, drones, autonomous vehicles, robots, urban buildings (eg, virtual building managers communicating with each other), mechanical and computer vision applications, personal assistants, video games, and the like. Likewise, interactions between real-time 3D virtual replicas can occur if cooperation between real objects is required. According to one embodiment, manipulating the at least one real-time 3D virtual replica or the corresponding at least one real object results in a change to the context data affecting the virtual-real pair, the change of the context data being at least one. It can affect the relationship between real-time 3D virtual replicas of the corresponding real objects. The term "context" or "context data" as used in this disclosure refers to data relating to the direct or indirect environment of a real-time 3D virtual replica and the corresponding real object, and includes other objects in the environment of the real object. In this disclosure, contexts can be further classified into micro-contexts and macro-contexts.

The term "micro context" refers to the surrounding context right next to a real object, such as any person, entity or condition that can directly affect real world elements. Micro-contexts can be used, for example, in 3D image data, 3D geometry, 3D entities, 3D sensor data, 3D dynamic objects, video data, audio Information such as data, text data, time data, metadata, priority data, security data, location data, lighting data, temperature data, and quality of service (QOS) may be included. The term "macro context" refers to an indirect or distant context surrounding a real object. Macro contexts can be derived by the server from a number of micro contexts, which represent many holistic aspects of the system, such as the current efficiency, air quality, climate change levels, company efficiency, traffic levels, urban efficiency, national efficiency, etc. Generate information. Macro contexts can be considered and calculated at various levels in accordance with designated machine learning functions and goals, including local level (eg, office or manufacturing plant), neighborhood level, city level, country level or even planet level. Thus, depending on the particular machine learning functions and goals, the same real world element data and micro context data can derive various types of macro contexts.

In some embodiments, the network may be, for example, a cellular network, and may include enhanced data rates for global evolution (EDGE), general packet radio service (GPRS), global system for mobile communications (GSM), Internet protocol multimedia subsystem, universal mobile telecommunications system (UMTS), and the like, and any other suitable wireless medium, such as microwave access (WiMAX), long term evolution (LTE) network, code division multiple access (CDMA), wideband code division multiple (WCDMA). Various technologies may be used, including access, wireless fidelity (WiFi), satellite, mobile ad-hoc networks (MANET), and the like. According to one embodiment, the network may include an antenna configured to transmit and receive radio waves that enable mobile communication between real objects and with a server. The antenna may be connected to the computing center via wired or wireless means. In another embodiment, the antenna is provided in a computing center and / or an area near the computing center. In some embodiments, to service user devices and / or real objects located outdoors, the antenna may be a millimeter wave (mmW) based antenna system or a combination of mmW based antenna and sub-6 GHz antenna system (herein referred to as 5G antenna). Referred to as taxonomy). In other embodiments, the antenna may include other types of antennas, such as 4G antennas, or may be used as support antennas for 5G antenna systems. In embodiments where an antenna is used to service a real-time 3D-based interactive device located indoors, the antenna may use Wi-Fi, which provides, but not limited to, data at 16 GHz.

In another embodiment, global navigation satellite systems (GNSS), collectively referred to as a plurality of satellite-based navigation systems, such as GPS, BDS, Glonass, QZSS, Galileo, and IRNSS, enable location of the device. Can be used to GNSS can use techniques such as trigonometry and triangulation and signals from a sufficient number of satellites to calculate the position, velocity, altitude and time of the device. In a preferred embodiment, the external positioning system is augmented by assisted GNSS (AGNSS) through the architecture of an existing cellular communication network, where the existing architecture includes 5G. In another embodiment, the AGNSS tracking system is further supported by the 4G cellular communication network. In indoor embodiments, the GNSS is further enhanced via radio wireless local area networks, such as but not limited to Wi-Fi, which preferably provides data at 16 GHz. In alternative embodiments, GNSS is augmented through other techniques known in the art through differential GPS (DGPS), satellite based augmentation system (SBAS), real-time kinematic (RTK) systems, and the like. In some embodiments, tracking of the device is implemented by a combination of AGNSS and inertial sensors in the device.

According to one embodiment, a server may be provided as hardware and software including at least a processor and memory, the processor may be configured to execute instructions contained in a memory coupled to the server, the memory storing instructions and data. It is configured to. For example, the processor can manage the corresponding real-time 3D virtual replica, simulation of real objects, 3D structure processing, context calculation, group analysis, management of at least one real object through rendering and virtual reinforcement of the real counterpart through real-time 3D virtual replica. Or may be configured to implement an artificial intelligence algorithm for the implementation of virtual compensation. In some embodiments, the processor also enables interactive interaction operations of real objects and real-time 3D virtual replicas by performing kinematic calculations on manipulation instructions to synchronize virtual and real pair movements. In one embodiment, the processing of manipulation instructions by the server processor is a complement to the processing performed by one or more real objects through its processor, which serves as support for the real objects to perform some heavy task processing. . In another embodiment, the processor also performs rendering of the media content including the video and audio streams sent to the user. The processor may also determine two or more media streams to be delivered to the user device based on the location, direction and / or viewing angle the user sees.

The memory is a permanent virtual containing a digital version of the real world, representing real world coordinates, such as the location, orientation, scale and dimensions of real world objects, physical properties and 3D structures of each of the real objects in the form of real-time 3D virtual replicas. Can store world systems The memory may also include a content or virtual copy editor configured to allow a user to create and edit a real-time 3D virtual copy of a real object. However, the permanent virtual world system may further include computer generated virtual objects that may not exist in the real world, such as pure virtual objects. In some embodiments, the permanent virtual world system is shared by two or more users, which means that any change in one or more real-time 3D virtual copies within the permanent virtual world system may be seen by two or more users. In this disclosure, the term “permanent” is used to characterize the state of a system that may continue to exist without a process or network connection running continuously. For example, the term "permanent" refers to the virtual world system and all the real-time contained within it, regardless of whether the user is connected to the virtual world system after the process used to create the real-time 3D virtual replica has stopped. 3D virtual replicas can be used to characterize virtual world systems that still exist. Thus, the virtual world system is stored in a nonvolatile storage location on the server. In this way, real-time 3D virtual replicas can interact and collaborate with each other when they are configured to achieve specific goals, even when users are not connected to the server.

In some embodiments, the memory may further store events in the permanent virtual world system. Saving the event allows the event detection module to detect and play the event for further review, for example. Events represent a disruption of the general event flow. The general event flow can be determined within a parameter range or characteristic. In another embodiment, events are identified through a rule based system implemented in a server. In another embodiment, events are identified through machine learning algorithms implemented in the server.

Memory stores data that can be read with the aid of computer readable media or electronic devices such as hard drives, memory cards, flash drives, ROMs, RAM, DVDs, or other optical discs, as well as other recordable and read only memories. May be any suitable type capable of storing information accessible by a processor, including other media. Memory can include temporary storage in addition to persistent storage. Instructions may be executed directly (eg, machine code) or indirectly (eg, scripts) by the processor. The instructions may be stored in a target code format for direct processing by the processor, or in any other computer language, including scripts or collections of independent source code modules that may be interpreted or precompiled on demand. have. The data can be retrieved, stored or modified by the processor in accordance with the instructions. The data may be stored, for example, in a computer register as a table, an XML document, or a flat file with a plurality of different fields and records in a relational database. The data may be formatted in any computer readable format. A processor may refer to a single dedicated processor, a single shared processor, or a plurality of individual processors, some of which may be shared. In addition, the explicit use of the term “processor” should not be construed as referring solely to hardware capable of executing software, but is not limited to implicitly digital signal processor (DSP) hardware, network processors, application specific semiconductors (ASICs). integrated circuits, field programmable gate arrays (FPGAs), microprocessors, microcontrollers, and the like.

According to one embodiment, the replica editor stored in the server's memory includes software and hardware configured to allow a user to model and edit a real-time 3D virtual replica of a real object. The replica editor may be, for example, a computer-aided drawing (CAD) software application that can store data and instructions needed to enter and edit a virtual replica. The Clone Editor provides explicit data associated with each digital copy, representing the shape, location, location and orientation, physical properties, 3D structure, and data and commands that describe the expected functionality and impact of the real-time 3D virtual copy and the permanent virtual world system as a whole. And input of commands. In general, explicit data cannot be obtained through the sensing mechanism, but through the replica editor, such as building materials, wall thickness, electrical installations and circuits, water pipes, fire extinguishers, emergency exits, window positions, mechanical performance parameters, mechanical sensors and valve positions, etc. It may contain data that needs to be input digitally. As used herein, “instructions” refers to code (eg, binary code) that is understood by a processor and represents the behavior of real-world elements in a real-time 3D virtual copy.

Modeling techniques that transform real objects into real-time 3D virtual replicas with explicit data and commands and make them available to permanent virtual world systems can be based on readily available CAD models of real objects. For example, a machine owner can provide or enter an existing digital CAD model of his machine to the manager of a permanent virtual world system. Similarly, a building owner can provide a building information model (BIM) along with building details that will be stored in the server's permanent virtual world system, which can contain information that is invisible or not readily available through a sensing mechanism. Can be. In such embodiments, the owner of these real objects may be responsible for adding each real-time 3D virtual replica to the permanent virtual world system, which may be achieved, for example, through incentive systems or legal requirements. In some embodiments, a manager, official, or other relevant organization of a permanent virtual system may collaborate with a real object owner to enter a real-time 3D virtual replica into the permanent virtual world system, thereby creating a permanent virtual world system within the server. To realize things faster and more thoroughly. In other embodiments, radar imaging, such as synthetic aperture radars, real-opening radars, light detection and ranging (LIDAR), inverse aperture radars, monopulse radars, and other types of imaging techniques integrate real objects into a permanent virtual world system. Can be used to map and model real objects. Apart from the modeling techniques used to create the virtual replicas, the information in each virtual replica must provide sufficient details about each corresponding real world element, thereby providing highly accurate real-time 3D virtualization of each real world object. The replica becomes available. Whenever possible, real-time 3D virtual replicas are enriched and synchronized through multi-source sensor data. Thus, in some embodiments, the real-time 3D virtual replica includes explicit data and command inputs via a replica editor and multiple source sensor data inputs through a plurality of IoT sensing mechanisms.

According to one embodiment, multiple IoT sensing mechanisms mounted on real objects and user devices may include one or more temperature sensors, proximity sensors, inertial sensors, infrared sensors, contamination sensors (eg, gas sensors), pressure sensors, light sensors, ultrasonic waves. Sensor, smoke sensor, touch sensor, color sensor, humidity sensor, moisture sensor, electrical sensor, or a combination thereof. By providing a plurality of connected elements with sensor mechanisms that constantly capture data from the real world, the permanent virtual world system and each real-time 3D virtual replica stored on the server reflects the real world situation through real-time multi-source sensor data. Remain updated

According to one embodiment, the sensing mechanism attached to or configured in proximity to the real object may include a motion capture sensor including an optical sensor and an inertial sensor or a combination thereof. The optical tracking sensing mechanism can use marker tracking or markerless tracking. In marker tracking, markers are applied to real objects. The marker can be an active and passive infrared light source. Active infrared light can be generated through an infrared light source that can emit infrared light periodically or continuously. Passive infrared light may represent an infrared reverse reflector that reflects infrared light back to the source. One or more cameras are configured to continuously find the marker, and the server may use algorithms to extract the location of the real object and various parts from the marker. The algorithm may need to resolve missing data if one or more markers are outside the camera's field of view or temporarily obscured. In markerless tracking, the camera continues to retrieve an image of the real object and compare it with the image of a real-time 3D virtual replica that is stored and calculated on the server. The inertial tracking sensing mechanism can use devices such as accelerometers and gyroscopes that can be integrated into an inertial measurement unit (IMU). The accelerometer measures linear acceleration, which can be integrated to find the velocity, which is then integrated again to find the position for the initial point. The gyroscope measures the angular velocity, which can also be integrated to determine the angular position relative to the initial point. To increase the tracking accuracy of multiple data points, sensor fusion techniques using a combination of optical and inertial tracking sensors and algorithms can be used.

In yet another embodiment, one or more transceivers may be implemented to receive and transmit communication signals to and from the antenna. Preferably, the transceiver is an mmW transceiver. In an embodiment where a 5G antenna is used, the mmW transceiver is configured to receive the mmW signal from the antenna and send data back to the antenna. Thus, in another embodiment of the sensor fusion technique, optical sensors, inertial sensors, position tracking provided by mmW transceivers, and accurate tracking, low delay, and high QOS capabilities provided by mmW based antennas may be measured in sub centimeters or submillimeters. Position and orientation tracking can be enabled, which can increase accuracy when tracking real-time position and orientation of real objects. The sensing mechanisms and software required to enable this sensing fusion technique may be referred to herein as a tracking module. The user device may also include a tracking module that fuses inertial tracking from one or more IMUs and mmW transceivers. In another embodiment, sensor fusion makes it possible to receive location data from the GNSS tracking signal and augment this data with mmW signals and inertial tracking to provide accurate location and orientation. In some embodiments, tracking can be accomplished using various techniques known in the art, such as time of arrival (TOA), angle of arrival (AOA), or other tracking techniques known in the art (eg, visual imaging, radar techniques, etc.). Can be performed using.

In some embodiments, synchronizing real objects with real-time 3D virtual replicas via a sensing mechanism connected to a server via a network may be applied to the real-time 3D virtual replicas to augment the real-time 3D virtual replicas and provide additional physical properties of the corresponding real objects. Provide feedback. In another embodiment, such synchronization also enables the implementation of virtual reinforcement or virtual compensation for the real object counterpart. In some embodiments, virtual beats or rewards are made possible through virtual resources representing storage and computing functions that are available on the server and that can be shared with real objects over the network through the implementation of a virtual machine. In another embodiment, virtual reinforcement or compensation is enabled through the virtual sensor, which uses the virtually available data that can be used to compensate for missing real data. The virtual sensor can further utilize the use of each real-time 3D virtual copy representing the 3D structure of the virtual world and the real world, so that the real object can recognize other objects in the real world without having to recognize such objects in the real world, It can be recognized through the real-time 3D virtual counterpart of the.

For example, in a factory robot scenario configured to transport material from one place to another, if a physical visual sensor (such as a camera or light sensor) fails or is missing from the robot, the robot may be a wall, table, or other physical. Virtual vision sensors can be utilized by using a virtual map of a factory that includes 3D coordinates and 3D structures of each item to detect and thus avoid obstacles already placed within a permanent virtual world system, such as an object. As another example, a pizza delivery drone can find a desired destination using a virtual model of a city, and can only use real visual sensors to detect and avoid obstacles that may not exist in a permanent virtual world system. In an example of a medical application, a doctor can manipulate a real-time 3D virtual replica of a surgical device with a real counterpart of an operating room remotely in virtual or augmented reality. Other staff (eg, doctors, nurses, etc.) can see the virtual avatar of the doctor performing the surgery and help him when needed. For greater accuracy, the camera captures real patients and operating rooms and integrates them into virtual world versions that are displayed to remote doctors, allowing him to see real-time room conditions. In another example, the real object may experience a lack of computation or storage capacity for a particular task, send its request to the server, and the server may send the requested calculation and storage to the real object.

In another example, the real entity represents an IoT device other than the one described above, including surveillance cameras, traffic lights, buildings, streets, train tracks, appliances, or any other device that can be connected to a network.

In one embodiment, the user of the at least one real time 3D virtual copy is a human user, an artificial intelligence user, or a combination thereof. In another embodiment, the object manipulator is a human object manipulator, an artificial intelligence object manipulator, or a combination thereof. In these embodiments, the real-time 3D virtual replica may further include a virtual bot and a virtual avatar of the user or object operator. The human avatar may be configured to display the physical characteristics of the human user or may be configured to have different visual aspects and characteristics.

In some embodiments, the server may be configured to use artificial intelligence algorithms and group analysis to manage collectively and autonomously or to support management of a plurality of real-time 3D virtual replicas, which leads to the corresponding management of one or more real objects. do. Management may be performed by virtual bots stored and calculated on the server, which may or may not be connected to physical bots in the real world. In another embodiment, the robot may be configured to manage one or more real objects such that management or manipulation commands are sent in real time to corresponding real-time 3D virtual replicas of the real objects through a virtual world system stored and computed on a server. do. In these embodiments, the server may be further configured to use artificial intelligence algorithms to enable the plurality of real-time 3D virtual replicas to cooperate and interact with each other based on one or more goals. Thus, even if a real object has limited communication in the real world, a plurality of real-time 3D virtual replicas can closely cooperate and interact in the virtual world, leading to corresponding interaction and cooperation in the real world.

According to one embodiment, the physical properties and real world coordinates of the real-time 3D virtual replica are configured to correspond to that of the real object, to enable a natural interface and enhanced experience with the real object via the real-time 3D virtual replica. Physical properties may include, but are not limited to, dimensions, texture, mass, volume, refractive index, hardness, color, pressure, and temperature. The data of each real-time 3D virtual copy can be arranged in an accurate data structure. Real world coordinates can include the current position (eg, three-dimensional coordinates) and orientation data (eg, three-dimensional angles) of the real object. Configuring physical properties and real world coordinates of real-time 3D virtual replicas based on real objects not only increases the realism of the objects displayed to the user, but also allows each part of the real object to be accurately represented in the real-time 3D virtual replica. It can facilitate the precise control of the object in 6 degrees of freedom.

Representation of spatial data is an important issue in the programming of permanent virtual world systems, including the rendering and display of computer graphics, visualization, solid modeling and related areas.

The data structure used to represent the permanent virtual world system and each real-time 3D virtual replica is not limited, but may include one or more octrees, quadtrees, BSP trees, sparse voxel octrees, 3D arrays, kD trees, It includes point clouds, wire-frames, B-Rep, constructive solid geometry trees, bintrees, and hexagonal structures.

According to one embodiment, validating one or more changes on a real object through a real-time 3D virtual replica or validating one or more changes on a real-time 3D virtual replica through a real object may be shared by a plurality of virtual and physical pairs. Applying the changes on the data points. In some embodiments, the changes applied to the plurality of data points include one or more of one or more rotational movements, translational movements, selection of one or more behaviors, programming of one or more behaviors, setting of one or more parameters, or a combination thereof. . Changes can be applied directly to real objects to generate real-time, ground truth experience effects on real-time 3D virtual replicas. Similarly, changes can be applied directly to real-time 3D virtual replicas to create real-time, ground truth experience effects on real objects.

In some embodiments, manipulating the at least one real object via the real time 3D virtual replica requires previous virtual selection of the real time 3D virtual replica activated via the 3D user interface, and with the selected real time 3D virtual replica and the corresponding real object. It is necessary to send a selection command. In some embodiments, virtual selection and manipulation commands for validating changes to real objects via real-time 3D virtual replicas may be provided through a natural user interface (NUI) provided by the user device.

For example, a user may use voice recognition, touch recognition, facial recognition, stylus recognition, air gestures (eg, hand poses and gestures and other body / accessory exercises / poses), head and eye tracking, voice and voice utterance, And interact with real-time 3D virtual replicas via NUI free from artificial constraints imposed by input devices such as mice, keyboards, remote controls, etc., such as, for example, relevant machine learning related to visual, voice, pose and / or touch data. have. In other embodiments, manipulation instructions for validating changes to real objects via real-time 3D virtual replicas may be provided through a generic user interface that imposes artificial restrictions, such as a mouse, keyboard, remote control, and the like. In any case, user real-time 3D-based interactions with real-time 3D virtual replicas, among other things, may include, among other things, one or more user devices such as mobile phones, laptops, mobile game consoles, head mounted displays, cross cockpit collimation displays, head-up displays, and smart contact lenses. Can be provided through. In addition, real-time 3D based interaction via the user interface may be provided in one or more of augmented reality, virtual reality, composite reality, or a combination thereof.

In some embodiments, the user device and the real object may refer to the same device. For example, a land vehicle may represent a real object that can be controlled by a real or artificial intelligence user. However, the vehicle may allow the user to interact with the vehicle, send commands to the self-driving artificial intelligence system, or even control the vehicle itself via such an interface and thus allow the car to operate as a user device. May comprise an augmented reality user interface (eg, on a windshield or window)

In an exemplary embodiment, a real object may refer to a factory machine, such as one or more industrial robots used for painting, welding, assembling, packaging, labeling, pick and place, and the like (for a printed circuit board, for example). In other exemplary embodiments, real objects include aircraft (eg, airplanes, drones, helicopters, etc.), land vehicles (eg, cars, motorbikes, trucks, etc.) and marine vehicles (eg, boats, cargo ships, Submarines, etc.). Bidirectional real-time 3D based interaction and management of industrial machines can be useful for remotely managing multiple industrial machines in real time while being able to monitor changes occurring in any part of the manufacturing plant. For example, a government agency to have better control of a running vehicle, in case an ambulance needs a vehicle to move in some way, such as during an accident or natural disaster that would need to go through heavy traffic. The vehicle's two-way real-time 3D based interaction and management can be useful.

A method for enabling two-way interactive manipulation of real-time 3D virtual replicas and real objects includes providing a permanent virtual world system comprising a structure in which at least one real-time 3D virtual replica of at least one real object is represented; Synchronizing the at least one real time 3D virtual replica with the corresponding at least one real object via a sensing mechanism that sends feedback (IoT) back to the real time 3D virtual replica to augment and update the real time 3D virtual replica model; Improving accuracy and providing certain physical properties to real-time 3D virtual replicas based on synchronization; Receiving selection and / or manipulation commands input via a real object or user device; Processing and executing selection and / or manipulation instructions; And updating the permanent virtual world system with the at least one modified real-time 3D virtual replica and sending the updated model to the corresponding user device. In some embodiments, some of the processes may be used to support processing performed locally by real objects.

The above description does not include a comprehensive list of all aspects of the present disclosure. This disclosure is contemplated to include all suitable combinations of the various aspects set forth above, as well as all systems and methods disclosed in the following detailed description and particularly pointed out in the claims that are filed with the application. This combination has particular advantages not specifically mentioned in the above description. Other features and advantages will be apparent from the accompanying drawings and the description below.

By referring to the following detailed description in conjunction with the accompanying drawings, the foregoing aspects and numerous accompanying advantages will be better understood.
1A-1B show schematic diagrams of a system that enables bidirectional manipulation of real-time 3D virtual replicas and real objects, in accordance with one embodiment of the present invention.
2A and 2B show schematic diagrams of a system according to one embodiment of the invention, showing in detail a representation of a relationship between a real world and a virtual world system.
3 shows a schematic diagram of a system according to an embodiment of the invention, showing in detail representations of various operational components of a user device.
4 shows a schematic diagram of a system according to an embodiment of the invention, showing in detail representations of various operating components of a real object.
5 illustrates a flowchart of a method for enabling bidirectional manipulation of real-time 3D virtual replicas and real objects when manipulating real-time 3D virtual replicas through direct manipulation of real objects, according to one embodiment of the invention.
FIG. 6 illustrates a flowchart of a method for enabling bi-directional manipulation of real-time 3D virtual replicas and real objects when manipulating real objects through manipulation of real-time 3D virtual replicas, according to one embodiment of the invention.
7 illustrates a flowchart of a server implementation method according to an embodiment of the present invention, showing in detail an interactive manipulation of real-time 3D virtual replicas and real objects, according to one embodiment.

In the following description, reference is made to the drawings showing various embodiments by way of example. In addition, various embodiments will be described below with reference to some examples. It is to be understood that the embodiments may include changes in design and structure without departing from the scope of the claimed subject matter.

1A-1B are schematic diagrams of systems 100a-b that enable bi-directional manipulation of real-time 3D virtual replicas and real objects, in accordance with one embodiment of the present invention. In FIG. 1A, the real object 102 is communicatively and continuously connected to a real-time 3D virtual copy 104 stored and calculated on the server 106 and displayed via the user interface 108 from the user device 110. Are fully synchronized. The real object 102, the server 106, and the user device 110 are communicatively connected via the network 112.

In the example embodiment of the system 100a of FIG. 1A, the user 114 allows the user to enable real-time and low latency remote virtual selection and manipulation of the real object 102 via the real-time 3D virtual replica 104. Device 110 may be used. User 114 is a change to a human user (e.g., a remote manufacturing plant operator or manager, remote vehicle driver, homeowner, city manager, etc.) or an artificial intelligence (AI) user (e.g., a physical object 102). Software and / or hardware adapted and trained through artificial intelligence algorithms to autonomously manipulate the real-time 3D virtual copy 104 to validate it. Upon selecting and validating one or more changes on the real-time 3D virtual replica 104, the user device 110 sends selection and manipulation commands to the server 106, which server 106 processes the selection and manipulation commands in real time. Commands are sent to each actuator of the real object 102 to produce the desired effect on the actuator. A functional group refers to any device that affects the environment of a real object 102 such as a leg, wheel, arm, or finger of a robot. The actuator is a mechanism that allows the functional group to perform an operation and may include an electric motor, a hydraulic cylinder, a pneumatic cylinder or a combination thereof. For example, when the real object 102 has a lot of degrees of freedom due to a number of joints, one actuator may be able to provide the required rotational and translational motion to provide the desired degree of freedom per joint. It may need to be placed in each joint. In addition, the processed instructions are sent to the user device 110 which only needs to perform lightweight operations on the media content needed to properly represent the updated real-time 3D virtual copy 104 on the user device 110.

In another exemplary embodiment of the system 100a of FIG. 1A, the object operator 116 (eg, an industrial machine operator) is a real time and low latency remote update of a real time 3D virtual replica stored and calculated on the server 106. The real object 102 can be manipulated directly to enable this. The object manipulator 116 may be a human object manipulator or an AI object manipulator. The user interface that enables direct manipulation of the real object 102 may depend on the characteristics of the real object 102 and may include one or more screens, buttons, pedals or other control mechanisms. After validating one or more changes to the real object 102, the real object 102 shares commands with the server 106 in real time. Real object 102 processes and executes instructions simultaneously. The server 106 can then process, in real time, the manipulation instructions needed to update the real time 3D virtual replica in the same manner as the action performed by the real object 102. In some embodiments, the processing performed by server 106 is a complement to the processing performed by real object 102, which serves as support for real object 102 to perform certain heavy task processing. Thus, server 106 may update real-time 3D virtual replica 104 and send the updated real-time 3D virtual replica 104 to user device 110. In some embodiments, most of the heavy-load processing is performed at server 106 so that user device 110 performs the lightweight operations required to present the updated real-time 3D virtual replica 104 on user device 110. Just do it.

According to one embodiment, after the real-time 3D virtual replica 104 is created, stored, and calculated on the server 106, the real-time 3D virtual replica 104 is attached to the real object 102, or the real object 102. It may be synchronized with the real object 102 via a combination of sensing mechanisms located in the vicinity or a combination thereof. Some sensing mechanisms may be attached to the functional groups, joints, connectors, and actuators of the real object 102. The combination of sensing mechanisms can provide a plurality of data points that can be shared between the real time object 102 and the real time 3D virtual replica 104 via the network 112. Shared data points may enable accurate synchronization between real object 102 and real-time 3D virtual replica 104 with accurate tracking of real object 102.

According to one embodiment, real-world position and orientation data and physical properties of the real-time 3D virtual reality copy to enable an enhanced experience with the natural user interface 108 and the real object 102 via the real-time 3D virtual copy 104. Are configured to correspond with that of the real object 102 via the shared data point.

Configuring the physical properties and real position and orientation data of the real time 3D virtual replica 104 based on the real object 102 not only increases the realism of the object displayed to the user 114, but also creates a real time 3D virtual replica. Through real-time 3D virtual replicas, it can play a role in enabling precise control of an object with six degrees of freedom that can be reflected as an exact manipulation of the real object.

According to one embodiment, validating one or more changes to the real object via the real time 3D virtual replica 104 or to the real object 102 via the real time 3D virtual replica 104 may be performed by the real time 3D virtual replica 104. And applying changes on a plurality of data points shared between the real object 102 and the real object 102. In some embodiments, the changes applied to the plurality of data points further comprise one or more rotational movements, translational movements, selection of one or more behaviors, programming of one or more behaviors, setting of one or more parameters, or a combination thereof. The modification can be applied directly to the real object 102 to cause a real time, ground truth experience effect on the real time 3D virtual replica 104. Similarly, modifications can be applied directly on the real time 3D virtual replica 104 to cause a real time, terrestrial verification experience effect on the real object 102. In some embodiments, manipulating the at least one real time 3D virtual replica 104 or the corresponding at least one real object 102 results in a change to the context data affecting the virtual-real pair, and the context data This change in can affect the relationship between the real-time 3D virtual replica 104 and the corresponding at least one real object 102. For example, adjusting the temperature of the air regulator via the real object 102 or its virtual counterpart has a direct impact on the real object 102 in its environment as well as on the air regulator ambient temperature. In another example, instructing a factory's lift truck to transport heavy loads from one area to another can trigger another object on the road to clear the lift truck. As another example, when a street light turns green, another vehicle that is part of the context in terms of the street light may begin to move automatically as a result of the light change.

In some embodiments, manipulating the at least one real object 102 via the real time 3D virtual replica 104 requires a previous virtual selection of the real time 3D virtual replica 104 activated via the 3D user interface and in real time. The selection command is sent to the 3D virtual replica 104 and the corresponding real object 102. In some embodiments, virtual selection and manipulation instructions for validating changes to real object 102 via real-time 3D virtual replica 104 may cause a natural user interface 108 (NUI) implemented in user device 110. Can be provided through. For example, a user may use voice recognition, touch recognition, facial recognition, stylus recognition, air gestures (eg, hand poses and gestures and other body / accessories / postures), head and eye tracking, voice and voice utterance, And a real-time 3D virtual replica 104 via NUI free from artificial constraints imposed by input devices such as a mouse, keyboard, remote control, etc., such as machine learning on visual, voice, voice, pose and / or touch data, for example. ) Can be interacted with. In another embodiment, manipulation instructions for validating changes to the real object 102 via the real-time 3D virtual replica 104 may also be provided through a generic user interface that imposes artificial constraints such as a mouse, keyboard, remote control, and the like. It may be. In any case, user real-time 3D-based interactions with real-time 3D virtual replica 104 include mobile phones, laptops, mobile game consoles, head mounted displays, cross cockpit collimation displays, head-up displays, and smart contact lenses, among others. May be provided through one or more user devices 110. In addition, real-time 3D based interaction via the user interface 108 may be provided in one or more of augmented reality, virtual reality, mixed reality, or a combination thereof.

According to one embodiment, real-time 3D virtual replica 104 is part of a wider permanent virtual world system 118 that is stored and computed on server 106, and permanent virtual world system 106 is a plurality of other real-time 3D virtual. The replica 104 contains the data structure in which it is represented. Thus, any bidirectional commands between the real time object 102 and the real time 3D virtual copy 104 or between the real time 3D virtual copy 104 and the real object 102 pass through the permanent virtual world system 118.

The data structures used to represent the permanent virtual world system 118 and each real-time 3D virtual replica 104 may be, without limitation, one or more octrees, quadtrees, BSP trees, sparse voxel octrees, 3D Arrays, kD trees, point clouds, wire-frames, B-Rep (boundary representations), CSG trees (constructive solid geometry trees), binary trees and hexagonal structures. The data structure functions to accurately and efficiently represent data of each geometric shape of a virtual object in a permanent virtual world system. The correct choice of data structure includes the source of the data, the precision of the geometry to be found during rendering; Whether rendering is performed in real time or prerendered; Whether rendering is performed via a cloud server, through a user device, or a combination thereof; The specific application in which the permanent virtual world system is used (eg, a higher definition level may be required than in other types of applications in medical or scientific applications); Memory capacity from the server and user devices and thus the desired memory consumption.

In some embodiments, network 112 may be, for example, a cellular network, and may include enhanced data rates for global evolution (EDGE), general packet radio service (GPRS), global communication system for mobile communications (GSM), IMS Internet protocol multimedia subsystem (UMTS), universal mobile telecommunications system (UMTS) and the like, as well as any other suitable wireless media such as microwave access (WiMAX), long term evolution (LTE) network, code division multiple access (CDMA), WCDMA (WCDMA) Various technologies are available, including wideband code division multiple access (Wireless Fidelity), satellite, mobile ad-hoc network (MANET), and the like.

According to one embodiment, the network 112 may include an antenna configured to transmit and receive radio waves that enable mobile communication between the real object 102 and the server. The antenna may be connected to the computing center via wired or wireless means. In another embodiment, the antenna is provided in a computing center and / or an area near the computing center. In some embodiments, to service the user device 110 and / or real object 102 deployed outdoors, the antenna may be a millimeter wave (mmW) based antenna system or a combination of mmW based antenna and sub-6 GHz antenna system. And collectively referred to as 5G antennas. In other embodiments, the antenna may include other types of antennas, such as 4G antennas, or may be used as a support antenna for 5G antenna systems. In embodiments where an antenna is used to service a user device 110 located indoors, preferably the antenna may use Wi-Fi which provides data at 16 GHz, although not limited.

In other embodiments, global navigation satellite systems (GNSS) such as GPS, BDS, Glonass, QZSS, Galileo, and IRNSS may be used for positioning of the user device 110. Using signals from a sufficient number of satellites and techniques such as trigonometry and triangulation, the GNSS can calculate the position, velocity, altitude and time of the user device 110. In a preferred embodiment, the external location system is augmented by assisted GNSS (AGNSS) through the architecture of an existing cellular communication network for use in the location system, where the existing architecture includes 5G. In another embodiment, the AGNSS tracking system is further supported by a 4G cellular communication network location system. In indoor embodiments, the GNSS is preferably further augmented through a wireless wireless local area network, such as but not limited to Wi-Fi, which provides data at 16 GHz. In alternative embodiments, GNSS is augmented through other techniques known in the art through differential GPS (DGPS), satellite-based augmentation systems (SBAS), real-time kinematic (RTK) systems, and the like. In some embodiments, tracking of user device 110 is implemented by a combination of AGNSS and inertial sensors in user device 110.

1B illustrates a plurality of object operators 116, including human operator 116a, artificial intelligence (AI) operator 116b, or a combination thereof, capable of flying one or more land objects 102a (eg, a vehicle), or flying. An exemplary embodiment of a system 100b for manipulating at least one real object 102, such as an object 102b (eg, a drone or an airplane) is shown. An object operator 116 operating at least one real object 102 sends an operation command to the real object 102 using an appropriate interface on the real object 102.

Manipulation commands continuously and in real time update each one or more real-time 3D virtual replicas that are updated continuously and in real time and, if applicable (eg, changes or events affecting other real-time 3D virtual replicas 104). Each one or more real-time 3D virtual replicas via network 112 via a permanent virtual world system 118, which can generate a change in context data around each one or more real-time 3D virtual replicas 104). Is sent to 104. A user 114 comprising a human user 114a, an AI user 114b, or a combination thereof may each be one or more through a user device 110 connected to a permanent virtual world system 118 via a network 112. Continuous and real time updates of the real time 3D virtual replica 104 can be observed.

Similarly, user 114 may manipulate one or more real-time 3D virtual replicas 104 through user device 110. Manipulation commands may be individually transmitted in real time to the corresponding one or more real objects 102 via the network 112 via the permanent virtual world system 118. The object operator 116 can observe a continuous real-time update of the motion of the real object 102. As shown in FIG. 1B, the plurality of users 114 are not limited to VR / AR head mounted display 110a, mobile phone 110b, laptop 110c, smart contact lens 110d and smart vehicle 110e. It is possible to simultaneously view changes in the real time 3D virtual replica 104 via several user devices 110, including.

For example, the human operator 116a transmits a command to each real-time 3D virtual replica 104 of the vehicle via the network 112 via the permanent virtual world system 118. (102a)). One or more users 114 of the permanent virtual world system 118 can see the changes that occur through each one or more real-time 3D virtual replicas 104. In some embodiments, at least one real-time 3D virtual replica 104 may further include a virtual bot and a user's avatar. Virtual bots are AI users 114b for responding as automated agents to humans or human-like behavior by using artificial intelligence algorithms and group analysis required to autonomously manage multiple real-time 3D virtual replicas 104 simultaneously. It can be configured, where one or more real-time 3D virtual replica 104 leads to the corresponding management of the corresponding real object 102. Virtual bots may or may not be connected to physical robots in the real world. The human avatar may be configured to display the physical characteristics of the human user or may be configured to have different visual aspects and characteristics. In another embodiment, an artificial intelligence device or artificial intelligence program, such as a robot or machine, may be configured as an AI object operator 116b to manage one or more real objects 102, whereby the management or manipulation commands may be directed to a permanent virtual world. The system 118 is transmitted in real time to the corresponding real-time 3D virtual replica 104 of the real object 102. Thus, the artificial intelligence device or program may serve as the AI object manipulator 116b of one or more real objects 102.

In some embodiments, as shown with an example of the smart vehicle 110e, the user device 110 and the real object 116 may in some cases refer to the same device. For example, smart vehicle 110e may refer to a real object that can be controlled by a real or artificial intelligence user 114. However, the smart vehicle 110e has an augmented reality user interface (e.g., which allows the user to interact with the vehicle, send commands to the self-driving artificial intelligence system, or even control the vehicle itself via the interface (e.g., , On a windshield or window), thus making smart vehicle 110e serve as user device 112.

In some embodiments, the plurality of real-time 3D virtual replicas 104 may use artificial intelligence algorithms to cooperate and interact with each other based on one or more goals. Thus, even though the real object 102 has limited communication with each other in the real world, the plurality of real-time 3D virtual replicas 104 can cooperate and interact closely in the permanent virtual world system 118, which corresponds to the corresponding in the real world. It can lead to interaction and cooperation.

2A and 2B show schematic diagrams of systems 200a-b in accordance with one embodiment of the present disclosure, detailing a representation of a relationship between a real world and a virtual world system. Some elements of FIGS. 2A-2B may be similar to those of FIGS. 1A-1B, and thus the same or similar reference numerals may be used.

Referring to system 200a of FIG. 2A, one or more servers 106 may be provided as hardware and software that includes at least a processor 202 and a memory 204. Processor 202 may be configured to execute instructions contained in memory 204, and memory 204 is further configured to store instructions and data.

The processor 202 may be configured to access and execute instructions and data contained in the memory 204, including real-time processing of manipulation instructions coming from either the real object 102 or the user device 110. For example, the processor 202 may use the corresponding real-time 3D virtual replica 104, simulation of the real object 102, 3D structure processing, group analysis, rendering, and real object 102 via the real-time 3D virtual replica 104. Through the implementation of the virtual reinforcement or virtual compensation of, it may be configured to implement an artificial intelligence algorithm for the management of at least one real object 102. The processor 202 may also enable interactive interactive operation of the real object 102 and the real-time 3D virtual replica 104 by performing kinematic calculations on manipulation instructions. In one embodiment, the processing of manipulation instructions by the processor 202 is a complement to the processing performed by the real object 102, which serves as support for the real object 102 to perform some heavy task processing operation. . In another embodiment, the processor 202 further performs rendering of media content including video and audio streams sent to the user. The processor 202 may also determine two or more media streams to be delivered to the user device based on the viewing position, direction and / or viewing angle of the user. Processor 202 may refer to a single dedicated processor, a single shared processor, or a plurality of individual processors, some of which may be shared. Furthermore, the explicit use of the term "processor" should not be construed as exclusively referring to hardware capable of executing software, and implicitly includes digital signal processor (DSP) hardware, network processors, and application specific integrated circuits (ASICs). a circuit, a field programmable gate array (FPGA), a microprocessor, a microcontroller, and the like.

The memory 204 may store information accessible by the processor 202, including instructions and data that may be executed by the processor 202. The memory 204 may be read with the aid of an electronic device, such as a computer readable medium or hard drive, memory card, flash drive, ROM, RAM, DVD or other optical disk as well as other recordable and read only memory. May be any suitable type capable of storing information accessible by the processor 202, including other media storing data. The memory 204 may include temporary storage in addition to permanent storage. Instructions may be executed directly (eg, machine code) or indirectly (eg, scripts) by the processor 202. The instructions may be stored in an object code format for direct processing by the processor 202 or in any other computer language including scripts or collections of independent source code modules that may be interpreted or precompiled as needed. Can be. Data may be retrieved, stored or modified by the processor 202 in accordance with instructions. The data may be stored, for example, in a computer register as a table with a plurality of different fields and records, an XML document, or a flat file in a relational database. The data may be formatted in a computer readable format. The memory 204 can store the content editor 206 and the permanent virtual world system 118. The content editor 206 may be located near the real object 102 by the user, as well as other real-time 3D virtual replicas 104 of the real object 102 as well as other objects that may be included in the permanent virtual world system 118. It allows to create and edit objects that can be (eg, other machines, tables, walls, etc.). In addition, the real-time 3D virtual replica 104 may be stored in the permanent virtual world system 118 to make them available in real world locations, including locations and orientations for other real objects 102. Permanent virtual world system 118 may include a virtual version of the real world, including the location and orientation of the real world, scale, dimensions, physical properties, and 3D structures of real objects. However, the permanent virtual world system 118 may also include computer generated virtual objects that may not exist in the real world, such as pure virtual objects.

In some embodiments, memory 204 may further store events in persistent virtual world system 118. Storing an event allows, for example, an event detection module (not shown) to detect and play the event for later review. Events represent a disruption of the general event flow. The general event flow can be determined within a parameter range or characteristic. In another embodiment, events are identified through a rule based system implemented in server 106. In other embodiments, events are identified through machine learning algorithms implemented in server 106. For example, since an event may refer to a vehicle crash, the persistent virtual world system 118 may immediately detect a collision occurring on a real object that may later be reproduced, for example, to assist in the judicial investigation.

At least one real-time 3D virtual replica 104 is a real object that includes the physical properties 210 of the real object 102 along with a data point 208 that includes the position and orientation data 212 of the real object 102. It may include a plurality of data points 208 shared with 102. The data point 208 is a constant of manipulation instructions sent via at least one real object 102 or at least one corresponding real-time 3D virtual replica 104 of any change that may occur in any of the real-virtual pairs. Enable tracking and synchronization. Data point 208 may be determined via a sensing mechanism that includes hardware and software mounted on or near at least one real object 102.

The physical characteristics 210 of the real-time 3D virtual replica 104 of the permanent virtual world system 118 may include, without limitation, dimensions, shapes, textures, masses, volumes, refractive indices, hardness, colors, pressures, and temperatures. Physical properties 210 may be edited via content editor 206, which may be an omputer-aided drawing software application. Modeling techniques for converting real-world objects, such as real objects 102 or other objects, into three-dimensional objects may be based on techniques known in the art. For example, a machine manufacturer can provide an already existing digital CAD model of a machine that can be integrated into the permanent virtual world system 118. In another embodiment, radar imaging, such as synthetic aperture radar, real-opening radar, light detection and ranging (LIDAR), inverse aperture radar, monopulse radar, and other types of imaging techniques, provides real world objects with a permanent virtual world system 118. Can be used to map and model real-world objects prior to integration. In other embodiments, one or more physical properties of the real object 102 such as dimensions, shapes, textures, volumes, temperatures and colors may be obtained directly through the sensing mechanism and edited through the content editor 206.

Real-time processing performed by the processor 202 is received for manipulation instructions received when manipulating the real object 102 via the real time 3D virtual replica and when manipulating the real time 3D virtual replica through the real object 102. Kinematic calculations for manipulation commands. Motion capture combined with kinematic calculations, for example, when processing mechanical commands received from a real object 102 that can be used to update a real-time 3D virtual replica 104 when moving a mechanical arm through an appropriate interface. The technique reproduces the movement of the real object 102 in the real time 3D virtual replica 104.

Motion capture techniques use various optical sensing mechanisms, inertial sensing mechanisms, or a combination thereof. The optical tracking sensing mechanism can use marker tracking or markerless tracking. In marker tracking, the real object 102 has a marker. The marker can be an active and passive infrared light source. Active infrared light can be generated through an infrared light source that can periodically or continuously emit an infrared light flash. Passive infrared light may refer to an infrared retroreflector that reflects infrared light back to the source. One or more cameras are configured to continuously find the marker, and the processor 202 may use algorithms to extract the location of the real object 102 and various portions from the marker. The algorithm may need to address data loss when one or more markers are outside the camera's field of view or temporarily hidden. In markerless tracking, the camera continuously retrieves an image of a target, such as a real object 102, and compares it with an image of a real-time 3D virtual replica 104 included in the server 106. The inertial tracking sensing mechanism can use devices such as accelerometers and gyroscopes that can be integrated into an inertial measurement unit (IMU). The accelerometer measures linear acceleration, which is then integrated to find the velocity and then integrated again to find the position for the initial point. The gyroscope measures the angular velocity, which is also integrated to determine the angular position relative to the initial point.

In some embodiments, synchronizing the real time object with the real time 3D virtual replica via the server 106 enables the implementation by the processor 202 of virtual reinforcement or virtual compensation of the real object 102. In some embodiments, virtual augmentation or compensation is via virtual resources representing storage and computing capabilities that are available at server 106 and can be shared with real objects 102 via network 112 through implementation of virtual machines. It becomes possible. In another embodiment, virtual augmentation or compensation is activated through a virtual sensor, indicating using virtually available data that can be used to compensate for missing real data. The virtual sensors can further utilize the use of each real-time 3D virtual replica 104 representing the 3D structure of the virtual world and the real world, such that the real object 102 does not require such object recognition in the real world without requiring their real-time 3D. The virtual replica 104 may recognize other objects in the real world. In one embodiment, sensor fusion techniques using a combination of optical and inertial tracking sensors and algorithms may be used to increase the accuracy of tracking multiple data points 208. In yet another embodiment, one or more transceivers may be implemented to receive and transmit communication signals from the antennas. Preferably, the transceiver is an mmW transceiver. In an embodiment where an mmW antenna is used, the mmW transceiver is configured to receive the mmW signal from the antenna and send data back to the antenna. Thus, in another embodiment of sensor fusion technology, optical, inertial, and position tracking and precision tracking, low latency, and high QOS functions provided by mmW antennas are provided in sub-centimeters or sub-millimeters. Position and orientation tracking may be enabled, which may increase accuracy when tracking the real time position and orientation of the real object 102. In another embodiment, sensor fusion also makes it possible to receive location data from the GNSS tracking signal and augment this data with mmW signals and inertial tracking to provide accurate location and orientation. In some embodiments, tracking is a variety of techniques known in the art, such as time of arrival (TOA), angle of arrival (AOA), or other tracking techniques known in the art (eg, visual imaging, radar techniques, etc.). It can be performed using.

When manipulating the real time 3D virtual replica 104 via the real time object 102, the processing performed by the processor 202 enables input from the real object 102, and the real time 3D virtual within the server 106. The replicas are updated and their inputs are converted into output video and optionally an audio stream, which are then streamed to the user device 110 to display the updated real-time 3D virtual replica 104. Upon receiving input from the manipulations performed on the real object 102, the processor 202 may perform preprocessing operations of the media content, including sound and video compression and assembly. Sound and video compression can be performed using audio and video codecs, respectively, which are then assembled into a container bit stream. The audio codec may be any encoding technique capable of receiving audio output and generating an audio data stream such as WMA, AAC or Vorbis. In some embodiments, the audio codec may support encryption of the audio stream. Similarly, the video codec may be any encoding technique capable of receiving video output and generating a video data stream such as WMV or MPEG-4. In some embodiments, the video codec may support encryption of the video stream. Preferably any stream for other codecs or other ways of generating an audio or video stream that also supports the encryption of the associated input data stream or that allows subsequent encryption of any stream to the resulting audio or video stream or otherwise. Can be used. The container bit stream may be any suitable bit stream configured to accommodate one or more data streams, such as ASF or ISMA. However, other suitable container bit streams may also be used, preferably allowing subsequent encryption of the resulting container bit stream.

The processor 202 may further determine two or more media streams to be delivered to the user device 110 based on the location, direction, and / or viewing angle that the user device 110 sees; And the media stream can be rendered. After determining two or more media streams, processor 202 performs media rendering in such a way that user device 110 only needs to perform lightweight calculations on the processed media content in order to properly present the processed media content to the user. Can be done.

The rendering of media content may include various rendering techniques capable of forming two or more photorealistic 3D media streams of media representing the real object 102, including warping, stitching (such as two or more media streams). stitching) and interpolation. Rendering may include a more complex reconstruction process based on input stream data. For example, rendering may rely on a combination of standard image reconstruction techniques such as stitching, warping, interpolation, and extrapolation. For example, extrapolation may be necessary in areas where there is no (visual) information available or limited based on available media streams to fill in gaps or holes in media data. However, the reconstruction process is not limited to computer vision technology, and in any combination includes one or more of reconstructed three-dimensional geometric information, material parameters, and light fields that may correspond to the flow of light in the captured scene, etc. It should be understood that more spatial data about scenes can be considered. Spatial data can be used to re-render scenes captured with 3D rendering technology. In one or more embodiments, rendering of the output stream may be applied to deep learning techniques and / or neural networks that may be applied to recreate an image or frame of the output stream from a series of images or frames of a media stream of the same scene obtained from different perspectives. It may include using. This may enable complex reconstruction and generation of the output stream even if at least a portion of the scene is not captured completely or completely in detail.

In some embodiments, the processor 202 is not limited to two-dimensional visual output data from an application, for example, to receive stereoscopic output of the application and related commands, and to receive two video streams or one interlaced. A video stream can be generated and transmits visual data for each eye of the user. Similarly, the processor 202 may also generate an audio stream carrying spatial sound data as well as data streams for other multidimensional multi-mode data. In one embodiment, the plurality of media streams may be further processed such that the quality of the output stream is focused on the location the viewer is actually viewing, such as based on the determined gaze direction or at the center of the frame. In addition, the media stream may be processed to enable predicted motion reconstruction, or to extrapolate the media stream, including the viewer pre-reconstructing the next viewing position and its region. In addition, additional processing may be applied taking into account the focal length of the eye (eg, determined by the relative position and orientation of the pupil) to further improve the quality and fidelity of the output stream. Non-limiting examples are focus distance dependent shift and parallax effects as well as defocus blurring of this portion of the scene that may be determined to be out of focus for the viewer.

When processing a manipulation command received from a real-time 3D virtual replica through the user device 110 to manipulate the real object 102, the processor 202 is dependent upon the available actions depending on the nature of the real object 102. It is possible to access a number of predefined processing instructions on the basis, to match the associated processing instructions and operation instructions, and to send execution instructions to each machine actuator to affect a plurality of actuators. The manipulation instruction may include one or more rotational movements, translational movements, selection of one or more behaviors, programming of one or more behaviors, setting of one or more parameters, or a combination thereof. In addition, because the physical properties of the real-time object 102 are stored and synchronized to the server 106, the speed and feel of movement of certain portions of the mechanical arm of the real-time 3D virtual replica are simulated based on the capabilities of the real object 102 Therefore, real life is limited. In embodiments in which the manipulation instructions include only translational or rotational movements, processor 202 may process the instructions using reverse kinematic calculations. Reverse kinematics are generally used to determine joint parameters that provide a desired position for each functional group based on the desired position. Manipulation commands include more complex movements, including a plurality of sequential steps (eg, mechanical arms that perform robot seating, standing, punching, avoiding obstacles, or pick-and-drop). In an embodiment, the processing instructions may use an integrated combination of forward kinematics and reverse kinematics. Forward kinematics uses equations to calculate the position of the end functional group from the value specified for the joint parameter.

In an exemplary embodiment, real object 102 refers to a factory machine, such as one or more industrial robots used for painting, welding, assembling, packaging, labeling, picking up, and placing (eg, for a printed circuit board). can do. In other exemplary embodiments, real objects may include aircraft (eg, airplanes, drones, helicopters, etc.), land vehicles (eg, cars, motorbikes, trucks, etc.) and marine vehicles (eg, boats, A vehicle, such as a cargo ship, a submarine, etc.). Bidirectional management of industrial machines can be useful for remotely managing multiple industrial machines in real time while being able to monitor changes occurring in any part of the manufacturing plant. For example, a government agency to have better control of a running vehicle, in case an ambulance requires a vehicle to move in some way, such as during an accident or natural disaster that would need to go through heavy traffic. Two-way vehicle management can be useful.

For example, instructing the mechanical arm to perform one or more motions may comprise a real time 3D virtual replica 104 of the mechanical arm in a space that includes all or most of the objects available in the workspace in which the mechanical arm is located. It may include what the user sees through the VR / AR glasses or other head mounted display. The user extracts multiple options from the memory 204 and prompts the processor 202 of the server 106 to send the option to be displayed to the user via the user device 110, the real-time 3D virtual replica 104. The real-time 3D virtual replica 104 of the mechanical arm can be touched to remotely and virtually select the mechanical arm through. Options may include, for example, moving, rotating, and performing pick-and-drop operations. Depending on which option the user selects, the processor 202 may proceed to compare the operation instruction with a preprogrammed execution instruction. For a simple move task that includes only translational or rotational movements, the real-time 3D virtual replica 102 user can select a mechanical arm (eg, by touch, air gesture, mouse or button, etc.) on the user interface, and the mechanical arm Move to the desired position and direction to perform the desired action. In another example, processor 202 may perform certain processing tasks for operational instructions, such as tasks requiring heavier computing operations, and send preprocessed instructions to the mechanical arm. The processor of the mechanical arm can then perform other processing tasks before executing the instructions. More complex tasks may include working on one or more objects, such as allowing a user to interact with the environment of the mechanical arm. For example, in the case of pick-and-drop motion, the user may first touch the mechanical arm, select the pick-and-drop motion, select an object and then select a target position at which the object should be dropped. Processor 202 then compares the operation instructions with the available processing instructions, processes the instructions in a logical order, and proceeds with the transfer of the processed instructions for execution on the machine arm. The mechanical arm may also perform some processing instructions.

In another example, a user can manipulate the real-time 3D virtual replica 104 to rotate the mechanical arm. The user can touch the mechanical arm to virtually select the mechanical arm via the real-time 3D virtual replica 104 and then rotate the mechanical arm. The processor 202 can process the commands and rotate the machine arm in real time by sending each command to the corresponding actuator. However, the speed at which the mechanical arm can rotate through the real-time 3D virtual replica 104 can be limited to the speed at which the mechanical arm can reach in consideration of the safety factor.

2B illustrates a system 200b that further illustrates the relationship between the real world 214 and the virtual world system 116. Real world 214 may include a section 216 including a plurality of users 114, which may be human users 114a or AI users 114b, using one or more user devices 110; And a section 218 including an object manipulator 116, which may be an object manipulator 116a or an AI object manipulator 116b, for manipulating one or more real objects 102. The elements of sections 216 and 218 are physical attributes 210 and their location and orientation such that the data points 208 of the real world 214 correspond to the data points 208 of the permanent virtual world system 118. Share data point 208 with permanent virtual world system 118, which includes 212. The real world 214 is graphically included in the permanent virtual world system 118 (eg, IoT Other elements not included in section 216 and section 218, such as elements that do not share data points in real time with the permanent virtual world system 118, because sensing mechanisms such as sensors are not installed on them. It may further include (not shown).

Permanent virtual world system 118 is within a real world 214 including a user and / or other object operator virtual replica 222, a real object virtual replica 224, and other virtual replicas 224 corresponding to other objects. Arrange the real-time 3D virtual replica 104 of the elements into the 3D structure 220.

Any type of manipulation command sent from the user device 110 to the real object 102 is shared to the real object 102 via the shared data point 208 via the permanent virtual world system 118, and each real time. Update the 3D virtual replica and, if applicable, the virtual replica and the context of each real object 102 in real time. Similarly, manipulation commands sent via the real object 102 update in real time through a shared data point 208 that can be viewed by one or more users 114a via each user device 110 in real time. It updates the virtual replica and the context of each real object 102, if applicable.

3 shows a schematic diagram of a system according to one embodiment of the present disclosure, showing in detail a representation of an operating component of a user device 110. The operating components are all input / output (I / O) modules 302, a power supply 304, a memory 306, a sensor 308, a transceiver 310 and a network interface operably connected to the processor 314. 312).

I / O module 302 is implemented as computing hardware and software configured to interact with a user and provide user input data to one or more other system components. For example, I / O module 302 generates user input data based on real-time 3D-based interactions and processes the user input data prior to being sent to another processing system via network 112 such as server 106. And provide to 314. In another example, I / O module 302 may be configured to connect external computing pointing devices (eg, touch screens, mice, 3D controls, joysticks, game pads, etc.) and / or text input devices (eg, keyboards, tools, etc.). It may include. In yet other embodiments, I / O module 302 may provide more functionality, less functionality, or different functionality than the functionality described above.

The power source 304 is implemented as computing hardware configured to provide power to the user device 110. In one embodiment, the power supply 304 may be a battery. Power source 304 may be embedded in or removed from the device and may be rechargeable or non-rechargeable. In one embodiment, the devices may be resupplied by replacing one power source 304 with another power source 304. In another embodiment, power supply 304 may be recharged by a cable attached to a charging source, such as a universal serial bus (USB), Firewire, Ethernet, Thunderbolt, headphone cable, attached to a personal computer. In another embodiment, the power source 304 may be recharged by inductive charging, where the electromagnetic field is used to transfer energy from the inductive charger to the power source 304, where the inductive charger and the power source are connected to each other via different cables. It doesn't have to be plugged in, just put it near. In another embodiment, a docking station can be used to facilitate charging.

Memory 306 may be implemented as computing software and hardware adapted to store application program instructions and store telemetry metadata of devices captured by sensor 308. Memory 306 may be read with the aid of a computer readable medium or an electronic device, such as a hard drive, memory card, flash drive, ROM, RAM, DVD or other optical disk as well as other recordable and read only memory. It may be any suitable type capable of storing information accessible by the processor 314, including other media storing data. Memory 306 may include temporary storage in addition to permanent storage.

Sensor 308 may be implemented as computing software and hardware adapted to obtain various telemetry metadata from a user and determine / track the location and orientation of the user with their movement. The sensor 308 may include, for example, one or more of an inertial measurement unit (IMU), an accelerometer, a gyroscope, an optical sensor, a haptic sensor, a camera, an eye tracking sensor, and a microphone. The IMU is configured to measure and report speed, acceleration, angular momentum, kinetic speed, rotational speed and other telemetry metadata of the user device 110 using a combination of an accelerometer and gyroscope. The accelerometer in the IMU can be configured to measure the acceleration of a real-time 3D based interaction device, including the acceleration due to the gravitational field of the earth. In one embodiment, the accelerometer in the IMU may include a three-axis accelerometer capable of measuring acceleration in three orthogonal directions. Light sensors, haptic sensors, cameras, eye tracking sensors, and microphones can be used to capture input details from the user and the user's environment whenever manipulating real-time 3D virtual replicas directly, which can be used to capture voice and haptic real-time 3D It may be sent to the server 106 to determine one or more media streams to be delivered to the user device 110 depending on the underlying interactions as well as environmental factors such as lighting and sound and the location and orientation the user sees.

The transceiver 310 may be implemented as computing software and hardware configured to allow a device to receive wireless radio waves from an antenna and send data back to the antenna. In some embodiments, mmW transceivers may be used that are configured to receive mmW wave signals from and send data back to the antenna when interacting with immersive content. The transceiver 310 may be a two-way communication transceiver 310.

In one embodiment, the tracking module 316 combines the performance of the IMU, accelerometer and gyroscope with the location tracking provided by the transceiver 310 and the accurate tracking, low latency and high QOS features provided by the mmW based antenna. This can be implemented by enabling sub-centimeter or sub-millimeter location and orientation tracking, which can increase accuracy when tracking the real-time location and orientation of the user device 110. In a further embodiment, the tracking module 316 makes it possible to receive location data from the GNSS tracking signal and augment this data with mmW signals and inertial tracking to provide accurate location and orientation. In some embodiments, tracking can be accomplished using various techniques known in the art, such as time of arrival (TOA), angle of arrival (AOA), or other tracking techniques known in the art (eg, visual imaging, radar techniques, etc.). Can be performed using.

Network interface 312 communicatively connects to network 112, receives computer readable program instructions from network 112 sent by server 106, and executes the user for execution by processor 314. It may be implemented as computing software and hardware for transmitting computer readable program instructions for storage in the memory 306 of the device 110.

Processor 314 may be implemented as computing hardware and software configured to receive and process user input data. For example, processor 314 may provide imaging requests, receive imaging data, process environment or imaging data into other data, and process user input data and / or imaging data to provide user real-time 3D based interaction data. And receive the server 106 request, receive the server 106 response, and / or provide user real time 3D based interaction data, environmental data, and content object data to one or more other system components. have. For example, the processor 314 may receive user input data from the I / O module 302 and implement each of the application programs stored in the memory 306. In another example, processor 314 may include position, position, or other telemetry metadata (eg, user hand movements, controller manipulation, driving trajectory) from sensor 308, from transceiver 310, or a combination thereof. Information about the camera and the like). In addition, the processor 314 may implement analog or digital signal processing algorithms, such as primitive data reduction or filtering. In a particular embodiment, the processor 314 is configured to perform a lightweight computational task on media content received from the server 106, such as the calculations required to accurately represent a real-time 3D virtual copy on the user device 110. Can be.

4 shows a schematic diagram of a system according to one embodiment of the present disclosure, detailing the representation of various operating components of real object 102. Operational components may include input / output (I / O) modules 402, power supplies 404, memory 406, sensors 408 attached to actuators 410 and effectors 412, transceivers 414, and networks. Interface 416, all of which are operatively coupled to the processor 418.

I / O module 402 is implemented as computing hardware and software configured to interact with an object operator and provide object operator user input data to one or more other system components. For example, I / O module 402 interacts with object manipulators based on real-time 3D-based interactions, generates user input data, and communicates to other processing systems via network 112, such as server 106. It may be configured to provide user input data to the processor 418 before transmitting. In another example, I / O module 402 may be an external computing pointing device (eg, touch screen, mouse, 3D control, joystick, lever, steering wheel, game pad, etc.) for selecting objects and options and / or It may be implemented as a text input device (eg, a keyboard, a button, a dictation tool, etc.) for inputting an operation command configured to interact with the real object 102. In yet another embodiment, I / O module 402 may provide more, less, or different functionality to the aforementioned functionality.

The power source 404 is implemented as computing hardware configured to provide power to the real object 102 and may follow a description similar to that described in FIG. 3.

Memory 406 may be implemented as computing software and hardware adapted to store application program instructions and data, and may follow a description similar to that described in FIG. 3.

The sensor 408 may, for example, provide a data point that may be synchronized and shared with the server 106 and to provide the server 106 with a data representation of one or more physical attributes of the real object 102. Can be adapted to determine and track the position and orientation of the plurality of actuators 410 and the actuators 412 of the real object 102. In some embodiments, the sensor 408 may be implemented in another area of the real object 102 or in an area surrounding the real object 102. For example, the sensor 408 may be disposed on a plurality of joints and connectors of the real object 102. Sensor 408 may include, for example, motion capture equipment including the optical sensor, inertial sensor, or a combination thereof described with reference to FIG. 2. In other embodiments, other sensors 408 may also be included that can provide data representations of other features of the real object 102, such as thermometers, pressure sensors, humidity sensors, etc., depending on the characteristics and functions of the real object 102. have.

The transceiver 414 may be implemented as computing software and hardware configured to allow the real object 102 to receive wireless radio waves from the antennas and send data back to the antennas. In some embodiments, mmW transceiver 414 may be used that may be configured to receive mmW wave signals from the antenna and send data back to the antenna when interacting with immersive content. The transceiver 414 may be a bidirectional communication transceiver 414.

In one embodiment, tracking module 420 is implemented by combining the performance of IMUs, accelerometers and gyroscopes with the location tracking provided by transceiver 414 and the accurate tracking, low delay and high QOS capabilities provided by mmW based antennas. This may enable tracking of sub centimeter or submillimeter position and direction, which may increase accuracy when tracking the real time position and orientation of the real object 102. In another embodiment, the tracking module 420 makes it possible to receive location data from the GNSS tracking signal and augment this data with mmW signals and inertial tracking to provide accurate location and orientation. In some embodiments, tracking can be accomplished using various techniques known in the art, such as time of arrival (TOA), angle of arrival (AOA), or other tracking techniques known in the art (eg, visual imaging, radar techniques, etc.). Can be performed using.

The network interface 416 is communicatively connected to the network 112, receives computer readable program instructions from the network 112 sent by the server 106, and executes the device for execution by the processor 418. May be implemented as computing software and hardware for transmitting computer readable program instructions for storage in a memory 406.

The processor 418 processes the operation instructions that are entered directly through the I / O module 402 or coming from the server 106, and send the processed instructions to the actuator 410 to perform the necessary operations of the actuator 412. Can be configured to For example, processor 418 may receive user input data from I / O module 402 and implement each of the application programs stored in memory 406. In another example, processor 418 receives location, position, or other telemetry metadata from sensor 408, transceiver 414, or a combination thereof, and transmits the information to server 106 to update the real-time 3D virtual replica. ) Can be sent. The processor 418 may also implement analog or digital signal processing algorithms such as raw data reduction or filtering. In some embodiments, processor 418 may share some computational tasks with server 106.

FIG. 5 shows a flowchart of a method 500 according to an embodiment of the present invention, detailing the manipulation of the real-time 3D virtual replica 108 via direct manipulation of the real object 102, according to one embodiment. . The method 500 may be executed by a system according to one embodiment of the present disclosure, such as, for example, the systems described with respect to FIGS. 1A-4.

Method 500 includes a persistent virtual world system 118 that includes a data structure in which at least one real-time 3D virtual replica 104 of at least one real object 102 is represented, as seen at block 502. By providing the server 106. Providing a real time 3D virtual replica 104 of the real object 102 generates and / or graphically generates a real time 3D virtual replica 104, including physical properties and three-dimensional real world coordinates of the real object 102. For editing, it may include using a content editor, which is stored and calculated on server 106. The real time 3D virtual replica 104 may be accessed by the user 114 from the user device 110 via the appropriate user interface 108.

The method 500 then continues by synchronizing the real-time 3D virtual replica 104 with the real object 104, as seen at block 504, which is a real object such as an actuator, a functional group, a joint, and a connector. Acquiring data from a plurality of sensors (eg, a camera disposed in proximity to real object 102) on various portions of 102 or in an area around real object 102. The plurality of sensors may generate a plurality of data points that are communicated to the server 106 and shared with the real-time 3D virtual replica 104. In some embodiments, the sensor connected to the real object 102 also provides feedback to the real time 3D virtual replica 104, represented by the dashed line 506. The feedback data may further provide additional physical properties of the real object 102 to augment the real time 3D virtual replica 104 and increase the accuracy of the real time 3D virtual replica 104 relative to the real object 102. When synchronizing the real-time 3D virtual replica 104 with the real object 102, the server 106 transfers the synchronized real-time 3D virtual replica 104 to the real object 102 via a network through the permanent object virtual world system 118. And the user device 110. After synchronizing the real time 3D virtual replica 104 with the real object 102 and the user device 110, the operations performed on the real object 102 or the real time 3D virtual replica 104 may be performed on the virtual or real counterpart. It should be understood that it has a direct effect.

Continuing the process, as shown at block 508, the object manipulator 116 can proceed with direct manipulation of the real object 102. In block 510, the real object 102 may proceed with the processing of the operation command. For example, referring to FIG. 4, the object operator 116 may enter an operation command that may be sent to the processor 418 via the appropriate I / O module 402. The processor 418 can access the processing instructions and data in the memory and can proceed with the processing of the manipulation instructions from the object operator 116. The processor 418 may also perform the kinematic calculations necessary for the actuator to determine which joint to move to perform the task. The method 500 then proceeds as the real object 102 executes the manipulation command, as shown at block 512. Executing the command may include transmitting electrical signals to a plurality of actuators for activation that move the functional groups required by the real object 102 to perform the desired task.

Since the system is synchronized, manipulating the real object 102 at block 508 sends an operation command to the server 106. After receiving the manipulation command, the server 106 also recreates the movement of the real object 102 in the real-time 3D virtual replica 104, as shown in block 514, as well as the video and audio streams. May perform processing tasks for commands, such as kinematic calculations used to render the task to send the data to the user device 110. Processing the manipulation command by the server 106 further includes receiving and processing the position and orientation of the object operator through the real object 102 and the position and orientation of the user 114 through the user device 110. can do. In some embodiments, the processing performed at server 106 complements the processing performed at real object 102. Thus, real object 102 may perform some of the processing instructions, while server 106 may support real object 102 by performing heavier task calculations.

After the server 106 processes the operation command at block 514, the method 500 is modified at least one that is synchronized at the user device 110 of the server 106, as shown at block 516. It continues by updating the permanent virtual world system with the real time 3D virtual replica 104. In step 518, user device 110 continues by outputting an updated permanent virtual world system 118 that includes a real time 3D virtual copy 104 that may have undergone a change, which is the updated real time 3D virtual copy. Performing a lightweight operation on the received media stream to properly display 104. The method 500 then checks 520 whether there are more manipulation instructions coming from the real object 102, in which case the method 500 blocks by the object operator 116 manipulating the real object 102. Return to 508. If no further instructions, as shown at termination 522, the process may end.

6 shows a flowchart of a method 600 according to one embodiment of the present invention, showing in detail the manipulation of a real object 102 through direct manipulation of a real-time 3D virtual replica 104. The method 600 may be executed by a system according to one embodiment of the present disclosure, such as, for example, the systems described with respect to FIGS. 1A-4. Some steps shown in FIG. 6 correspond to the steps shown in FIG. 5, and like reference numerals and descriptions in the drawings may be used in FIG. 6.

The initial portion of method 600 is the same as the initial portion of method 500. Thus, the method 600 shares the dotted lines 506 with the method 500 as well as the blocks 502 and 504. The method 600 virtually selects the real time 3D virtual replica 104 from the user device 110 via the appropriate user interface 108, as shown at block 602, and then selects the selected real time 3D virtual replica 104. And by sending a selection command to the corresponding real object 102. In some embodiments, the selection command sent by the user device 110 to the server 106 includes position and orientation data of one or more users 114. The server 106 and the real object 102 may proceed with the processing of the selection commands in blocks 604 and 606, respectively. The method 600 is shown in block 608 where the user 114 manipulates the real-time 3D virtual replica 104, followed by the server 106 and the real object 102 at blocks 610 and 612. Proceed by processing each operation instruction. In some embodiments, the processing performed at the server 106 is a complement to the process performed at the real object 102 and therefore may support the real object 102 by performing heavier task calculations.

Subsequently, the real object 102 proceeds to the execution of the command for the real object 102 at block 614. At the same time, server 106 may update the permanent virtual world system with at least one modified real-time 3D virtual replica 104, as shown at block 616. The user device 110 can proceed with the output of the updated permanent virtual world system 118 including the real-time 3D virtual replica 104 that may have undergone a change, as shown in block 618. And performing a lightweight operation on the received media stream to properly display the updated real-time 3D virtual replica 104 on the device 110. The method 600 then checks, at inspection 620, whether there are more manipulation instructions coming from the user device 110, and then the method 600 determines that the user 114 has a real-time 3D virtual copy 104. ) Continues through the user device 110 to return to block 608. If no further instructions, as shown at end 622, the process may end.

7 illustrates a flowchart of a server implementation method 700 in accordance with one embodiment of the present invention that enables bidirectional manipulation of real-time 3D virtual replicas and real objects, in accordance with an embodiment of the present invention. The method 700 includes, for example, a method 500 for manipulating a real time 3D virtual replica through direct manipulation of the real object shown in FIG. 5 and a real object through direct manipulation of the real time 3D virtual replica depicted in FIG. 6. It is possible to incorporate a method 600 of manipulating. In some embodiments, method 700 may be executed by a system in accordance with one embodiment of the present disclosure, such as the system described in connection with FIGS. 1A-4.

The method 700 may begin at steps 702 and 704 by providing a permanent virtual world system that includes a structure in which at least one real-time 3D virtual copy of at least one real object is represented. Next, at block 706, the method 700 includes at least one real-time 3D virtual copy and at least one corresponding real object that sends feedback back to the real-time 3D virtual copy to augment the real-time 3D virtual copy. Continue at block 708 by synchronizing, increasing the accuracy of the real-time 3D virtual replica on the real object, and providing specific physical properties to the real-time 3D virtual replica based on the synchronization.

The method 700 proceeds with the reception of a selection and / or manipulation command entered via either the real object or the user device at block 710, and then for each real object and real-time 3D virtual replica at step 712. Proceed with processing and execution of the selection and / or operation commands. In some embodiments, some processing by the server may be used to support processing performed locally by the real object. The method 700 continues by updating the virtual world system with the modified one or more real-time 3D virtual replicas stored and calculated on the server, as shown in step 714, and sending the updated model to the computing device. The method then checks at check 716 whether there are more instructions, and if so, the method 700 returns to block 710. Otherwise, the method 700 may end at the end 718.

While specific embodiments have been shown and described in the accompanying drawings, such embodiments are merely illustrative of the invention and various limitations may occur to those skilled in the art, and the invention is not limited to the invention, and the invention is not limited to the specific structures and arrangements shown and described. It is not limited to. Accordingly, the description is to be regarded as illustrative instead of restrictive.

Claims (20)

A system that enables two-way interactive manipulation of real-time 3D virtual replicas and real objects,
A permanent virtual world system comprising a data structure in which at least one real-time 3D virtual copy of at least one corresponding real object is represented, the permanent virtual being stored and calculated on a server comprising a memory and at least one processor World system;
At least one corresponding real object communicatively and permanently connected to the at least one real-time 3D virtual replica via a network via the permanent virtual world system stored and calculated on the server; And
At least one user device communicatively and permanently connected to said at least one corresponding real object via said network via said permanent virtual world system stored and calculated on said server,
The at least one real time 3D virtual copy is the at least one corresponding real time through a plurality of sensing mechanisms providing a plurality of data points shared between the at least one real object and the at least one corresponding real time 3D virtual copy. A virtual physics property and a virtual world coordinate of the at least one real time 3D virtual copy synchronized with an object, wherein the virtual physics property and virtual world coordinates correspond to physical properties and real world coordinates of the at least one corresponding real object A system that enables bidirectional interactive manipulation of real objects. 2. The method of claim 1, wherein validating one or more changes on the real time 3D virtual copy via the at least one user device after virtually selecting a real time 3D virtual copy is effected on the at least one corresponding real object. A system that enables two-way interactive manipulation of real-time 3D virtual replicas and real objects, characterized in that it causes a. 2. The method of claim 1, wherein validating one or more changes on the at least one corresponding real object results in a real time corresponding effect on the corresponding real time 3D virtual replica. System that enables two-way interactive operation. 2. The real-time 3D virtual replica of claim 1, wherein manipulating the at least one real-time 3D virtual replica or the corresponding at least one real object produces a change in context data affecting a virtual-real pair. And a system that enables bidirectional interactive manipulation of real objects. 10. The system of claim 1, wherein the permanent virtual world system is shared by two or more users over a network. The method of claim 1, wherein the synchronization comprises providing feedback to the real-time 3D virtual replica to augment the real-time 3D virtual replica and provide additional physical properties of the corresponding real object, wherein the synchronization also includes a virtual sensor. , By using virtual resources, or combinations thereof, to enable reinforcement of the one or more real objects through the corresponding real time 3D virtual replicas. System. The method of claim 1, wherein the change that is validated for one of the real-time 3D virtual replica or the real object is a rotational movement, translational movement, selection of one or more behaviors, programming of one or more behaviors or setting of one or more parameters, or a combination thereof. A system that enables two-way interactive manipulation of real-time 3D virtual replicas and real objects, including a combination. The method of claim 1, wherein the server uses artificial intelligence algorithms and group analysis to help the user manage a plurality of real-time 3D virtual replicas in concert and autonomously, or to manage the corresponding real objects. Further configured, the artificial intelligence algorithm also enables cooperation and interaction between the real-time 3D virtual replicas based on one or more goals of generating corresponding cooperation between the corresponding real objects. A system that enables bidirectional interactive manipulation of virtual replicas and real objects. The system of claim 1, wherein the memory is further configured to store an event in the permanent virtual world system, wherein data from the event is used by the event detection module to detect and play back the event for later review. A system that enables interactive interactive manipulation of real-time 3D virtual replicas and real objects. A method that enables interactive interactive manipulation of real-time 3D virtual replicas and real objects,
Providing a permanent virtual world system on a server, the persistent virtual world system comprising a data structure in which at least one real-time 3D virtual copy of at least one corresponding real object is represented, the virtual physical property of the at least one real-time 3D virtual copy. And the virtual world coordinates correspond to physical properties and real world coordinates of the at least one corresponding real object;
Synchronizing the real-time 3D virtual copy with the at least one corresponding real object via a combination of sensing mechanisms, each mechanism sharing a plurality of data shared between the at least one corresponding real object and the real-time 3D virtual copy. Synchronizing said providing points;
Receiving a selection and / or manipulation command input via one or more interfaces on the at least one corresponding real object or user device, wherein the selection and / or manipulation command may modify modifications to a plurality of shared data points. Wherein the instructions are transmitted over a network via the permanent virtual world system stored and calculated on the server;
Processing the selection and / or manipulation commands; And
Updating the permanent virtual world system comprising at least one updated real-time 3D virtual replica. 11. The method of claim 10, wherein virtually selecting a real time 3D virtual copy and then validating one or more changes on the real time 3D virtual copy via the at least one user device is a real time corresponding effect on the at least one corresponding real object. A method for enabling two-way interactive manipulation of real-time 3D virtual replicas and real objects, characterized in that it results in a. 11. A two-way conversation between a real-time 3D virtual replica and a real object according to claim 10, wherein validating one or more changes on the at least one corresponding real object results in a real-time corresponding effect on the real-time 3D virtual replica. How to enable mold operation. 11. The method of claim 10, wherein manipulating the at least one real-time 3D virtual replica or the corresponding real object generates a change in context data of the real object, affecting the relationship between the real-time 3D virtual replica. How to enable interactive interactive manipulation of real-time 3D virtual replicas and real objects. 13. The method of claim 12, wherein the permanent virtual world system is shared by two or more users over a network. 15. The method of claim 14, wherein the user is a human or artificial intelligence user. 11. The method of claim 10, further comprising validating one or more changes on one of the real-time 3D virtual replica or the real object, wherein the one or more changes comprise rotational motion, translational motion, selection of one or more behaviors, one or more changes. A method for enabling two-way interactive manipulation of real-time 3D virtual replicas and real objects, comprising programming of actions or setting of one or more parameters, or a combination thereof. 11. The real-time 3D virtual replica and the real of claim 10, wherein the synchronization further augments the one or more real objects via the corresponding real-time 3D virtual replica by using a virtual sensor, virtual resource, or combinations thereof. How to enable bidirectional interactive manipulation of an object. 13. The method of claim 12, wherein the at least one corresponding real object is a factory machine or vehicle. 13. The method of claim 12, further comprising using artificial intelligence algorithms and group analysis to help the user manage a plurality of real-time 3D virtual replicas in unison and autonomously, or for managing the corresponding real objects. And wherein the artificial intelligence algorithm also enables collaboration and interaction between the real-time 3D virtual replicas based on one or more goals of creating corresponding collaborations between the corresponding real objects. How to enable two-way interactive manipulation of real objects. 11. The method of claim 10, further comprising storing an event in the virtual world system in a memory of the server, wherein data from the event is used by the event detection module to detect and play back the events for further review. A method for enabling two-way interactive manipulation of real-time 3D virtual replicas and real objects.

Images