BR102015030766A2 - associated methods in virtual reality and augmented reality for building low cost simulators, experimental activities or training - Google Patents

Abstract

A real environment with objects for interaction (110) is digitized through a virtual scenario creation method (300), generating a content composed of virtual scenario and 3d objects (130). this content is displayed through a display apparatus (200), for example oculus or google cardboard, to a user (150) who has his head movements captured and converted to visual navigation using a method (500). In addition, according to the range and speed of this movement, retractable display panels (700) may be displayed for the purpose of providing guidance and enabling, through a method of receiving commands and selections (800), to perform activities. by preprogrammed triggers within the environment (900). furthermore, this invention is also characterized by an object interaction method (600) that unites virtual reality with augmented reality to allow manipulation of 3d objects within the virtual reality environment without the need for acquisition or connection to other control devices by using markers (170) from an electronic digital content platform, ensuring that the manipulated object can be exchanged in the virtual environment with just one touch.

Description

"METHODS ASSOCIATED WITH VIRTUAL REALITY AND INCREASED REALITY FOR BUILDING LOW-COST SIMULATORS, EXPERIMENTAL TRAINING OR TRAINING" low-cost, portable simulators in the fields of virtual reality, augmented reality, and intelligent content distribution, which enables navigation in a virtual environment to display and manipulate objects in three dimensions.

[002] The methods employed in NEXXA allow object manipulation in the virtual reality scene to be done without gloves or connecting other devices, and are intended for lay users to perform experimental or training activities regardless of time and place. , intended primarily for education. Fundamentals of the Invention Augmented Reality refers to the process of capturing video data from an environment and enriching (or "augmenting") the video with computer generated graphics, typically making characters or objects unlikely to that environment. appear in the scene presented on video, giving the user the impression that that object exists in the real world or around the physical world.

For augmented reality to be pleasurable, the user experience needs to be transparent so that the placement of the object on the screen appears correctly positioned relative to the real world. In addition, augmented reality graphics must adjust in real time in response to user changes in the real world or camera movements integrated into the viewing device. This process requires accuracy in capturing and calculating the visual transformation of the marker associated with the object.

Virtual Reality begins in relation to stereoscopy for images. The earliest experiments date back to the first half of the nineteenth century, when Charles Wheatstone built the first equipment of its kind, which gave rise to the twentieth-century Holmes stereoscope that was popular in the second half of the same century and would be the equivalent of what it is today. known as virtual reality. In the 1940s, the View-Master stereoscope was responsible for spreading the embryo of three-dimensional technology to the population. The first Stereo Head Mounted Display (HMD) was built in 1965, but only in the 1980s did a portable virtual reality device emerge. The first successful device, US 5,253,832, built by Fakespace Lab in 1991, called BOOM Display is an example of a head-mounted virtual reality device.

[006] When using smartphone-attached devices to provide virtual reality experience, the first case was Hasbro's my3D, with a plastic accessory with two biconvex lenses and an iPhone 4 snap-in mold. The my3D command input solution was the presence of two holes on the underside of the docking cabinet, which made it possible for the user's fingertips to perform touch commands at the bottom of the screen.

[007] Virtual Reality technology integrated with smartphones opens up new possibilities for video, e-commerce and data visualization, especially for education and training, allowing the student immersive learning. For the right use of this technology, environments and interactions need to be realized in a way that users feel fully transported to the virtual environment.

However, with state-of-the-art technologies available, the virtual reality user has limitations to experience interaction with objects in immersive environments or can only do so through the use of expensive accessories. The manipulation of various objects present within the virtual scenario also does not have in the state of the art a simple solution from the point of view of the user experience.

Another limitation found in state of the art solutions is the preparation of environments, which are relatively static and do not change by performing activities by the user.

In addition, executing commands within the virtual environment is another hindrance to the spread of technology among the lay public. State-of-the-art solutions require the interconnection of various equipment to act as command triggers within the virtual scenario, and this arrangement is often complex to prepare and configure.

In terms of the digital logistics of interactive content delivery, there is a lack in the art of a distribution platform that allows a user with a specific interest to enjoy interactive virtual reality content.

To illustrate these difficulties, some of the prior art documents and their disadvantages are listed below.

[0013] Patent application published under US2010 / 0277575 entitled "METHOD AND APPARATUS FOR PROVIDING A 3D IMAGE VIA MEDIA DEVICE" describes a structure composed of prisms, lenses and lockable enclosure for the purpose of providing the Three-dimensional image display capability for portable media devices such as iPods, smartphones and palmtops. The three-dimensional effect is achieved by displaying two two-dimensional images on the device screen. Images that are static or continuous, in video form, were previously captured by two independent cameras spaced close to the human eye. These images, displayed stacked on the handheld screen, pass through a set of prisms responsible for vertical alignment and anamorphic adjustment. They then pass through a pair of focal adjustment lenses until they finally reach the eye of the wearer. The lens, prism and smartphone set comes with a case that can be made of plastic or disposable materials.

[0014] The apparatus and methods described in the document presented above are not able to solve the user problem that needs to be trained or qualified in a specific and interactive environment. A first reason is that it does not have navigation features in the environment. The user cannot choose the viewing angle of the 3D environment as the application does not describe the use of gyros or another alternative to identify the position of the user's head, nor does it envisage the use of content that provides interactive viewing from changes. directed by the user. Other drawbacks include, but are not limited to, the fact that it is a mid-cost array that employs a medium complexity optical array, involving the use of multiple prisms for changing horizontal to vertical alignment, which also results in higher volume. of the final set.

[0015] Patent application published under US20140267637A1 entitled "HYBRID STEREOSCOPIC VIEWING DEVICE" describes a hardware device intended to be superimposed on a tablet with an associated software platform. In this inventive concept the tablet screen is divided into upper and lower portion. The upper portion, in turn, is divided horizontally, so as to display side by side two images that will compose the three-dimensional effect. The lower portion of the screen is for the placement of buttons to control the virtual environment displayed on the upper portion. The hardware is mounted over the top of the screen and consists of a plastic screen that isolates the two displayed images so that each one is directed to one of the focal lenses and then to each of the user's eyes, transmitting the three-dimensional effect. .

[0016] This application already has a method for controlling the environment, including reorienting the cameras according to head movement, and interacting with objects. However, it does not solve the problem because: - It does not provide rich manipulation of the object in real time as the axial virtual controls act on only two axes, and since they are positioned at the bottom of the screen, a constant exchange of attention between the immersive environment and the operation of the controls. - Severely limits the number of pixels available for the generation of images that make up the virtual scene. Considering the use of an iPad mini, a tablet of renowned quality, less than half of the screen area will be dedicated, which means that a resolution of a mere 480x384px is available for image generation, while on a smartphone it would be 1920xl080px. This will happen to all tablets as they have low pixel density compared to smartphones. The result will be a poor user experience with less than 10% of the quality of a smartphone. - Being a hardware dedicated exclusively to the use with tablets, it has a high cost, requiring the user or student already have this equipment, which is not a possible solution in a scale environment.

Patent application published under EP2626834A2 describes a book for use with augmented reality applications. The book is composed of several semi-rigid sheets, all with marking patterns on the edges and a non-asymmetrical pattern figure in the center, capable of indicating for an augmented reality application, what is the book's size and position orientation scale, which by in turn represents an object for each page.

Although the above document describes the visualization effect of a virtual object and allows it to be viewed from various angles, comprising the three Cartesian axes plus the depth (distance) of the element, it has no degree of immersion with the environment and does not allow the manipulation of the object by hand, leaving open the problem of virtual distance training, in the sense that it demands not only the effect of visualizing an object in augmented reality, but a continuous and iterative process with manipulation. about multiple objects in a virtual reality environment.

[0019] Patent application published under number BRl020120199440A2 describes a set of methods for augmented reality visualization of a three-dimensional architectural project using a smartphone or tablet (device). The architectural model should preferably be viewed with the device pointing to the terrain. One of the methods described in the document allows capturing the values provided by the smartphone or tablet accelerometer sensor, which change as the tablet moves in the user's hand, to change the view of the 3D object inserted in the real scene, in this case an architectural project about a ground. Other methods described in the document include: - The "walk" shift through the three-dimensional object, which can be done using buttons on the smartphone or tablet screen itself. - Selection from one of several architectural projects, extracted from the database. - Jumps between various points of view of architectural design, requiring a long navigation.

[0020] The above application does not solve the problem of experimental distance training, as it presents to the user only a three-dimensional object over the real-world images captured by the device's camera, leaving both ambiance in the scene open. virtual training about the interactions between this scene and the correlated objects.

Patent application published under the number BR1020130273554A2 discloses the "VIRTUAL REALITY DISPLAY SYSTEM" consisting of a process and an apparatus. The apparatus is described by different configuration possibilities and computing, video and sensor components. Computing possibilities include simple processors and arrangement of various networked computers. For the display of images, the document provides head-mounted display systems, systems consisting of individual electronic shutters for each eye, systems that adopt polarized lenses, retinal display systems and even contact lens display systems. For sensors that determine the interaction between the user and the virtual scene, the document provides eye tracking systems, head tracking systems, gloves that transmit data according to finger movement, and even gesture tracking systems in the air.

Although the document described above allows the construction of an apparatus that enables the user to visualize a virtual reality world as well as the manipulation of objects in it, its assembly complexity includes the preparation of a structure with motion detection sensors. and / or implants requiring medical intervention. Another hindering factor is the cost of the solution that is not a limiter for simulators purchased by aerospace companies, but is impractical for everyday use by a popular in your home. In addition, the document does not provide a solution for mass distribution of new environments and interactive objects that configure a current need in the education market.

US 7,215,322 B2 discloses "INPUT DEVICES FOR AUGMENTED REALITY APPLICATIONS" and comprises an array of elements that are fixed to a surface and a method of recognizing and triggering commands. The document describes a solution for performing a command interface when a user has a head-mounted screen that can be a virtual reality goggle. This method consists of attaching small opaque disks to a flat surface, typically a wall, and by mapping the arrangement of these disks, associating pushbuttons on an interface that is displayed within the virtual reality environment. The buttons on this virtual interface are triggered when the user's hand moves in the real environment, covering the attached disks, so that the head-mounted camera no longer sees the disk.

[0024] The method described by the above patent alone provides a solution to the problem of executing commands within the virtual reality environment, given that the user has his visual field occupied by displaying the scene on the head mounted screen. . This solution is interesting because it does not require the acquisition, interconnection and configuration of a joystick or other interaction device. However, it is unsuitable for home use as it requires the lay user to install and calibrate the disks, and requires an object-free wall to be available for interactive use. This requires use in a predetermined location, preventing the goal of use anywhere at any time (portability). Another aspect that prevents it from being adopted as a command interface solution when using experimental learning and training simulators is that it requires the user to always be facing the wall, preventing interactions with objects behind his back. , for example.

Patent application published under US20070038944A1 describes an augmented reality system which includes means for capturing real environment image data, means for processing images for virtual environment generation, means for identifying objects. real images and processing of their images to be used as markers in the virtual environment and finally means that allow the overlay of the actual object image data with the virtual image in a virtual position corresponding to that of the predefined marker object.

The system described by the patent comprises, in addition to the image capture means, which may consist of the use of cameras, and the marker identification algorithms already present in the state of the art, a tracking system, a measurement system of distances and a rendering system. The document argues that the combination of these three systems allows the construction of virtual scenarios and the positioning of objects with greater accuracy. However, the method described requires the acquisition and configuration of several elements before each capture, which makes the process time consuming, costly and dependent on highly qualified personnel, not being a solution to the problem that arises for a platform. building virtual scenarios for large-scale training and education.

[0027] Patent application published under US20140071164A1 "CONTROLLING AN AUGMENTED REALITY OBJECT" describes a method of interacting with objects in augmented reality by using cameras and methods for analyzing and recognizing gesture produced by users' hands. and its intervention on objects displayed on the display device screen. Similarly, application BRPI0904132A2 entitled "VIRTUAL REALITY ELECTRONIC DEVICE AND VIRTUAL REALITY CREATION METHOD" describes interactions between real-world images and three-dimensional objects on a device screen.

However, in both requests there is no direct relationship between the use of an augmented reality marker to control an object in a virtual scenario, which is different from displaying an actual scenario as a background as claimed in the present invention. . This also means that both documents do not represent a solution to the problem of large-scale and distance experimental training. In addition, it is well known that the use of gesture recognition in control interfaces often implies the use of specialized hardware and a greater amount of computational power, which would make it impossible for lay users to use only their personal devices ( phones and tablets).

Patent application published under the number W02014070120A2 describes an interaction method using augmented reality, with the purpose of displaying a high level of detail in virtual objects and, as a consequence, allowing the visualization of small parts in a large object. proportion. The document describes how a marker is captured by a device, such as a smartphone, tablet or PDA camera, and interpreted by the method that now displays the augmented reality object on the device screen. As the device shifts relative to the marker, the three-dimensional object of augmented reality follows the same pattern. The proposed technique claims that its differential is the possibility of approaching the marker up to 0.01 of its total fraction, which allows the experience to perceive rich details of the object.

Although the above document presents a solution for augmented reality object visualization, it is not sufficient to solve the training and education problem where the student needs to interact with objects in a virtual setting as if they were there. .

[0031] Patent application published under number KR2014004029 entitled "METHOD AND APPARATUS FOR PROVIDING CONTENTS INCLUDING AUGMENTED REALITY INFORMATION" describes a method for inserting previously captured information and thus generating new content for augmented reality. Although the document includes a solution for augmented reality content capture, enrichment, and storage, it does not provide for the interaction between augmented reality and virtual reality and thus does not provide a solution to the exposed problem of virtual experiential teaching / training.

Regarding the virtual reality image display device, it is known from the state of the art to use Head Mounted Displays (HMD) which are screens built on displays that are worn by the user in a similar manner to that of a face mask. diving. One of the documents that can be used as a reference is US D701,206 which describes Oculus VR's "VIRTUAL REALITY HEADSET". Traditional HMDs consist of hardware that integrates the mask structure, the electronics and a small liquid crystal panel. This set has an external wired connection for power and to allow communication and data transfer from a computer.

[0033] This arrangement has a number of drawbacks, such as: - Need to purchase specific use hardware. - Need to have a powerful computer to use. - Depends on an application setup and installation process. - Absence of a platform for automatic and personalized download of contents. - High cost.

Other alternatives are available in the state of the art that serve the purpose of stereoscopic image display for use with NEXXA.

Preferably the devices that adapt smartphones for head mounting can be adopted. Examples of these gadgets include: - Samsung Gear VR: Compatible with some smartphone models Samsung has medium acquisition cost. - Beenoculus, BR3020140061054, a Brazilian design that uses foam adapters to fit a wide range of smartphones. It has low cost. - Google Cardboard: A Google project that has been publicly available, allowing you to create your own virtual reality device from a cardboard box, and is virtually compatible with any smartphone that has the hardware requirements needed to run desktop applications. virtual reality. It has acquisition cost close to zero.

Aiming at solving the education / training problem at scale, the Cardboard project proves to be an easy-to-implement alternative to providing only the display of stereoscopic images, as it uses regular cardboard to make up the virtual reality glasses case, as well as to be broadly compatible with commercially available smartphones.

However, within the scope of executing the action command within the virtual environment, none of the three head mount apparatuses presents a simple and widely compatible solution. Although the Samsung Gear VR implements a touch button on the side, it is limited to use with only three smartphone models from the same manufacturer. The Google Cardboard, on the other hand, has a magnetic trigger, implemented by a neodymium ring, affixed to the side of the cardboard box that when moved causes the smartphone's magnetometer to detect the change of the magnetic field that is transformed into a trigger signal. But this solution has the drawback of only working on smartphones where the magnetometer is positioned on top of the device. Another possibility available in the prior art is to connect an external command device, such as a remote control, joystick or mouse and keyboard, which has drawbacks such as the need for connections (wired or wireless), setup instructions and purchase cost.

To solve this problem, Nexxa not only allows the use of the above three alternatives, but also presents a method that brings two new ways of executing commands within Virtual Reality. Related Art The aim of this invention is to provide lay users with a technological alternative that enables them to simulate in virtual reality, by manipulating objects, performing experimental activities in environments that could not be replicated both at home and in the living room. in a very inexpensive and easy-to-use alternative.

[0039] This simulation in virtual reality allows, with the method used, the total immersion of the user who sees himself inside the scene, since his entire visual field is covered by the projection of images, allowing to explore distant locations as if he were there. This method overcomes the problem of alternatives present in the state of the art as it allows the use, not only of a specific model of HMD, but a profusion of existing models in the state of the art, standing out for their compatibility.

In addition to the advantages of immersion, this invention provides a solution to the problem of three-dimensional object manipulation within the virtual environment. Few alternatives are available in the prior art. One is the use of sensor sleeves, which are unable to solve the problem presented due to the high cost, the need for wired interconnections and the complexity of the installation. Another possibility is the use of touch interfaces on a piece of the device screen, which would limit the use to tablets, not representing a scale solution. There is also the alternative of using a joystick control common in video games, but this alternative does not solve the problem as it adds cost to the final solution and increases the complexity of configuration between more than one device. Finally there would be the option of using sensors and hand gesture recognition techniques of the user, but which generally depend on specialized hardware and increased processing use to provide satisfactory results, which would limit the audience served, thus not configuring a solution to the problem.

Faced with the exhaustion of options available in the state of the art to solve the problem of manipulating three-dimensional objects in an immersive environment, this invention includes an innovative method that allows the mixing of virtual reality and augmented reality technologies, associating each one of them. interactive objects of the scene a marker, allowing the user to control in his hands and manipulate on all rotational axes of a spherical system (x, y, z, w) said object, in order to enable the manipulation of virtual objects that do not are available to the public at no cost and bringing simplicity to the solution.

[0042] Another need to solve the problem presented is, besides object manipulation, a method that allows choosing from a list of options presented in the virtual environment. In view of the fact that the present invention does not require the connection of joystick controllers and that it does not make use of other state of the art alternatives, it was necessary to develop an innovative method that would allow the display, listing and choice of options within the system. Virtual reality environment.

The solution to this problem is achieved by a set of methods that transmit visual orientation within the virtual scene through retractable panels, monitor head angle to scroll through the list of options, and analyze the audio captured by the microphone for interpretation of "click" type commands in the system.

A solution that can provide user feedback, configured in the form of a visual and sound feedback, for the steps successfully completed in the set of activities performed on the scene is essential for the use of this teaching and training resource. virtual. An illustrative example is a virtual operating room where a patient needs to undergo an intervention. The training of a surgical assistant could be done in stages, starting with the presentation of the procedure to be performed, the choice of the instrument for the first incision, the determination of the location, direction, depth and extension of the cut. After the student completes this step, an innovative method, called a trigger, recognizes the movement pattern and returns feedback, such as a message indicating that a score has been earned, a video is presented with the continuation of the surgical procedure. , or another video showing the invasion of some organ and consequent death of the patient.

Finally, it was identified that in order to make a solution to the experimental virtual reality teaching / training problem viable, it would be necessary to combine an alternative for the three-dimensional capture of the scenarios, which would allow speed, simplicity and low cost, essential characteristics for scaled content generation. For this purpose, an existing state-of-the-art solution has been designed for use with NEXXA which allows, with a mounting bracket, to assemble an array of low-cost wide-angle cameras that capture images of the environment from all angles and deliver this information. for a system that processes image stitching, automatically renders and creates the three-dimensional container.

In addition, this invention enables all of these properties to be achieved with low investment in technological resources, both on the side of the institution that produces and makes the content available, using a large-scale, personalized delivery distribution platform and student / user side, without the need to purchase expensive equipment and facilitated by methods that provide simplicity of installation, operation and maintenance, ensuring in particular that distance learning students can benefit from practical experiences that could only be carried out in specific laboratories or in environments with high cost simulators.

For the accomplishment of the foregoing and related purposes, one or more embodiments comprise the aspects to be described below and specifically defined in the claims. The following description and the accompanying drawings provide certain illustrative details of the aspect of the described embodiment. These aspects, however, indicate only some of the many ways in which the principles of various modalities can be used.

BRIEF DESCRIPTION OF THE FIGURES The features, nature and advantages of the present disclosure will be more apparent from the detailed description set forth below when read in conjunction with the drawings, in which the same references refer to the same elements, in which: ] Figure 1 is an illustration of a flowchart containing the methods and components that make up the NEXXA according to an illustrative embodiment.

[0050] Figure 2 is a block diagram of the elements that comprise a Virtual Reality display and control apparatus according to an illustrative embodiment.

[0051] Figure 3 is an illustration of a process flowchart for the method of capturing real environments and transforming them into a 3D Virtual Reality scenario container according to an illustrative embodiment.

[0052] Figure 4 is the first part of a flowchart illustration of a process for the method of navigation within virtual environments, showing initial user access and guidance according to an illustrative embodiment.

[0053] Figure 5 - is the second part of a flowchart illustration of a process for the method of navigation within virtual environments, showing the performance of sensors and influence of controls on the virtual scenario, according to an illustrative embodiment.

[0054] Figure 6 is an illustration of a process flowchart for the method of interacting with three-dimensional objects within the virtual scene using augmented reality markers according to an illustrative embodiment.

[0055] Figure 7 is an illustration of a process flowchart for the method of displaying floating panels and highlighting selections in a list within the virtual environment according to an illustrative embodiment.

Figure 8 is an illustration of a flowchart of a process for the method of capturing commands and choosing options within the virtual scene according to an illustrative embodiment.

[0057] Figure 9 is an illustration of a flowchart of a process for triggering method for pre-programmed activities within the virtual scene, according to an illustrative embodiment.

[0058] Figure 10 - is an illustration of the user making use of NEXXA, wearing the display and control apparatus and manipulating a handheld device displaying the augmented reality marker on the display, which enables the manipulation of three-dimensional objects within the virtual scene, according to an illustrative embodiment.

DETAILED DESCRIPTION OF THE INVENTION The present invention is composed of a set of six methods, which associated with a mechanical, electronic, optical and digital apparatus (200) allows a user, generally a student or apprentice, from from anywhere, total visual and auditory immersion, including the manipulation of three-dimensional objects, within a virtual environment for learning and / or performing experimental activities in restricted-access environments without having to do so. investments in equipment, nor prepare configurations

[0060] Reference is now made to Figure 1 illustrating a flowchart containing the methods and components that make up the present invention, in which a real environment with objects for interaction (110) is digitized through a virtual scenario creation method (300). ), generating content consisting of virtual scenery and 3D objects (130). This content is displayed through a display apparatus (200), such as Oculus or Google Cardboard, to a user (150). This user's navigation within the virtual environment is ensured by another method (500) which uses head movements to guide the visualization.

Factors such as the range and speed of user movements determine the display of retractable display panels (700) for the purpose of providing guidance. By the time the user already knows what actions to perform within the environment, a method of receiving commands and selections (800) receives inputs, while another method of triggers (900) monitors actions performed by triggering preprogrammed activities. within the environment.

In addition, this invention is also characterized by an object interaction method (600) that unites Virtual Reality with Augmented Reality to allow manipulation of 3D objects within the virtual reality environment without the need for acquisition or connection to other control devices. The developed alternative allows using camera capture of pre-registered figures, also known as markers (170), to guide the manipulation of 3D objects within the virtual scenario. These markers are delivered to the user by an electronic digital content delivery method (400), allowing various objects to be used within the scenario and to be exchanged with one tap.

Each of the methods cited above is covered in detail below, with references to the figures illustrating their flows. Simplified Method for Creating Virtual Environments from Real Scenarios As illustrated in Figure 3, a sequence of panoramic photographs of the environment is captured by a 360 degree camera (310), the technique of which is already known from the state of the art. These photographs are then grouped in a logical sequence on an editing platform (320), where they are snapped together so that they are continuously displayed in the three dimensions (330), resulting in a three-dimensional digital model of the original environment (340). With this three-dimensional model defined, 3D objects can be added to the scene. For this a treatment process is applied to the content. This process consists in the application of textures, definition of transparencies and reflections from the lighting points of the scene (350). The generation of perspectives, shadows and light of objects in the final scenario are obtained from trigonometric calculations (360). To create a virtual environment that is close to reality, high definition images and textures, virtual lamps, shadows and high polygon models are used. Visual navigation method within a virtual environment created by stereoscopic images applied to a fixture bracket and lens [0065] A visual navigation method in a virtual reality environment is used on the NEXXA platform, which comprises spectacles (200 ) composed of a rigid frame with space for placing a smartphone (210), two biconvex lenses (220), a shield (230) that separates the images transmitted by each lens to the respective eye, an elastic strap (240) to allow that the assembly is secured to the head, keeping hands free and a magnetic ring (250). The smartphone serves at the same time as a processing element (201), display screen (218), ambient audio output interface (214), image capture (216) for marker detection and object interaction, audio capture for input of controls (215) and gyro (217) to inform the position of the head.

[0066] An electronic platform for teaching material contains virtual reality content, preferably laboratories and their experimental activities, previously captured and associated with an anchor discipline.

The user accesses the teaching materials platform (410) routinely for his studies. When this access is made, only the contents of the subjects associated with their enrollment are displayed (420). As content is read (430), a page may appear with an interactive experimental activity (440). When this occurs, the guidelines and requirements for using NEXXA (450) are displayed and the user can decide (460) whether to go to Virtual Reality or to continue reading the traditional material.

[0068] If you choose to enter Virtual Reality, scenario data (RV) and interactive 3D objects and their markers (470) are loaded.

After loading the data, the device's gyroscope and magnetometer (compass) are initialized (510), preferably a smartphone, then the device's initial angles and sensor starting data are captured ( 520) to set the initial placement and perform the virtual environment start rendering (530).

The data generated by the gyroscope is monitored for the purpose of capturing a change in device angle (540), which is reflecting the movements of the user's head. When the gyro data changes, its new value is used to update the position of the virtual cameras (541) and a new scene rendering is done (551). This continuous process guarantees the user the sensation of being immersed in virtual reality.

[0071] To enable "walking" a new solution has been implemented that eliminates the inconvenience of acquiring and configuring joysticks or having to look at a fixed position on the stage in order to push a virtual button. This new solution makes it possible to use the headset that comes with all smartphones sold in the market. These headsets by default feature 3 buttons that are used to answer, increase and decrease call volume.

This virtual walking method that is implemented on the NEXXA platform intercepts the interruptions generated by the headset buttons and associates the Vol + command with the camera shift forward (561); the backward camera shift command (562) and the 'answer' button, which is the call answer button, is associated with the ΌΚ 'or' Enter 'command depending on the context of the scenario. For some older headsets where there is only the call answer button, this is associated with forward camera movement.

As cameras are moved by pressing the headset buttons, new renderings of the virtual environment are made (551), giving the user the feeling of virtual walking.

[0074] These changes can be noticed as the sequence of images generated by the virtual environment capture method is used to assemble a 3D container with an internally applied virtual scenario. And to this scenario are associated the four positional coordinates, sent by the smartphone gyroscope.

The image is displayed split on the smartphone screen to generate the parallax effect of the environment captured with the camera matrix, rotated and rotated according to the movement of the head, captured by the gyroscope. In the navigation method described above, every 50ms is obtained the gyro rotation matrix which is transformed into a textual representation.

[0076] This string is sent to a transformation process that returns a 4x4 matrix, later converted to a Quaternion.

The Quaternion defines the position of the two virtual cameras that represent the human eye and result in the parallax effect to achieve visual stereoscopy and the sensation of immersion in a 3D environment by the user. Method of interaction with three-dimensional objects within a virtual reality environment, using predefined figures [0078] In order to allow the existence of a virtual reality environment that simulated the experience of a real practice laboratory, it was necessary besides navigating through the virtual environment the manipulation of objects present in the scene. For this, we used, in an innovative way, a method existing in the state of the art, which associates still images with 3D objects, popularly known as Augmented Reality.

A previously defined set of objects is modeled in three dimensions. Each of these objects is associated with a two-dimensional image. This group of images and associations that can be manipulated in the scene, we call interactive object base, and a device with camera, screen and processing is used (smartphone, tablet, notebook or computer).

[0080] When the 2D image is displayed in front of the camera, a pattern recognition is made, and associated 3D object is displayed on the screen. As the image is manipulated forward, the device camera recognizes its distance and angle and applies the equivalent transformations to the 3D object, conveying the sensation of manipulation.

When the user of the digital learning platform chooses to enter interactive content (460) the scenery and object data (470) is loaded and the device camera (240) is initialized (601). After capturing the initial angles of the device (602), sensor data (603) and the initial rendering of the scene (604), the positioning quaternion in the virtual scene is monitored to check whether there is an interaction with any object in that visual ray ( 610). If so, a visual guidance guide for positioning and placing the marker in the user's hands with the correct position in the virtual scene (611) is displayed. When the actual marker image captured by the smartphone camera with the orientation guide in the virtual environment (620) overlaps, a search (621) compares the image captured by the camera with a marker previously registered in the base. interactive objects and the corresponding 3D object is loaded (622) and rendered within the scene (623). From this moment on the quaternion with position and angle of the 3D object is associated with the position and angle of the image captured by the camera and if the position of the image is changed (630) the position of the 3D object in the scene is updated (631).

In addition, every 50 milliseconds is checked if the image captured by the camera has been changed (640). If this occurs, the process of searching (621), loading (622), and displaying the 3D object (623) is repeated with the new object, giving the user the illusion that they are exchanging the manipulated object in their hand.

Displaying Retractable Status Panels with Highlighting from List Options in a Virtual Environment In order for the user to view details of their progress in activities performed in the virtual environment, it is necessary to display informational panels, which cannot disrupt the visualization. of elements that can be in virtually any position. As a solution to this problem, a method has been developed that displays a retractable panel whenever a particular gesture is made by the user with the head.

Once entered into the virtual environment (701) positional coordinates are recorded via the smartphone gyroscope that integrates the Virtual Reality device. When the user turns the head, the change in device angle (710) is noticed. If the change has occurred around the Y axis (711), it is checked whether it was in the range of 20 ° to 30 ° in modulus (can be positive or negative) (712). If so, check to see if the position remains in this range for at least 2 seconds (713), if this happens, and there is not yet a retractable panel displayed (714) it is displayed (715), otherwise what was in display is collapsed (716).

When a retractable panel is displayed (715) the linked content is loaded so that pertinent information appears to the user (720). If there is a linked list of options in this information, this list is displayed (730).

From the moment a 3D object or panel with information in a list is revealed in the virtual environment, a sequence of one or more slates relating to it and its interaction with the environment is displayed. These assignments can be met by choosing an option from a list that is displayed directly in the virtual environment or by performing other activities. To choose an option from a list, a method has been developed that continuously measures the angle of travel of the user's head around the X axis.

If an X-axis head rotation (740) is detected, the angle of this movement is analyzed. A 60 ° range is divided into four 15 ° stretches that correspond to the options available in the list. If the head is in range 1, the first option is highlighted graphically (751) and so on to range 4 (754).

After highlighting highlights an answer option, another method is in charge of sending the command to confirm the choice of answer.

Finally, the position of the virtual cameras (760), rendering of the virtual environment (771) and rendering of the floating signage panels (772) are updated and the method continues to operate until it is interrupted by pressing the exit button. (780). Electrically triggered command capture method: [0090] For entering action commands within the Virtual Reality environment, while the user is fully in the field of view, Nexxa presents two alternatives. strike any side of the Virtual Reality glasses, usually with the impact of your fingertip. The sound wave caused by percussion in the hollow structure is captured by the smartphone's integrated microphone (821). Detection of a command is given by constant analysis of the 'Τ' last amplitude peaks (824), which are discounted from a previously analyzed median threshold (823), which avoids false positives after a moment of silence. When two amplitude peaks with a value three times greater than the median threshold (825) are detected (827) at a 't' interval (826), a command equivalent to that of a trigger (828) is triggered.

The second possibility is the detection of a signal from the P2 connector used for headphones / headsets connection. A click on the headset 'answer' button is received by the smartphone hardware (820) and sent as a platform input and freckle that interprets as a trigger signal (828) the command of a virtual button.

Regardless of which method is used to receive the select command from the platform, the next step is to search for feedback recorded for the action performed (829). If there is a registered feedback (830), it is checked if it is of the nature of an associated activity (840). If so, it generates an activity execution trigger, described by another method below. Otherwise the message registered as feedback is sent for execution in a floating panel (841).

Trigger for performing an activity in a virtual environment An experimental activity needs to be performed in a confined space in the virtual environment, just as to hold a door handle close to it.

[0094] An existing problem is determining when and where an action will be performed in the virtual environment.

[0095] In the present method when the user moves his head, the coordinates of this movement are compared with those previously defined in virtual space (910). If these user camera position coordinates match those of a previously defined event, a trigger occurs and an activity is initiated, providing an experience for the user. A floating panel is displayed, with information on how to perform this interaction, for example, which markers can be used, or the description of an animation to display (911). The interactive event could be playing a 360 ° video, displaying a visual hint, or manipulating an object. If the display uses a marker (920) then the interaction method will be triggered. If the content is a 360 ° video (921), it will load and fit the scene (922). In case of displaying a visual hint (930) the user may alternate their markers so that it corresponds to that indicated in the visual hint.

When the physical or digital marker in the user's hand is aligned in the field of vision of virtual reality with the visual hint, a state-of-the-art mechanism (Vuforia SDK) recognizes the visual pattern. Consequently a nested event is triggered, such as displaying a second video or associating an object in virtual reality with the object in the user's hand.

The moment the object is associated with the physical or digital marker in the user's hand the visual cue disappears from the environment and only the virtual object is displayed.

It will be readily understood by those skilled in the art that modifications may be made to the invention without departing from the concepts set forth in the preceding description. Such modifications should be considered to be included within the scope of the invention. Accordingly, the particular embodiments described in detail above are illustrative only and not limitative of the scope of the invention, which should be given the full extent of the appended claims and any and all equivalents thereof.

Claims (15)

Method (300) for creating virtual environments from real scenarios, the method is characterized by the fact that it comprises: capturing (310) a sequence of panoramic photographs of the environment by a 360 degree camera; group (320) the photographs in a logical sequence on an editing platform; fitting (330) the photographs so that they are continuously displayed in the three dimensions, resulting in a three-dimensional digital model of the original environment (340); apply (350) textures, colors, transparencies and reflection from existing lighting points and the viewpoints through which the environment will be viewed in order to calculate the perspectives, shadows and light of objects; get (360) the final scenario. Method according to claim 1, characterized in that to create a virtual environment that is close to reality, high definition images and textures, virtual lamps, shadows and high polygon models are used. 3. Visual navigation method (400) for use on a didactic platform, FEATURED by the fact that it comprises: accessing (410) the didactic material platform by the user; enter (415) registration number; display (420) content of subjects associated with their enrollment; perform (430) content reading; execute (440) page with interactive content; display (450) guidelines and requirements for using virtual reality; proceed (460) with virtual reality; load (470) scenery (RV) data and interactive 3D objects and their markers. 4. Method (500) for loading (470) scenery (RV) data from a didactic platform, characterized by the fact that it comprises: initializing (510) gyroscope and the device's magnetometer (compass); capture (520) the initial angles of the device and the starting data of the sensors to set the initial positioning; perform (530) the initial rendering of the virtual environment; capturing (540) a change in device angle; update (541) the position of the virtual cameras; render (551) the scenario again. Method (500) according to claim 4, characterized in that updating (541) the position of the virtual cameras further comprises: moving the camera forward (561) using the Vol + button; move the camera backward (562) using the Vol- button; Press the 'answer' button to register the image. Method (500) according to Claim 4, characterized in that if in step (540) the angle of the device has not changed, the method further comprises: controlling (550) whether the headphone has been pressed; if not: render (551) the virtual environment; render (552) the floating signage panels; if so, return to step (541). Method (600) for loading (470) data from interactive 3D objects and their markers, characterized by the fact that it comprises: initializing (601) the camera; capture (602) initial device angles and sensor data; initially rendering (604) the scene to monitor the placement quaternion in the virtual scene; verify (610) the existence of interaction in the virtual scene's range of vision; display (611) a visual orientation guide for positioning and fitting the marker in the user's hands; detect (620) marker using camera; seek (621) association of the captured image based on interactive objects, when comparing the image captured by the camera with a marker previously registered in the interactive objects base; load (622) the corresponding 3D object; render (623) the 3D object within the scene; check (630) if the marker has changed angle or position; update (631) position of the 3D object to get the 3D object within the scene. A method according to claim 7, characterized in that if in step (630) the marker did not change the angle or position, the method further comprises: verifying (640) whether the displayed marker is different; if so, return to step (621); if not, press (501) the exit button. 9. Method (700) for displaying retractable status panels highlighting list options in a virtual environment, characterized by the fact that it comprises: entering (701) the virtual environment and recording positional coordinates via the smartphone's gyro; check (710) change in device angle; check (711) change around Y axis; checking (712) angle between a range of 20 ° to 30 ° in modulus; check (713) whether the position remains within this range for at least 2 seconds; check (714) for retractable display panel; retract (716) retractable panel; check (725) for linked picklist; display (730) list; check (740) rotation about X axis; highlight (751, 752, 753, 754) at least one desired option. Method (700) according to claim 9, characterized in that if in step (710) the angle of the device has not been altered, the method further comprises: rendering (771) the virtual environment; render (772) floating signage panels; press (501) the button to exit. Method (700) according to Claim 9, characterized in that if in step (714) there is no retractable panel on display, the method further comprises: displaying (715) retractable panel; upload (720) matching content; repeat steps 725 through 754. 12. Method (800, 900) for capturing electrically triggered commands, FEATURED by the fact that it comprises: pressing (820) the 'answer' button; capture (821) microphone audio in 'T' range; analyze (822) median audio threshold in the range 'Τ'; initiate (823) discount from median level of audio curve; analyze (824) amplitude of the discounted audio curve in 'Τ'; detect (825) amplitude with 3x the median plateau at 'Τ'; analyze (826) audio at 't' interval; detect (827) amplitude with 3x the median plateau at 'Τ'; send (828) platform select command; seek (829) if there is feedback for the action to be taken; check (830) for registered feedback; check (840) if feedback is an associated activity; point (910) the camera to the right position; display (911) floating panel with interaction information; interact (920) using marker; initialize (601) the camera; capture initial angles of device (602), sensor data (603); initially render (604) the scene; monitor the positioning quaternion in the virtual scenario; verify (610) the existence of interaction in the virtual scene's range of vision; display (611) a visual orientation guide for positioning and fitting the marker in the user's hands; detect (620) marker using camera; seek (621) association of the captured image based on interactive objects, when comparing the image captured by the camera with a marker previously registered in the interactive objects base; load (622) the corresponding 3D object; render (623) the 3D object within the scene; check (630) whether the angle or position of the marker has changed; update (631) position of the 3D object to get the 3D object within the scene. Method (800, 900) according to claim 12, characterized in that if steps (825, 827) do not detect amplitude with 3x median plateau of 'Τ', the method returns to step (820 ). Method (800, 900) according to claim 12, characterized in that if in step (840), feedback is not considered an associated activity, the method comprises: sending (841) the registered message to the floating panel; update (760) position of virtual cameras; render (771) the virtual environment; render (772) the floating signage panels. Method (800, 900) according to Claim 12, characterized in that if step (920) does not use a marker, the method comprises: verifying that the interaction is a 360 ° video; if so: play (922) 360 ° video in the virtual scene; display (930) visual orientation; update (760) position of virtual cameras; render (771) the virtual environment; render (772) the floating signage panels if not: display (930) visual orientation; update (760) position of virtual cameras; render (771) the virtual environment; render (772) the floating signage panels.