WO2022018480A1

WO2022018480A1 - Methods for augmented haptic and immersive interaction

Info

Publication number: WO2022018480A1
Application number: PCT/IB2020/056782
Authority: WO
Inventors: Julio Alberto Mendoza Mendoza; Hazur Sahib SOCCONINI ALVARADO; Gabriel SEPÚLVEDA CERVANTES; Víctor Javier GONZÁLEZ VILLELA; Chadwick CARRETO ARELLANO
Original assignee: Julio Alberto Mendoza Mendoza; Socconini Alvarado Hazur Sahib; Sepúlveda Cervantes Gabriel
Priority date: 2020-07-19
Filing date: 2020-07-19
Publication date: 2022-01-27
Also published as: MX2023000463A

Abstract

The present invention relates to a set of methods for use in portable systems for augmented physical interaction using haptic and immersive feedback devices for individual or group applications. The uses may be entertainment, medical, remote working, training, communication and the Internet of things, in videogames and in the process of the virtual design of pieces and components, among other uses, including portable applications. One of the greatest advantages of these methods is improved use of services and products without needing to be present, as in the case of social contingencies, or without needing to have space or money for equivalent physical products. The user or users are chosen, without limit, from human beings, any non-human entity, such as an animal, a robot, an android or an artificial intelligence, or a combination of all of the above.

Description

HAPTIC AND IMMERSIVE AUGMENTED INTERACTION METHODS FIELD OF THE INVENTION

The present invention relates to a set of haptic and immersive augmented interaction methods for use in portable augmented physical interaction systems using feedback-enabled haptic and immersive devices for individual or collective applications. The uses can be for entertainment, medical, remote work, training, communications and internet of things, in video games and in the virtual design process of parts and components, among others. Including portable applications. One of the greatest advantages of these methods is the improvement in the use of services and products without having to be in person, as in the case of social contingencies, or without having to have space or money for equivalent physical products. The user or users are chosen without limitation from among human beings, any non-human entity such as an animal, a robot, an android, an artificial intelligence or a combination of all of the aforementioned.

BACKGROUND OF THE INVENTION

From the state of the art, six technologies have been identified that precede the present invention and these are:

one . Augmented reality: allows one or more users to mix the real world with added graphics.

2. Haptic interfaces: the science called haptics allows users to extend their sense of touch to virtual objects. Such interfaces may have force augmentation configurations or exoskeletons.

3. Teleoperation: It is the interaction with a remote system or machine.

4. Position sensors: these allow monitoring the position and orientation of an object of interest, for example, one or several parts of a human body.

5. Telemetry interfaces: These devices allow sending and receiving information in a wired or wireless way between 2 or more processing units.

6. Processing units: they are responsible for the control and processing of the data and information of the rest of the components described, their core is a microcontroller or microprocessor.

Some notable and similar works are:

1. INROL Laboratory South Korea: where a virtual object immersed in a virtual environment is controlled with multiple users through haptic teleoperation (hftps://www.inrol.snu.ac.kr/haptics-yr).

2. Heudiasyc CNRS Abaunza-Castillo: where a real quadcopter is controlled from a virtual quadcopter, immersing the user in a virtual environment and with haptic feedback to the user.

“Teleoperation of a drone based in a virtual environment” (https://wyvwyoutube co ni/watch?y hnjpyi/UHLa4). 3. SONY INTERACTIVE ENTERTAINMENT INC with patents WO2016/205143 and US10296086B2 in which they do not claim specific or detailed applications of augmented reality mixed with the real world, but which make use of the same components indicated here.

It should be noted that, in the first two cases mentioned, the user's measurement system is external to him, that is, they are non-portable cameras placed outside the user's body, while documents WO2016/205143 and US10296086B2 describe portable variants, being these are the closest documents to our invention described in this document. However, in said documents, the methods of use object of the present invention and above all technological applications that use augmented reality are not claimed.

SUMMARY

The present invention relates to a set of methods adapted to use a portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices comprising: at least one wearable haptic interface (4), at least one portable unit of processing (6), at least one sensor on board the haptic interfaces (4a), at least one portable sensor external to the haptic interface that can be placed in any other part of the user's body (2) or outside of it as preferred, at least one telemetry equipment (7), at least one augmented reality immersive viewer (3), at least one virtual object augmented over the real world (5), at least one world base (1) which can be portable or fixed. The uses can be entertainment, medical, remote work, training, communications and Internet of Things, in video games, or in the virtual design process of parts and components among others, including portable applications. One of the greatest advantages of these methods is the improvement in the use of services and products without having to be in person, as in the case of social contingencies, or without having to have space or money for physical products. From the current state of the art, it is known that most of the existing user measurement systems are external to the user, that is, they are non-portable cameras placed outside the user's body. It should be noted that the invention can be extended to its application for non-human users, which can be other forms of life such as animals for veterinary applications, or the users can be artificial intelligences or robots.

Other aspects of the invention will become apparent from the detailed description, taken in conjunction with the figures, illustrating, by way of example, the principles of the invention.

BRIEF DESCRIPTION OF THE FIGURES

Figure 1 shows the usual physical components of the augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices and an illustration for its use. in the tactile visual augmented sports entertainment method. The figure illustrates with two human users playing table tennis.

Figure 2 illustrates the use of the augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices in different geographical locations, that is, through teleoperation and coordination of at least one world base for each user. The figure illustrates two human users playing augmented and tactile table tennis in two different geographical locations.

Figure 3 illustrates an application of the augmented physical interaction system for individual or collective applications with a user and their mobile device, which can be without limitation an electronic tablet or a cell phone or smartphone, note that said mobile device already embeds several components and that said application can be used on board a vehicle or for some recreational or educational purpose, in this case the user's interaction with a virtual touch menu of his mobile phone without ceasing to be aware of the real environment that surrounds him, in this case In this case, the haptic device functions as a mouse or three-dimensional mouse, the user being able to choose between haptic interaction for two-way tactile feedback (towards the user and towards the augmented object) total, limited, intermittent, or null. Figure 3 illustrates a user walking in a physical environment (the location of a road is illustrated), said user uses his mobile (6) in this case placed on his waist and visualizes and touches an enlarged object (5).

Figure 4 is an extension of figure 3 where more than one user can share and feel a virtual object between them, this can be a plan or design of an object or a simple video game avatar, in this case both users decide who it is the base world or else it changes via request (a button to receive and send). Figure 4 illustrates a case similar to that of figure 3 but with two users.

Figure 5 is an illustration for the method that allows the editing, visualization and/or design of virtual and augmented objects to share said objects between a team of work users or a single user, using CAD/CAM software for this purpose. The figure illustrates a couple of users editing from their desks and computers an augmented object in a common room.

Figure 6 is an illustration for the method that allows multiple control of a robot or vehicle using an augmented object representative thereof. The figure illustrates a pair of users altering a common augmented object, which at the same time alters the behavior of a robot.

Figure 7 illustrates all the implicit coordinate frames: the haptic device external monitoring and sensor frame (Xse, Yse, Zse), the haptic device frame (Xh, Yh, Zh), the viewer frame (Xv, Yv, Zv), that of the world base (Xb, Yb, Zb), that of the augmented objects (Xo, Yo, Zo) and that of the user's computer or processor (Xu, Yu, Zu). Figure 7 takes up the example of users playing tennis augmented by visual touch. DETAILED DESCRIPTION OF THE INVENTION

The present invention relates to a set of haptic and immersive augmented interaction methods, for use in a portable augmented physical interaction system with haptic and immersive feedback devices for individual or collective applications. The present invention extends its application to human and non-human users, which may be other forms of life such as animals, androids, robots, or avatars or programming routines managed by artificial intelligence. The minimum components of such a system are:

1. At least one world base (1), this is a marker that is placed in a common work point for all users, this always indicates the position that each user has, this base is unique if the users occupy the same workspace or geographical location or multiple which allows tele-existence if the users are in different geographical locations, it is in charge of updating for each user the values related to the virtual object or serving as a global frame of reference.

2. At least one portable sensor (2) external to the haptic interface that is placed in any other part of the user's body. It is adapted to measure distances and relative positions of the haptic interface with respect to the frame of the immersive and global viewer in order to detect the overlapping of the wearable haptic interface with the virtual object. For example, in the case of using a haptic glove, there are sensors monitoring markers at specific points on the hand and fingers in order to detect if there is increased contact between the hand and the virtual object. The sensors are chosen without limitation, cameras, ultrasonic or radiofrequency arrays or non-motorized kinematic chains that incorporate joint sensors such as encoders, potentiometers, inertial measurement units (IMUS), etc., including optical, electrical or flow interruption sensors. Pressure.

3. At least one immersive viewer (3), these can be immersive or augmented reality glasses, in the former a video of the real world is taken and is projected to the user together with an overlay of images of the virtual object. In the second case, a virtual environment is superimposed on the real world through specialized lenses.

4. At least one wearable haptic interface (4), whose shape is chosen without limitation from among gloves, footwear, limbs, torso, hat or full body or a force-enhancing device such as an exoskeleton, in addition it is adapted to sense an augmented object and alter it, that is, a virtual object superimposed on reality.

5. At least one sensor on board the haptic interface (4a), these are adapted to detect velocities, accelerations and forces, both rotational and translational, of the part of the body covered by the wearable haptic interface and whose measurement allows behavior to be altered of the augmented virtual object.

6. At least one virtual object augmented on the real world (5), these virtual objects are placed superimposed on the real world (5), having the world base (1) as a reference.

7. At least one portable processing unit (6), adapted to perform calculations, measurements, haptic control, sending and receiving telemetric data, communication with other units and the world base, as well as the superimposition of the virtual object in the real world , each user carries one of them. 8. At least one telemetry equipment (7), this is adapted to send the data of each user, as well as of the world base (1) and of the virtual objects among themselves. they are two-way data communicators that can use wireless communication technologies such as Bluetooth or WiFi or any other available communication technology.

9. A set of position and orientation spatial coordinate frames, these are composed of three translation basis coordinates and three rotation basis coordinates) and are at least: a. The immersive viewer (Xv, Yv, Zv), this is the user-level reference; b. The world base (Xb, Yb, Zb), this is established by the world base marker and is placed at world zero or the position that said base has, it can also be established on board a user, which serves as a base of reference for other users, it can also have a dynamic tracking and transmission system as a radar and thereby reduce the sensors and emitters or, have as many sensors and emitters as there are users; c. The one of each user (Xu, Yu, Zu), this is established by each user and is measured with respect to the world base

OR); d. The one of the increased object (Xo, Yo, Zo), this is measured with respect to the base world and consequently each user; and. The haptic device (Xh, Yh, Zh), this is measured at the user level, with respect to the viewer and with respect to the external monitoring device, it is also considered as an end effector; F. The external sensor and monitoring of the haptic device (Xse, Yse, Zse), this is measured at the user level and is the link between the viewer and the haptic device. Between this sensor and the haptic device, the user forms a kinematic chain with his own body, which can be placed next to the viewer.

EXAMPLES OF USE OR APPLICATION OF THE INVENTION

The description of the methods for the use of said portable haptic and immersive system with its application examples to be protected are as follows:

one . Eligible Touch Augmented Sports Entertainment System: Since virtual objects are designable and do not need to be physically manufactured because they are virtual objects, they are programmed on a readable medium. You can have a sports menu where a team of players chooses which sport they want to practice, thus varying fields or virtual game objects (balls, mitts, etc.), this system even makes it possible to play in a team without the players being present in the same geographical location, only having a large enough space and an internet connection or any other data network. The method for using this system is as follows: a. At least one user chooses a pre-programmed sport of the augmented type; b. At least one fixed base unit loads said program and augmented interaction objects (nets, balls, rackets, etc.); c. At least one fixed base unit estimates the position of the users by means of the telemetry units and calculates the position of the augmented sports objects; d. At least one sensing unit of the haptic device together with the processing units of each user determine if there is contact between the position and orientation of the haptic device and the augmented object; and. At least one user interacts with the augmented sports objects by viewing them in their visors and altering their movement and force with haptic interfaces; F. Said alteration of force given by each user, through the on-board processing units and their respective telemetry units, is retransmitted to the world base or bases and with this it returns to point 3 of the method until the users or user decide to end the game, save and quit the game or change physical activity increased.

2. Augmented educational, labor or medical training tactile system: in a similar way to the previous one, but this time each student or apprentice or practitioner has an individual or common augmented simulator among all users, reducing costs compared to having equipment real. The method is similar to the one described in the augmented sports entertainment system, only that the augmented objects and scenarios change to educational, labor or medical material for carrying out practices, in this case, a change is available in step one, such that a user or group of users has the role of teacher and the others are only students, being the users with the role of teacher who decide when to start and end the method.

3. Augmented CAD/CAM design system or to share objects between video games: one application consists of giving current CAD/CAM design systems the ability to generate tactile and alterable augmented objects between users such that these changes can be stored and shared, facilitating the design without having to produce parts or prototypes and wait for manufacturing times or associated expenses. Another application is in video games where users can interact and share tangible avatars. The method is as follows: a. An augmented object is arranged, edited or created from some CAD/CAM or video game software; b. This augmented object is moved to the fixed base unit(s), c. the fixed base unit(s) estimates the position of the users via the telemetry units and calculates the position of the augmented objects; d. the sensing unit or units of the haptic device together with the processing units of each user determine if there is contact between the position and orientation of the haptic device and the augmented object; and. Users interact with augmented objects by viewing them in their visors and altering their shape, movement, or force with haptic interfaces; F. Said alteration of physical properties given by each user, through the on-board processing units and their respective telemetry units, is retransmitted to the world base or bases and with this it returns to point 3 of the method until the users or user decide to finish the modification; g. once the modification is complete, the base unit retransmits the modified augmented object to some computer where it is stored for later use.

4. Touch navigation system for mobile devices: a problem with current cell phones and/or smart tablets (tablets), is that they lose attention by focusing it on screens and keyboards, thus superimposing enlarged objects as they already do. immersive glasses, with this this problem is solved and also tactile capabilities are added, where the user obtains the ability to feel and share with other users, objects such as menus and virtual buttons augmented in reality, this can be used in social networks to share objects as icons or memes with tactile feedback. The method is as follows: a. the base unit (or units if a group or internet interaction is desired) that in this case is contained or can be adapted to the mobile device starts a menu with enlarged objects (said menu can be a chat or simple navigation windows); b. the sensing units of the haptic device together with the processing units of each user (which may be embedded in the users' mobile and viewer) determine if there is contact between the position and orientation of the haptic device and the augmented object; c. Users interact with the augmented objects by viewing them in their viewers and altering their shape, movement or force with the haptic interfaces, being able to share them or not as required by the application; d. said alteration of physical properties given by each user, through the on-board processing units and their respective telemetry units, is retransmitted to the world base or bases where the menu of increased objects is altered and the sequences are repeated, until the users or user end the program or leave the session.

5. Augmented vehicle teleoperation / teleexistence system: here one or more users can control a robot or vehicle remotely through a virtual avatar of said vehicle augmented in their individual real space, the real vehicle may or may not modify the virtual avatar for users to feel haptic feedback. The method is as follows: a. Real parameters of the vehicle to be controlled or of its components are measured; b. An enlarged object is available as a visual and tactile representation associated with the previous vehicle; c. Move said augmented object to the fixed base unit(s), d. the fixed base unit(s) estimate the position of the users via the telemetry units and calculate the position of the augmented objects; and. The sensing units of the haptic device together with the processing units of each human or non-human user determine if there is contact between the position and orientation of the haptic device and the augmented object; F. Users interact with augmented objects by viewing them in their visors and altering their shape, movement, or force with haptic interfaces and deciding how the actual vehicle or its components alter the augmented object; g. Said alteration of physical properties given by each user, through the on-board processing units and their respective telemetry units, is retransmitted to the world base or bases and the alteration of the augmented object or its components is replicated to the vehicle's actuators or their components. components; h. The augmented representation of the vehicle or its components is totally, null or partially altered by the real vehicle or its components; i. The method is repeated from step 1 until the users decide to abandon the cycle or the vehicle stops giving priority to the latter. Augmented system with force enhancement: in this case the wearable haptic device has force enhancement in the form of an exoskeleton and allows users to realistically feel bulky or heavy augmented virtual objects, this can be used for mechanical or rescue training where interacts with heavy and bulky items and parts. The method is similar to that described in the educational, labor or medical augmented tactile training system, but an exoskeleton is available in the haptic system, which allows interaction or the generation of the sensation of bulky and/or massive augmented objects. Augmented system of augmented brain-computer touch interaction: in this case, the user, in addition to having virtual touch on an augmented object, has the ability to control said object with signals from his brain through one of the many brain-computer interfaces available in the system. market, so it would be an experience that does not exist in the reality of touch and "telekinesis", this can be an aid to therapies or immersive playful experiences, for example, the user or users can have an enlarged arm that when manipulated with signals from your brain, allow direct feedback to your real arms via the haptic device placed on your real arms, such an experience can be seen as a kind of third person or out of body experience.The system can be used in rehab or other mixed use environments , this is an auxiliary of immersive playful experiences or therapies, providing a sensation in the user io or third-person users or out-of-body experience, there is also a combined augmented object control input between the device haptic interface and the brain-computer interface, this is what generates the third-person experience because the user, in addition to being able to control or not the augmented object with the wearable haptic interface (remember that this interface is controlled with body movement and there are users that lack it), you can also control said augmented object with your brain activity and whether or not said user or users move the haptic device, it will always return a reaction impulse to your body. The method is similar to that described in the educational, labor or medical augmented tactile training system, but there is also an augmented object control input combined between the haptic device and the brain-computer interface, this is what generates the experience in third person because the user, in addition to being able to control or not the augmented object with the wearable haptic interface (remember that this interface is controlled with body movement and there are users who lack it), can also control said augmented object with their brain activity and Whether or not said user or users move the haptic device, it will always return a reaction impulse to your body.

8. Guidance system for manual operations: an increased guide can be placed on a real object, to correctly direct the trajectory of a real drill through haptic feedback or the proper placement of a gutter or the positioning of a violin bow, being a three-dimensional plane said tactile augmented object. The method is similar to that described in the educational, labor or medical augmented tactile training system, but augmented object templates or tactile and visual augmented reality planes are available such that the user is forced by the haptic device to follow a pattern of movement and properly use some tool or real object or have access to visual touch planes for the intervention of a building or object on the built work or in black work, also changes step 5 because the increased object presents high resistance to being deformed forcing thereby allowing the user to adapt to its shape.

9. Augmented medical representation and reconstruction system: in this case there is a usual sensor, for example an ultrasound or X-ray that is capable of reconstructing an organ, based on its interpretation and a database of said interpretation, the The haptic system can generate the possible textures associated with said sensors and make it possible for a doctor to recreate the increased touch of an organ. This is useful for blind doctors in the analysis of an ultrasound or radiography, which are generally visual elements.

The method is similar to that described in the augmented educational, labor or medical training tactile system, only that the object to be monitored is non-deformable and is generated as a representation of a real object or organ through sensors placed on it, as obtained a visual touch representation, a blind person is able to interpret the magnified object without seeing it. 10. Industrial representation and reconstruction system: it is basically the same as the previous example, but with industrial applications such as flow monitoring in a pipeline, where an inspector can feel a representation of its content without breaking the tube and determine speed, viscosity , pressure, etc The method is similar to the one described in the previous case, it simply changes from medical to industrial area and therefore its standards and regulations.

11. Three-dimensional augmented tactile microscope: taking up the last two applications and increasing the dimensions (size or mass) of the virtual reconstruction of the monitored object (an artery and the blood that flows through it, for example), an enlarged and tactile interactive behavior of said object.

The method is a combination of those described in numbers 9 and 10 above, but the tactile and/or visual representation is amplified in dimensions or virtual mass in order to increase either sensation.

12. Two-way tactile augmented three-dimensional microscope: the same as above, but performing an action on the magnified magnified object affects the real object through robotic teleoperation, for example, a dentist can perform operations on a magnified magnified tooth that is replicate in a mini robot inside the patient's mouth.

The method is similar to the previous one and an additional step is added between steps 5 and 6 where the real object can be altered through the replication of the interaction with the augmented object by means of robotic mechanisms placed on the real object.

13. Sewing simulator or display objects: tailors or related workers can make virtual garments in an augmented world and test their designs or modifications on real models through combined touch (real touch plus haptic touch) or haptic visual touch advertising to promote objects such as footwear or furniture.

The method is similar to that of the industrial representation and reconstruction system, but a catalog of textures is available for the augmented object that can be superimposed on a real mold, a humanoid base such as a mannequin or on a user with the quality of being alive, that is to say, augmented objects, in addition to being three-dimensional volumes, can be three-dimensional surfaces.

14. Virtual orchestra and artistic applications: any user can play an augmented virtual instrument with real haptic feedback, being able to even form an orchestra of augmented instruments where only speakers are required for each instrument, the same can apply for other arts for example a sculptor or graphic designer can generate an augmented representation (copy) of a real object and work it in his augmented virtual world, only to print it later.

The method is similar to the one used for the tactile system for augmented educational, labor or medical training, but it also has a sound output linked to three-dimensional tactile interaction with augmented musical instruments. In the case of this and other artistic applications, there is the possibility of scanning a real item and using it as part or all of the augmented object, for example scanning a violin bridge and augmentedly mounting it on top of a virtual violin to alter its sound, or scanning a person and then sculpting them into a cartoon modification, etc. others. Teletouch: a virtual object associated with a real object or real person is generated and one or more users can feel that same virtual person or object but augmented in its location by means of the wearable haptic interface, this can have erotic, rehabilitation, personal communication between families or acquaintances, support for helpless people such as the elderly or children, rescue situations, etc.

The method is similar to the tactile augmented sports entertainment system, but the augmented object is a three-dimensional mesh or surface obtained from the geometry and shapes of each user, the haptic devices used here can also cover from partial to total body of each user and even be untranslatable in certain areas of the body to emulate a totally immersive experience. Humanoid robot with augmented visual touch teleexistence: suppose that being in another country we remember that we left the stove on, we activate a humanoid that is at home which we take to the kitchen via video, at that moment a virtual object associated with the stove and the hands of the humanoid that allow it to capture physical properties such as hardness, volume or inertia, in this way a version of the knob is reconstructed that we can feel and alter with the haptic interface present in our location to turn off the stove for example, if this is mixed with the application of tele touch it is possible to embrace or more actions with an augmented virtual representation of an object or person.

The proposed method is a joint and selectable action of the previous one and the augmented method of vehicle teleoperation/teleexistence. It is also multi-way, that is, one or more humanoids or artificial intelligences can be the users and control one or more humans or other beings through the action of the haptic device or exoskeleton, being useful for rescue or rehabilitation applications. Shared interaction of the same virtual object or its partialities, that is, suppose a virtual humanoid where 4 users independently control its two arms and two legs, to achieve a coordinated execution as in the concept of anime robots called "mecha" and that this virtual humanoid operates a real one. Augmented digital reading system: it is an augmented virtual book reader, that is, reading digital books with a similar touch sensation to read a physical book, users can be given the ability to feel a digital document and thereby increase the sale of books in those formats. The method is as follows: a. a digital document is available; b. it is superimposed the content of the digital document to an enlarged object that represents a book and its pages; c. The user decides whether to open it, close it, change the page or change the book through the haptic interface, as well as make annotations or place reading progress markers , each action can be saved to keep a history of events The sensors indicate if there is splicing or contact between the user and the augmented object d the user decides whether to share each book, or buy/sell it, and even if you have your own book collection, each action can be saved to keep track of events; and. multiple users can carry out the processes from "c" to "d" independently or there is a master user who determines the permissions and scope of each user; F. Steps from "a" to "e" are repeated until each user decides to leave their respective session or disconnects from their respective system. Electronic devices or augmented furniture: the user can do without having a television or a collection of multimedia files contained in physical readable media, being able to have them virtually but with the ability to interact tactilely with said objects (change the channel, select a legible medium of a virtual counter, having an augmented turntable, etc.), an augmented video or audio player, or an augmented avatar of your cell phone all with tactile feedback, this can also be used as a pre-purchase test method. device or actual physical object.

The method is the same as that of augmented digital reading, but it can be extended to furniture, electrical appliances and equipment or objects in general. In addition, it can be used as a set of augmented utensils or objects such as furniture, crafts, avatars, or visual, acoustic, and tactile virtual entertainment equipment augmented in reality, such as a turntable, television, or haptic interaction chat. touch to share memes or increased icons, etc. Augmented portable libraries: In a scenario, where it is not possible to build a building, several portable systems of augmented physical interaction can be carried for individual or collective applications through haptic and immersive feedback devices, with augmented virtual environments. preloaded with bookshelves and books, with the ability to share and exchange those books with other people in other geographical locations.

The method is similar to that of augmented digital reading, but they are arranged with shelves shared by as many users as the system supports or allows, and with the functionalities of a library: loan of books, return of the same, acquisition of new books, reward or fine systems, many copies of the same book, other services such as audio, newspapers, etc., and geographical restrictions or extensions, among others.

21. Augmented Password or Lock Systems: Augmented Virtual Items can be generated or augmented Virtual Items modified (by adding a note to an Augmented Book) and tell specific people where this information is to be used as a group password or else that it is capable of interacting with real locks through expansive haptic devices, this password generator or augmented physical locks, its use is restricted to one or more users who are capable of generating an augmented object that by itself or interacting with a real object serve as a password or key, ranging from writing a password in an augmented book that opens a virtual or augmented program or used to open a real button lock to an augmented key such that when printed it opens a real lock or either multiple objects designed by multiple users that when assembled are a key or lock or the s simple scan of a real key that can be shared by users or an augmented lock that allows access to a place or object whenever a real and an augmented lock are activated simultaneously, or finally an augmented key that is interactive with haptic locks ( that is, the haptic lock feels and acts according to the geometry of an enlarged key).

The method is similar to that used in the tactile augmented educational, labor or medical training system, but it is restricted to one or more users who are capable of generating an augmented object that by itself or by interacting with a real object serves as a password or key. , being able to be from the writing of a password in an augmented book that opens a virtual or augmented program or that is used to open a real button lock to an augmented key such that when printed it opens a real lock or multiple objects designed by multiple users that when assembled are a key or lock or the simple scanning of a real key that can be shared by users or an augmented lock that allows access to a place or object whenever a real and an augmented lock are activated simultaneously, or finally an augmented key that is interactive with haptic locks (i.e. the haptic lock adapts and acts according to the geometry of a increased key).

22. Haptic touch augmented mouse and keyboard or notebook and pencil system: these are applications derived from the fusion of several methods already described: the augmented book, the school or industrial training, the augmented passwords and locks, and the use on mobile devices. DESCRIPTION OF THE LABELS OF THE FIGURES

Claims

1. A portable system for augmented physical interaction with haptic and immersive feedback devices for individual or collective applications adapted for human and non-human users, chosen without limitation among humans, animals, androids, robots, avatars or artificial intelligences, characterized in that it comprises the following elements: a. At least one world base (1), this is a marker that is placed at some point of work common to all users, this always indicates the position that each user has, this base is unique if the users occupy the same work space. work or geographical location or multiple which allows tele-existence if the users are in different geographical locations, it is in charge of updating for each user the values related to the virtual object or serving as a global frame of reference; b. At least one portable sensor (2) external to the haptic interface that is placed in any other part of the user's body. It is adapted to measure distances and relative positions of the haptic interface with respect to the frame of the immersive and global viewer in order to detect the overlapping of the wearable haptic interface with the virtual object. The sensors are chosen without limitation from cameras, ultrasonic or radiofrequency arrays or non-motorized kinematic chains that incorporate joint sensors such as encoders, potentiometers, inertial measurement units (IMUS), optical, electrical or pressure flow interruption sensors; c. At least one immersive viewer (3), these can be immersive glasses or augmented reality glasses, in the first ones a video of the real world is taken and is projected to the user together with an overlay of images of the virtual object. In the second case, a virtual environment is superimposed on the real world through lenses; d. At least one wearable haptic interface (4), whose shape is chosen from a glove, footwear, limbs, torso, hat or full body or a force-enhancing device such as an exoskeleton, also adapted to feel an augmented object and alter it, that is, a virtual object superimposed on reality; and. At least one sensor on board the haptic interface (4a), these are adapted to detect velocities, accelerations and forces, both rotational and translational, of the part of the body covered by the wearable haptic interface and whose measurement allows altering the behavior of the object augmented virtual; F. At least one virtual object augmented on the real world (5), these virtual objects are placed superimposed on the real world (5), having the world base (1) as a reference; g. At least one portable processing unit (6), adapted to carry out calculations, measurements, haptic control, sending and receiving telemetric data and the superimposition of the virtual object in the real world, each user carries one of them; h. At least one telemetry equipment (7), this is adapted to send the data of each user, as well as of the world base (1) and of the virtual objects among themselves, they are bidirectional data communicators that can use wireless communication technologies such as Bluetooth or WiFi or any other communication technology available; i. A set of position and orientation spatial coordinate frames, these are composed of three translation basis coordinates and three rotation basis coordinates) and are at least:

A. The immersive viewer (Xv, Yv, Zv), this is the user-level reference;

B. The world base (Xb, Yb, Zb), this is established by the world base marker and is placed at world zero or the position that said base has, this can also be established on board a user, which serves as a reference base for other users, it can also have a dynamic radar-like tracking and transmission system and thereby reduce sensors and emitters or have as many sensors and emitters as there are users;

C. That of each user (Xu, Yu, Zu), this is established by each user and is measured with respect to the world base (1);

D. The one of the increased object (Xo, Yo, Zo), this is measured with respect to the base world and consequently each user);

E. The haptic device (Xh, Yh, Zh), this is measured at the user level, with respect to the viewer and with respect to the external monitoring device, it is also considered as an end effector;

F. The external sensor and monitoring of the haptic device (Xse, Yse, Zse), this is measured at the user level and is the link between the viewer and the haptic device, between it and the haptic device, the user forms a kinematic chain with its own body, it can be placed next to the viewfinder.

2. A method adapted to use a portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices, characterized in that the method allows the editing, visualization and/or design of virtual and augmented objects to share said objects between a team of work users or a single user, using CAD/CAM software and storage devices, which comprises the following stages: a. An enlarged object is arranged, edited or created from some CAD/CAM or video game software, b. said increased object is transferred to the fixed base unit or units, c. the fixed base unit or units estimate the position of the users via the telemetry units and calculate the position of the augmented objects, d. the sensing unit or units of the haptic device together with the processing units of each user determine if there is contact between the position and orientation of the haptic device and the augmented object, and. Users interact with augmented objects by viewing them in their viewers and altering their shape, motion, or force with haptic interfaces, f. Said alteration of physical properties given by each user, through the on-board processing units and their respective telemetry units, is retransmitted to the world base or bases and with this it returns to point f of the method until the users or user decide to finish the modification, g. once the modification is complete, the base unit retransmits the modified augmented object to some computer where it is stored for later use.

3. A method adapted to use a portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices, characterized in that it is adapted to be used with mobile devices such as phones or tablets where the user has the ability to feel and share with other users virtual objects such as menus and buttons augmented in reality, the method is as follows: a. the base unit (or units if a group or internet interaction is desired) that in this case is contained or can be adapted to the mobile device starts a menu with enlarged objects (said menu can be a chat or simple navigation windows); b. the sensing units of the haptic device together with the processing units of each user (which may be embedded in the users' mobile and viewer) determine if there is contact between the position and orientation of the haptic device and the augmented object; c. Users interact with the augmented objects by viewing them in their viewers and altering their shape, movement or force with the haptic interfaces, being able to share them or not as required by the application; d. said alteration of physical properties given by each user, through the on-board processing units and their respective telemetry units, is retransmitted to the world base or bases where the menu of increased objects is altered and the sequences are repeated, until the users or user end the program or leave session.

4. A method adapted to use a portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices, characterized in that the method allows a team of users or a single user to use and change from one augmented environment to another for example, from a basketball environment to a tennis environment, from a soccer environment to an American football environment, from a chess environment to an augmented reality orchestra or any other existing playful environment, all with the intention of combining a virtual gaming environment with the real world, which includes the following stages: a. At least one user chooses a pre-programmed sport of the augmented type, b. At least one fixed base unit loads said program and augmented objects of interaction, c. At least one fixed base unit estimates the position of the users through the telemetry units and calculates the position of the augmented sports objects, d. At least one sensing unit of the haptic device together with the processing units of each user determine if there is contact between the position and orientation of the haptic device and the augmented object, e. At least one user interacts with the augmented sports objects by viewing them in their visors and altering their movement and force with the haptic interfaces, f. Said alteration of force given by each user, through the on-board processing units and their respective telemetry units, is retransmitted to the world base or bases and with this they return to point c of the method until the users or user decide to end the process. game, save and quit the game or change physical activity increased.

5. A portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices in accordance with claim 1, characterized in that it can be used by a team of users in the same geographical location or in different geographical locations between said users.

6. A portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices in accordance with claim 1, characterized in that it is adapted with the ability to choose whether it is used only by a single real user or by a group of users. real.

7. A portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices in accordance with claim 1, characterized in that the portable sensors external to the haptic interface that can be placed in any other part of the user's body have the option to be placed on a surface or volume external to the user's body.

8. A portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices in accordance with claim 1, characterized in that the sensors and transmitters are generally chosen from radio frequencies, visual, acoustic, thermal, or any other available technology.

9. A method adapted to use a portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices in accordance with claim 4, characterized in that it is adapted to be used in augmented educational, labor or medical training applications where each student or apprentice or practitioner has an individual or common augmented simulator among all users, reducing costs compared to having real equipment, the augmented objects and scenarios, change to educational, labor or medical material for the realization of practices, in this case, there is a small change in step one, such that a user or group of users has the role of teacher and the others only be students, being the users with the role of teacher who decide when to start and end the method.

10. A method adapted to use a portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices in accordance with claims 2, 3, 4 and 9, characterized in that the augmented haptic interface has an augmentation mechanism force like an exoskeleton and this allows the user to interact with bulky or massive augmented and virtual objects.

11. A portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices in accordance with claim 1, characterized in that the telemetry equipment on board the world bases can be dynamic (one or more sensors and transmitters placed like a radar) or static (so many sensors or transmitters placed arbitrarily and fixed at the convenience of users).

12. A portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices in accordance with claim 1, characterized in that the world bases are mobile devices such as cell phones, tablets or laptops or fixed devices such as desktop computers and in whether or not they are incorporated: at least one telemetry equipment, at least one portable processing unit and at least one portable sensor external to the haptic interface.

13. A portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices in accordance with claim 1, characterized in that the portable sensors external to the haptic interface that can be placed in any other part of the body are embedded in the viewer immersive augmented reality.

14. A portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices in accordance with claim 1, characterized in that it is adapted with the variant of being placed in vehicles that are chosen from land, sea, air or a combination of the above.

15. A method adapted to use a portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices, characterized in that it can be selected whether or not the position and orientation of a vehicle, robot, humanoid robot, is altered or not. animal, human being, artificial intelligence or real object using an augmented or virtual one and vice versa, through the replicated action of one or multiple users that may or may not be human, comprising the following steps: a. actual parameters of the vehicle or human or non-human entity to be controlled or of its components are measured, b. an augmented object is available as a visual and tactile representation associated with the previous vehicle or human or non-human entity to be controlled, c. said augmented object is moved to the fixed base unit(s), d. the fixed base unit(s) estimate the position of the users via the telemetry units and calculate the position of the augmented objects, e. the sensing units of the haptic device together with the processing units of each human or non-human user, determine if there is contact between the position and orientation of the haptic device and the augmented object, f. human or non-human users interact with augmented objects by viewing them in their visors and altering their shape, motion, or force with haptic interfaces and deciding how the actual vehicle or its components alter the augmented object, g. Said alteration of physical properties given by each user, through the on-board processing units and their respective telemetry units, is retransmitted to the world base or bases and the alteration of the augmented object or its components is replicated to the actuators of the vehicle or its components. components, or towards humans or living beings that carry a haptic device, h. the augmented representation of the vehicle, human or non-human user or its components is totally, null or partially altered by the actual vehicle, human or non-human user or its components, i. The method is repeated from step 1 until the users decide to abandon the cycle or the vehicle or human or non-human entity to drive stops, giving priority to the latter.

16. A method adapted to use a portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices in accordance with claims 2, 3, 4, 9, 10 and 15, characterized in that it is adapted to be used in conjunction with other systems such as brain-computer interfaces in rehabilitation environments or other mixed uses, this is an aid to therapies or immersive playful experiences, providing a sensation in the user or third-person users or an out-of-body experience, there is also a control input of the augmented object combined between the haptic device and the brain-computer interface, this is what generates the third-person experience because the user, in addition to being able to control or not the augmented object with the wearable haptic interface, this interface is controlled with body movement and there are users who lack it, you can also control d that object increased with his brain activity and may or may not said user or users move the haptic device, it will always return a reaction impulse to your body.

17. A method adapted to use a portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices in accordance with claim 2, and is adapted to be used as a password generator or augmented physical locks, the use is restricted to one or more users who are capable of generating an augmented object that by itself or by interacting with a real object serves as a password or key, which may be from writing a password in an augmented book that opens a virtual or augmented program or that is used to open a real button lock to an augmented key such that when printed it opens a real deadbolt or multiple objects designed by multiple users that when assembled are a key or lock or a simple scan of a real key that can be shared by users or an increased lock that allows access to a place or object whenever it is e simultaneously activate a real lock and an augmented one, or finally an augmented key that is interactive with haptic locks, the haptic lock feels and acts according to the geometry of an augmented key.

18. A method adapted to use a portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices in accordance with claim 15, characterized in that it is adapted to work with shared interaction of the same augmented object or its partialities. That is, each user controls individual parts or components of a robot, vehicle, human or non-human being through the augmented object.

19. A method adapted to use a portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices, characterized in that it is adapted to be used as an augmented virtual book reader, that is, to read digital books with a sensation of tact similar to reading a physical book, the method is: a. some digital document is available; b. the content of the digital document is superimposed on an enlarged object that represents a book and its pages; c. The user decides whether to open it, close it, change the page or change the book through the haptic interface, as well as make annotations or place reading progress markers, each action can be saved to keep a history of events, the sensors indicate if there is splicing o contact between the user and the augmented object; d. the user decides whether to share each book, or buy/sell it, and even if he has his own collection of books, each action can be saved to keep track of events; and. multiple users can perform processes "c" to "e" independently or there is a master user who determines the permissions and scope of each user; F. Steps "a" to "f" are repeated until each user decides to leave their respective session or disconnects from their respective system.

20. A method adapted to use a portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices in accordance with claims 9 and 19, characterized in that it is adapted to be used as a portable library, with the ability to share and exchange said books and other virtual or augmented materials with other people in other geographical locations, there are shelves shared by as many users as the system will support or allow and with the functionalities of a library: loan of books, return of same, acquisition of new books, reward or fine systems, many copies of the same book, other services such as audio, newspapers and geographic restrictions or extensions among others.

21. A method adapted to use a portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices in accordance with claims 9, 10, 15 and 19, characterized in that it is adapted to be used as a set of utensils or augmented objects such as furniture, crafts, avatars, or virtual visual, acoustic, and tactile entertainment equipment augmented in reality, such as a turntable, television, or haptic-touch interaction chat for sharing memes or augmented icons, etc.

22. A method adapted to use a portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices in accordance with claim 4, characterized in that it is adapted to be used with erotic, social interaction and rehabilitation applications or well as support in rescue and attention to children or the helpless using unreliable haptics or full-body haptic outfits, the augmented object is a three-dimensional mesh or surface obtained from the geometry and shapes of each user, the haptic devices used here can also cover from the partial to the entire body of each user and even being untranslatable in certain areas of the body to emulate a totally immersive experience, the augmented objects, in addition to volumes, are tactile meshes that replicate the objects or users to be emulated.

23. A method adapted to use a portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices in accordance with claim 9, characterized in that it is adapted to be used as a guidance system for manual operations in training of use of tools or musical instruments or construction training and guidance, templates of augmented objects or tactile and visual augmented reality planes are available such that the user is forced by the haptic device to follow a movement pattern and properly use some tool or real object or have access to visual touch planes for the intervention of a building or object on the built work or in black work, the increased object has high resistance to being deformed thereby forcing the user to adapt to its shape, the increased object is non-deformable or very high resistance to being deformed to force the user to adapt to their geometries.

24. A method adapted to use a portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices in accordance with claim 9, characterized in that it is adapted to be used as a form of medical or industrial representation and reconstruction, for provision of visual touch representations, through the use of sensors and databases for the representation of organs or measurement of industrial processes such as flow in pipes, the object to be monitored is non-deformable and is generated as a representation of a real object or organ through sensors placed in it, how a visual touch representation is obtained, a blind person is able to interpret the augmented object without having to see it, the augmented object is non-deformable or very resistant to being deformed by the user to force the user to adapt to their geometries but it is generated in time real from a specialized sensor such as an ultrasound or X-ray machine, it is altered by the sensors that generate said augmented object, but not by the user.

25. A method adapted to use a portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices according to claim 24, characterized in that it is adapted to be used as an augmented reality haptic microscope, that is, increasing the tactile and visual dimensions of the monitored object.

26. A method adapted to use a portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices in accordance with claims 24 and 25, characterized in that it can be used in two ways, that is to say that in addition to monitoring the real object, allow it to be modified through the interaction of the virtual object and a replicating mechanism of the haptic action placed on the real object, the user can alter the augmented object and the augmented object simultaneously alters the real object it represents.

27. A method adapted to use a portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices in accordance with claim 21, characterized in that it can be used as a servo-tactile advertising method, for use in various objects such as furniture or footwear, a catalog of textures is available for the augmented object that can be superimposed on a real mold, a humanoid base such as a mannequin or on a living being, that is, the augmented objects as well as being volumes three-dimensional, they can be three-dimensional surfaces, in addition there are catalogs of increased objects.

28. A method adapted to use a portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices in accordance with claim 21, characterized in that it can be used in fashion applications as a way of performing increased touch or test a type of fabric or garment superimposed in an augmented way on a real person or on a mannequin, in addition there is a haptic catalog of textile textures.

29. A method adapted to use a portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices in accordance with claims 4 and 9, characterized in that it can be used in artistic applications such as in an augmented orchestra where musicians can play with real haptic feedback an augmented virtual instrument, being able to form an orchestra of augmented instruments where only speakers are required for each instrument, the same can apply to other arts for example a sculptor or graphic designer can generate an augmented representation (copy) of a real object from its three-dimensional reconstruction and work it in its augmented virtual world, to later only print it, the acoustic interaction through speakers with the augmented objects or the augmented objects are an in-situ reconstruction of real objects with a camera or any other technological means available.

30. A method adapted to use a portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices in accordance with claims 19 and 21, characterized in that it can be used as an augmented notebook where users have a notebook with haptic feedback pen both.

31. A method adapted to use a portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices according to claim 21, characterized in that it can be used as augmented mice and keyboards with haptic feedback.

32. A portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices according to claim 1, characterized in that the action of the haptic interface is adjustable between a total, partial, intermittent or null mode.

33. A portable augmented physical interaction system for individual or collective applications through haptic and immersive feedback devices in accordance with claim 1, characterized in that at least one world base (1) can be fixed and external to the user or users or be portable between one or more users who conveniently choose which of them has the world base (1).