US20220374070A1 - System and Method for Reconstructing a VR Avatar With Full Body Pose - Google Patents
System and Method for Reconstructing a VR Avatar With Full Body Pose Download PDFInfo
- Publication number
- US20220374070A1 US20220374070A1 US17/881,590 US202217881590A US2022374070A1 US 20220374070 A1 US20220374070 A1 US 20220374070A1 US 202217881590 A US202217881590 A US 202217881590A US 2022374070 A1 US2022374070 A1 US 2022374070A1
- Authority
- US
- United States
- Prior art keywords
- participant
- tracker
- data
- foot
- body pose
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims description 24
- 210000002683 foot Anatomy 0.000 claims abstract description 111
- 210000003423 ankle Anatomy 0.000 claims abstract description 42
- 238000004891 communication Methods 0.000 claims abstract description 30
- 208000013057 hereditary mucoepithelial dysplasia Diseases 0.000 claims abstract description 21
- 238000004519 manufacturing process Methods 0.000 claims abstract description 8
- 230000033001 locomotion Effects 0.000 claims description 42
- 210000003127 knee Anatomy 0.000 claims description 10
- 230000001133 acceleration Effects 0.000 claims description 6
- 210000004197 pelvis Anatomy 0.000 claims description 6
- 230000008569 process Effects 0.000 claims description 4
- 238000010801 machine learning Methods 0.000 description 16
- 230000003287 optical effect Effects 0.000 description 13
- 238000013507 mapping Methods 0.000 description 9
- 238000012549 training Methods 0.000 description 9
- 238000006243 chemical reaction Methods 0.000 description 4
- 238000009826 distribution Methods 0.000 description 4
- 210000001624 hip Anatomy 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000003384 imaging method Methods 0.000 description 3
- 238000002372 labelling Methods 0.000 description 3
- 239000003550 marker Substances 0.000 description 3
- 230000001537 neural effect Effects 0.000 description 3
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 2
- 244000309466 calf Species 0.000 description 2
- 230000004069 differentiation Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 210000002414 leg Anatomy 0.000 description 2
- 210000000689 upper leg Anatomy 0.000 description 2
- PEDCQBHIVMGVHV-UHFFFAOYSA-N Glycerine Chemical group OCC(O)CO PEDCQBHIVMGVHV-UHFFFAOYSA-N 0.000 description 1
- 206010021403 Illusion Diseases 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 210000000459 calcaneus Anatomy 0.000 description 1
- QBWCMBCROVPCKQ-UHFFFAOYSA-N chlorous acid Chemical compound OCl=O QBWCMBCROVPCKQ-UHFFFAOYSA-N 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 210000003414 extremity Anatomy 0.000 description 1
- 238000007667 floating Methods 0.000 description 1
- 230000009191 jumping Effects 0.000 description 1
- 210000001872 metatarsal bone Anatomy 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000003860 storage Methods 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
- G06F3/1454—Digital output to display device ; Cooperation and interconnection of the display device with other functional units involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
- G06F3/0346—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F3/04817—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance using icons
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F3/04845—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range for image manipulation, e.g. dragging, rotation, expansion or change of colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
- G06F3/147—Digital output to display device ; Cooperation and interconnection of the display device with other functional units using display panels
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
- G06F2203/048—Indexing scheme relating to G06F3/048
- G06F2203/04802—3D-info-object: information is displayed on the internal or external surface of a three dimensional manipulable object, e.g. on the faces of a cube that can be rotated by the user
Definitions
- each user In a VR application shared by two or more users, it is desirable for each user to see all of the other users within the shared experience, represented as visible avatars. Ideally, the full body pose of the avatar of each user would be available, so that each user can see all of the other users as full body avatars. Yet because systems such as the Quest track only the head and hands, as shown in FIG. 1 , only the upper body portion of a user's avatar can be reconstructed from data gathered by the Quest-like systems, as shown in FIG. 2 . Therefore, multiple users sharing the same VR experience, whether in the same room or remotely located, are able to see only partial representations of each other, visible as avatars possessing only the upper half of their body.
- the user straps a Lighthouse tracker to each ankle.
- the position and orientation of each of these trackers can be determined by time-varying structured light patterns impinging upon that tracker from a set of Lighthouse infrared beacons in the room.
- additional tracking apparatus is worn on the user's feet or ankles, the additional information thereby provided can be used to reconstruct the appearance of an avatar of the user's entire body. It is advantageous to do this full body tracking using only inside-out tracking, in order to preserve the desirable property that no external markers or optical apparatus is needed in the surrounding space. Yet affordable inside-out methods for tracking the feet or ankles do not report absolute positions for the user's feet.
- the present invention describes how to incorporate inexpensive inside-out methods of foot tracking to supplement inside-out tracked VR head and hands tracking, and then consequently to perform full body pose reconstruction, without the need for external markers or other optical apparatus.
- the present invention pertains to a system for producing a shared virtual reality.
- the system comprises a communication network.
- the system comprises a plurality of HMDs that are worn by participants to view the shared virtual reality in communication with the network.
- the system comprises a computer server in communication with the network which produces a full body pose of at least one of participant of the participants and transmits the full body pose via the network to all other participants HMD, so that each participant can see the full body pose in each participant's view into the shared virtual reality.
- the server may produce the full body pose over time.
- the present invention pertains to a method for reconstructing a VR avatar with full body pose from a participant.
- the method comprises the steps of sending timestamped data from an HMD on a head of a participant and a right-hand controller held by the participant's right hand and a left-hand controller held by the participant's left hand and feet trackers or ankle trackers on the participant's feet or ankles to a database on a server computer wirelessly.
- FIG. 8 is a pressure image of a foot pressure tracking insole.
- the system 10 for reconstructing a VR avatar with full body pose 12 from a participant.
- the system 10 comprises an inside-out tracked HMD 14 that is to be worn by the participant.
- the system 10 comprises a left-hand controller 16 to be held by a left hand of the participant that produces position data of the left hand of the participant.
- the system 10 comprises a right-hand controller 18 to be held by a right hand of the participant that produces position data of the right hand of the participant.
- the system 10 comprises a right tracker 20 worn on a right foot or right ankle of the participant.
- the system 10 comprises a left tracker 22 worn on a left foot or left ankle of the participant.
- the server computer 30 may produce a total number of 48 output scalar values, where six scalar values for the position and orientation of each of the head, hands, feet, upper torso and pelvis, +2 scalar values for each of a right shoulder and left shoulder, +1 scalar value for each of the right elbow, left elbow, right knee and left knee of the participant.
- the computer produces a body pose 12 output of the participant from the 48 scalar values.
- the body pose 12 output of the participant may include position and orientation of the right-hand and left-hand, right foot and left foot, upper torso, pelvis, head, forward and upward rotation of each shoulder about the upper torso and inward to outward turning of elbows and knees of the participant.
- the present invention pertains to a system 10 for producing a shared virtual reality.
- the system 10 comprises a communication network.
- the system 10 comprises a plurality of HMDs that are worn by participants to view the shared virtual reality in communication with the network.
- the system 10 comprises a computer server in communication with the network which produces a full body pose 12 of at least one of participant of the participants and transmits the full body pose 12 via the network to all other participants HMD 14 , so that each participant can see the full body pose 12 in each participant's view into the shared virtual reality.
- the server may produce the full body pose 12 over time.
- the present invention pertains to a method for producing a shared virtual reality.
- the method comprises the steps of producing with a computer server a full body pose 12 of at least one of participant of a plurality of participants. There is the step of transmitting the full body pose 12 via a communications network to an HMD 14 worn by each of all other participants, so that each participant can see the full body pose 12 in each participant's view into the shared virtual reality.
- ML is employed in order to learn a mapping from an HMD 14 and two controllers, as well as an IMU 32 and possibly a pressure imaging device and/or proximity sensors 40 placed on the feet or ankles.
- Semi-supervised learning is typically where there is a small amount of labelled data (supervised), and a large amount of unlabeled data (unsupervised).
- the labelled data is the combined operator-captured input from OptiTrack, Rokoko, shoes, HMD 14 +controllers.
- OptiTrack, Rokoko, shoes, HMD 14 +controllers There is also a rich source of data available online, however, labels for that on-line data are not available.
- the learning algorithm used here is a combination of these labelled data (supervised) and unlabeled data (unsupervised), it is a semi-supervised learning algorithm.
- the 3D pose for all users is transmitted wirelessly from the server computer 30 to the HMDs of all users, where the pose data is then used to construct, for each user, 3D computer animated representations of the avatars of all users.
- the other foot has pressure mainly on the rear of the foot, this would indicate the participant is walking and position the waist and knees and thighs and calves of each leg in the appropriate position for walking.
- this indicates also the participant is walking and adjust the waist and knees and thighs and calves are in mid-stride and position them accordingly to show the full body pose in mid-stride with the leg having no pressure on its insole being slightly bent and lifted off the ground being positioned to about to receive the weight of the participant as the participant's weight shifts while walking forward.
Abstract
A system for producing a shared virtual reality having a communication network. The system has a plurality of inside-out tracked HMDs that are worn by participants to view the shared virtual reality in communication with the network, right and left hand controllers that are held by the participant and right and left trackers that are worn on the ankles or feet of the participant. The system has a server computer in communication with the network which produces a full body pose of at least one of participant of the participants and transmits the full body pose via the network to all other participants HMD, so that each participant can see the full body pose in each participant's view into the shared virtual reality. A method for producing a shared virtual reality of participants.
Description
- This is a divisional of U.S. patent application Ser. No. 17/016,884 filed Sep. 10, 2020, now U.S. Pat. No. 11,409,358 which is a non-provisional of U.S. provisional patent application Ser. No. 62/899,493 filed Sep. 12, 2019, incorporated by reference herein.
- The present invention is related to reconstructing VR avatars with full body pose from inside-out tracked HMD. (As used herein, references to the “present invention” or “invention” relate to exemplary embodiments and not necessarily to every embodiment encompassed by the appended claims.) More specifically, the present invention is related to reconstructing VR avatars with full body pose from inside-out tracked HMD using hand controllers and foot or ankle trackers.
- This section is intended to introduce the reader to various aspects of the art that may be related to various aspects of the present invention. The following discussion is intended to provide information to facilitate a better understanding of the present invention. Accordingly, it should be understood that statements in the following discussion are to be read in this light, and not as admissions of prior art.
- Using a Virtual Reality head mounted display (HMD) which is capable of inside-out tracking, such as the Oculus Quest, it is possible to accurately track six degrees of freedom (6DOF) for a user's head and also for each of two handheld controllers—specifically, the xyz position as well as the three axes of orientation for each of the head, the left hand and the right hand. Because the Quest uses inside-out tracking, it has the desirable quality that it requires no external markers or other optical apparatus in the surrounding space.
- In a VR application shared by two or more users, it is desirable for each user to see all of the other users within the shared experience, represented as visible avatars. Ideally, the full body pose of the avatar of each user would be available, so that each user can see all of the other users as full body avatars. Yet because systems such as the Quest track only the head and hands, as shown in
FIG. 1 , only the upper body portion of a user's avatar can be reconstructed from data gathered by the Quest-like systems, as shown inFIG. 2 . Therefore, multiple users sharing the same VR experience, whether in the same room or remotely located, are able to see only partial representations of each other, visible as avatars possessing only the upper half of their body. - Prior work in the field has implemented foot position and orientation tracking by incorporating external optical equipment fixed to the room, such as the Kinect depth camera [MICROSOFT].
- Other prior work uses a motion capture system such as OptiTrack or Vicon, which requires the user to wear retroreflective markers on the feet and/or ankles and employs many high-speed infrared cameras [FUTURE REALITY LAB].
- In other systems, [Kruse, Lucie & Langbehn, Eike & Steinicke, Frank. (2018). I Can See on My Feet While Walking: Sensitivity to Translation Gains with Visible Feet. 10.1109/VR.2018.8446216; incorporated by reference, herein] the user straps a Lighthouse tracker to each ankle. The position and orientation of each of these trackers can be determined by time-varying structured light patterns impinging upon that tracker from a set of Lighthouse infrared beacons in the room.
- All of the above systems are based on external optical line of sight between the user's feet and/or ankles and external optical components, and therefore suffer from the deficiency that multiple users can block each other's line of sight view of the external optical equipment. The greater the number of users, the more problematic that becomes.
- Other prior work [TACTONIC] has affected foot position and orientation tracking by covering the floor with pressure imaging tiles. Again, this requires external equipment to be installed in the room.
- Other prior work in the field [ANTILATENCY] has affected foot tracking by placing small cameras on each foot and placing structured lights into floor tiles. Again, this requires external equipment to be installed in the room. Also, the foot mounted cameras are quite expensive.
- It would be possible for each user to wear a full motion capture suit based on IMUs, such as the Rokoko suit, which measures the joint angles of all of the major joints in the user's body. If combined with an inside-out tracking HMD such as the Oculus Quest, such a system would be able to infer the full body pose and the absolute position of each foot. However, this approach would require each user to wear a full body motion capture suit, which would be both prohibitively expensive and prohibitively cumbersome and intrusive for most applications.
- If additional tracking apparatus is worn on the user's feet or ankles, the additional information thereby provided can be used to reconstruct the appearance of an avatar of the user's entire body. It is advantageous to do this full body tracking using only inside-out tracking, in order to preserve the desirable property that no external markers or optical apparatus is needed in the surrounding space. Yet affordable inside-out methods for tracking the feet or ankles do not report absolute positions for the user's feet.
- The present invention describes how to incorporate inexpensive inside-out methods of foot tracking to supplement inside-out tracked VR head and hands tracking, and then consequently to perform full body pose reconstruction, without the need for external markers or other optical apparatus.
- The present invention pertains to a system for reconstructing a VR avatar with full body pose from a participant. The system comprises an inside-out tracked HMD that is to be worn by the participant. The system comprises a left-hand input device to either track or be held by a left hand of the participant that produces position data of the left hand of the participant. The system comprises a right-hand input device to either track or be held by the right hand of the participant that produces position data of the right hand of the participant. Hereafter, it is assumed input from the user's hands, be it camera based hand tracking (RGB or marker based IR), inside out tracked gloves via IMUs, neural interfaces, in the spirit of motion touch controllers/6dof remotes, or other method of gaining input from a user's hands. Referred to hereafter as left/right hand controller. The system comprises a right tracker worn on a right foot or right ankle of the participant. The system comprises a left tracker worn on a left foot or left ankle of the participant. The system comprises a motion capture suit to be worn by the participant. The system comprises a motion capture unit which captures motion of the participant from the motion capture suit worn by the participant. The system comprises a communication network in communication with the left hand and the right hand and the right tracker and the left tracker and the motion capture suit and the motion capture unit. The system comprises a server computer in communication with the network that constructs a full body pose of the participant using real time data from only the right hand controller and left hand controller and right tracker and left tracker and HMD and from previously stored data from the motion capture unit.
- The present invention pertains to a system for producing a shared virtual reality. The system comprises a communication network. The system comprises a plurality of HMDs that are worn by participants to view the shared virtual reality in communication with the network. The system comprises a computer server in communication with the network which produces a full body pose of at least one of participant of the participants and transmits the full body pose via the network to all other participants HMD, so that each participant can see the full body pose in each participant's view into the shared virtual reality. The server may produce the full body pose over time.
- The present invention pertains to a method for producing a shared virtual reality. The method comprises the steps of producing with a computer server a full body pose of at least one of participant of a plurality of participants. There is the step of transmitting the full body pose via a communications network to an HMD worn by each of all other participants, so that each participant can see the full body pose in each participant's view into the shared virtual reality.
- The present invention pertains to a method for producing a shared virtual reality of participants. The method comprises the steps of receiving a him and in t a server computer data from an inside-out tracked HMD worn on a head of a participant of the participants and two hand controllers held by the participant and trackers worn on feet or ankles of the participant. There is the step of constructing a full body pose of a participant using only the data from the HMD and the two hand controllers and the trackers. There is the step of displaying the full body pose of the participant in the shared virtual reality which is viewed by the HMD.
- Given two or more participants, each wearing an HMD, the present invention pertains to a full body pose of any given participant being transmitted via wireless transmission to all other participants, so that each participant can see the full body pose of all other participants in their view into a shared virtual reality world of the HMD.
- The present invention pertains to a method for reconstructing a VR avatar with full body pose from a participant. The method comprises the steps of sending timestamped data from an HMD on a head of a participant and a right-hand controller held by the participant's right hand and a left-hand controller held by the participant's left hand and feet trackers or ankle trackers on the participant's feet or ankles to a database on a server computer wirelessly. There is the step of applying mocap data in the database as labeling data by the server computer to map from input of the HMD, the right and left hand controllers and feet trackers or ankle trackers data to output of full body pose of the participant.
- The present invention pertains to a method for producing a shared virtual reality of participants. The method comprises the steps of sending data from an HMD, hand controllers and feet or ankle trackers worn or held by each participant to a server computer. There is the step of reconstructing a full body pose for each participant from each participant's data with the server computer. There is the step of sending the full body pose of all participants wirelessly to all HMDs, so that each participant sees the full body pose of all participants in the shared VR displayed in each participant's HMD.
- In the accompanying drawings, the preferred embodiment of the invention and preferred methods of practicing the invention are illustrated in which:
-
FIG. 1 shows a participant having an HMD and hand controllers. -
FIG. 2 shows upper body avatars in a virtual reality. -
FIG. 3 is a schematic representation of the system of the claimed invention. -
FIG. 4 shows a participant in the training phase. -
FIG. 5 is a representation showing sensors of a participant used to create a full body pose. -
FIG. 6 shows multiple full body avatars. -
FIG. 7 is a schematic representation of the sensors being used to produce the full body pose. -
FIG. 8 is a pressure image of a foot pressure tracking insole. -
FIG. 9 shows pressure zones associated with the pressure image ofFIG. 8 . -
FIG. 10 is a schematic representation of a foot tracker. - Referring now to the drawings wherein like reference numerals refer to similar or identical parts throughout the several views, and more specifically to
FIG. 3 thereof, there is shown asystem 10 for reconstructing a VR avatar with full body pose 12 from a participant. Thesystem 10 comprises an inside-out trackedHMD 14 that is to be worn by the participant. Thesystem 10 comprises a left-hand controller 16 to be held by a left hand of the participant that produces position data of the left hand of the participant. Thesystem 10 comprises a right-hand controller 18 to be held by a right hand of the participant that produces position data of the right hand of the participant. Thesystem 10 comprises aright tracker 20 worn on a right foot or right ankle of the participant. Thesystem 10 comprises aleft tracker 22 worn on a left foot or left ankle of the participant. Thesystem 10 comprises amotion capture suit 24 to be worn by the participant. Thesystem 10 comprises amotion capture unit 26 which captures motion of the participant from themotion capture suit 24 worn by the participant. Thesystem 10 comprises a communication network 28 in communication with the left-hand controller 16 and the right-hand controller 18 and theright tracker 20 and theleft tracker 22 and themotion capture suit 24 and themotion capture unit 26. Thesystem 10 comprises aserver computer 30 in communication with the network 28 that constructs a full body pose 12 of the participant using real time data from only theright hand controller 18 andleft hand controller 16 andright tracker 20 and lefttracker 22 andHMD 14 and from previously stored data from themotion capture unit 26. When thesystem 10 is used just for the end user, themotion capture suit 24 andunit 26 are not typically used. The server computer then already has access to data storage with details about each participants' full body which was previously collected by theunit 26. Themotion capture suit 24 andunit 26 are only needed to create the data about a participant's appearance for storage and later use. - The
right tracker 20 may have a 6° offreedom IMU 32 to produce tracked data from the right foot and awireless transmitter 34 which transmits the tracked data to theserver computer 30 through the communications network 28. The 6° of freedom are 3° of orientation and 3° of acceleration in an X, Y, Z coordinate system, and theleft tracker 22 has anIMU 32. Theright tracker 20 may have amicroprocessor 36 in communication with thewireless transmitter 34 and theIMU 32. Themicroprocessor 36 processes the tracked data and sends the tracked data that has been processed to theserver computer 30 through thewireless transmitter 34. - The
right tracker 20 may have a footpressure tracking insole 38 that is worn underneath the right foot. The footpressure tracking insole 38 produces pressure data which is provided to themicroprocessor 36, and theleft tracker 22 has a footpressure tracking insole 38 that is worn underneath the left foot. Each foot pressure tracking system produces six Center of Mass Values for convolutions; for example, each of the Phalange, Metatarsal (right and left sides of each foot), cuboid (splint into front and back), and the calcaneus. Theright tracker 20 may include aproximity sensor 40 which measures time-varying distance between the right foot and a floor upon which the participant is walking. - The
server computer 30 may receive input that includes position and orientation of each of theHMD 14, the left-hand controller and theright hand controller 18, gyroscopic orientation and accelerometer data from theIMU 32 of theright tracker 20 and lefttracker 22, and six pressure values of the right foot and the left foot from the footpressure tracking insole 38 of theright tracker 20 and theleft tracker 22. Theserver computer 30 may receive a total number of input scalar values of 42, where six scalar values are for the position and orientation of theHMD 14 and for each of theright hand controller 18 and theleft hand controller 16, six from each of theIMU 32 of theright tracker 20 and lefttracker 22, and six for each of the footpressure tracking insole 38 of theright tracker 20 and theleft tracker 22, theserver computer 30 produces abody pose 12 output of the participant from the 42 input scalar values. - The
server computer 30 may produce a total number of 48 output scalar values, where six scalar values for the position and orientation of each of the head, hands, feet, upper torso and pelvis, +2 scalar values for each of a right shoulder and left shoulder, +1 scalar value for each of the right elbow, left elbow, right knee and left knee of the participant. The computer produces abody pose 12 output of the participant from the 48 scalar values. The body pose 12 output of the participant may include position and orientation of the right-hand and left-hand, right foot and left foot, upper torso, pelvis, head, forward and upward rotation of each shoulder about the upper torso and inward to outward turning of elbows and knees of the participant. Preferably, there are no external markers or other optical apparatus about any participant in a surrounding space that is used by any participant for the VR where the full body pose 12 of each participant is displayed, or no external markers or other optical apparatus are used to produce a full body pose 12 of each participant in the VR. - The present invention pertains to a
system 10 for producing a shared virtual reality. Thesystem 10 comprises a communication network. Thesystem 10 comprises a plurality of HMDs that are worn by participants to view the shared virtual reality in communication with the network. Thesystem 10 comprises a computer server in communication with the network which produces a full body pose 12 of at least one of participant of the participants and transmits the full body pose 12 via the network to allother participants HMD 14, so that each participant can see the full body pose 12 in each participant's view into the shared virtual reality. The server may produce the full body pose 12 over time. - The present invention pertains to a method for producing a shared virtual reality. The method comprises the steps of producing with a computer server a full body pose 12 of at least one of participant of a plurality of participants. There is the step of transmitting the full body pose 12 via a communications network to an
HMD 14 worn by each of all other participants, so that each participant can see the full body pose 12 in each participant's view into the shared virtual reality. - The present invention pertains to a method for producing a shared virtual reality of participants. The method comprises the steps of receiving
server computer 30 data from an inside-out trackedHMD 14 worn on a head of a participant of the participants and two hand controllers held by the participant and trackers worn on feet or ankles of the participant. There is the step of constructing a full body pose 12 of a participant using only the data from theHMD 14 and the two hand controllers and the trackers. There is the step of displaying the full body pose 12 of the participant in the shared virtual reality which is viewed by theHMD 14. - Given two or more participants, each wearing an
HMD 14, the present invention pertains to a full body pose 12 of any given participant being transmitted via wireless transmission to all other participants, so that each participant can see the full body pose 12 of all other participants in their view into a shared virtual reality world of theHMD 14. - The present invention pertains to a method for reconstructing a VR avatar with full body pose 12 from a participant. The method comprises the steps of sending timestamped data from an
HMD 14 on a head of a participant and aright hand controller 18 held by the participant's right hand and aleft hand controller 16 held by the participant's left hand and feet trackers or ankle trackers on the participant's feet or ankles to a database on aserver computer 30 wirelessly. There is the step of applying mocap data in the database as labeling data by theserver computer 30 to map from input of theHMD 14, the right and left hand controllers and feet trackers or ankle trackers data to output of full body pose 12 of the participant. - The present invention pertains to a method for producing a shared virtual reality of participants. The method comprises the steps of sending data from an
HMD 14, hand controllers and feet or ankle trackers worn or held by each participant to a server computer. There is the step of reconstructing a full body pose 12 for each participant from each participant's data with the server computer. There is the step of sending the full body pose 12 of all participants wirelessly to all HMDs, so that each participant sees the full body pose 12 of all participants in the shared VR displayed in each participant'sHMD 14. - In the operation of the invention, to generate the full body pose 12 from the aforementioned sensor data, the present invention employs machine learning (ML), trained on a ground
truth reference system 10 that can track the full body pose 12 of exemplar users, such as an OptiTrack full-bodyoptical tracking system 10. The method contains two phases: Phase (1) is a training phase, in which an operator uses a full body ground truthmotion capture system 10 while wearing amotion capture suit 24, while also wearing an inside-out trackedHMD 14, holding two VR hand controllers, and employing tracking apparatus for either the feet or ankles, as shown inFIG. 4 . In this first phase, the ground truth full body pose 12 from the reference system is used as a label set to train the ML system that will produce full body pose 12 as output in response to the data from theHMD 14, hand controllers and feet or ankles trackers. Phase (2) is a run-time phase experienced by end users, in which only anHMD 14, hand controllers and inexpensive feet trackers are required. The ML algorithm uses only this run-time data to reconstruct a full body pose 12 for each user. This technique assumes input from the user's hands, be it camera based hand tracking (RGB or marker based IR), inside out tracked gloves via IMUs, neural interfaces, in the spirit of motion touch controllers/6dof remotes, or other method of gaining input from a user's hands. Referred to hereafter as left/right hand input interface. -
FIG. 5 illustrates the essential run-time process, which is a conversion of a small number of runtime sensors into complete position and orientation values for the joints of the body required to construct a full body avatar, referred to herein as the body pose 12. - The computation to convert from sensor data to body pose 12 can be done on a
server computer 30 which can be wirelessly connected to theHMD 14 and foot trackers, such as by Wi-Fi, Bluetooth, other wireless communication technology, or a combination of these; or theserver computer 30 may be located in theHMD 14 itself where one of theHMDs 14 worn by two or more participants is designated as the server computer and interacts with the other HMDs as aremote computer 30 would. The server then can wirelessly transmit the full reconstructed body pose 12 of every user to the HMDs of all users. This enables all users, whether in the same physical location or remotely located, to see each other within the shared virtual experience as full body avatars, as inFIG. 6 . - Conversion from Sensor Data to Body Pose
- In one embodiment, the sensor input may consist of the position and orientation of each of the
HMD 14, theleft hand controller 16 and theright hand controller 18, as well as gyroscopic orientation and accelerometer data from theIMU 32 for each foot, as well as six pressure values for each foot derived by applying weighted area convolutions to the pressure image for that foot from a pressure imaging sensor, as described in U.S. patent application Ser. No. 16/514,597, incorporated by reference herein. - The total number of input scalar values in this embodiment is 42: six for the position and orientation of the
HMD 14 and each of the two hand controllers, plus six for the data from each of the two IMUs, plus six for each of the two foot pressure images after convolution has been applied. - Also in this embodiment, the body pose 12 output may consist of position and orientation of the head, hands, feet, upper torso and pelvis, together with the forward and upward rotation of each shoulder about the upper torso, together with the inward to outward turning of the elbows and knees.
- The total number of output scalar values in this embodiment is 48: six for the position and orientation of each of the head, hands, feet, upper torso and pelvis, plus two for each of the shoulders, plus one each for the elbows and knees.
-
FIG. 7 illustrates the mapping from (1)HMD 14, (2)left hand controller 16, (3)right hand controller 18, (4)IMU 32 and convolution-reduced pressure image data for the left foot, and (5)IMU 32 and convolution-reduced pressure image data for the right foot to (6) the output body pose 12. - In the described embodiment, as the operator moves around while wearing both the input sensors and the full
motion capture suit 24, at each time-step the operator generates both 42 scalar input values (from the input sensors) and 48 body pose 12 values (from the motion capture system). In this way, a very large number of specific examples of a mapping from 42 input values to 48 output values can be accumulated. These can be thought of as “labeled” examples, since each output is being labeled with a corresponding input. - Given a new set of 42 input values X, a corresponding set of 48 output values is computed as follows: For each of the accumulated example mappings [Xi->Yi] from input values to output values, the magnitude of the vector distance between X and Xi is computed. The larger this difference, the less weight we give to output Yi in a weighted sum that creates a best mapping to a final output Y.
- For example:
-
SumW = 0 SumY = 0 For each mapping [Xi −> Yi] Begin W = (1 + | X − Xi | / | X |) −2 SumW += W SumY += W * Yi End Y = SumY / SumW - There may also be many examples of output poses, gathered from databases of human movement. Each such example will contain the 48 output values, but not the 42 input values. These can be thought of as “unlabeled” examples, since each output is not labeled by a corresponding input.
- These unlabeled examples can collectively be used to improve the quality of the final output, so that any given output is modified to better align with nearby unlabeled body poses. After running the above procedure, all of these unlabeled examples are looped over. For each unlabeled example output Yi, its vector distance is measured to the prior result Y. The smaller the distance from Yi to Y, the more some of Yi is mixed into Y.
- For example:
-
For each unlabeled Yi Begin t = (1 + C * | Y − Yi | / | Y |) −2 Y = (1−t) * Y + t * Yi End - A reasonable value for C in the above procedure is ½. Larger values of C can be used to decrease the influence of each example output Yi on the final output, and smaller values of C can be used to increase the influence of each example Yi output on the final output.
- Using an Ml Algorithm to Increase Conversion Efficiency
- Summary:
- The aforementioned conversion can be made much more efficient by using ML to split the computation into two phases: (1) A training phase, which uses data from many sensor data values to construct an efficient representation of the mapping from sensor data values to corresponding body pose 12 values, and then (2) a runtime phase, in which the constructed representation of the mapping is used to efficiently convert new sensor data values to corresponding body pose 12 values.
- ML is employed in order to learn a mapping from an
HMD 14 and two controllers, as well as anIMU 32 and possibly a pressure imaging device and/orproximity sensors 40 placed on the feet or ankles. - The output can vary from, at a minimum, only head, hands and feet positions and orientations to a sparse or, at a maximum, full skeleton reconstruction, depending on the dataset and runtime needs. Semi-supervised learning is used in the form of regression prediction models via variational autoencoders (VAE), which in one embodiment can be adversarial, in order to learn an unknown (latent) manifold of human poses.
- Regression, in its simplest form, is a curve that best fits a given function. In this case, instead of a line or k degree polynomial, the neural net approximates the function by adjusting weights. The classic case of using neural networks for classification, is where lines of separability are drawn between clusters of classes, but here, for 3D joint predictions, the result is not classification, but the output of positions in 3D space, thus the last classification step is removed.
- What is a VAE, and why use it? An autoencoder simply learns an encoding (a latent code) from a set of inputs, and simultaneously learns a decoding to recover the expected output from the code. This code can be overcomplete or undercomplete. Undercomplete is the case where the latent code contains insufficient information to properly map the encoded input to the decoded output. Overcomplete is the opposite case, where the latent code contains more information than is necessary to properly map the encoded input to the decoded output. The layer of the latent code is commonly referred to as a bottleneck layer, where the notion is to force the network to learn only the most salient or useful pieces of the data distribution, ignoring signal noise.
- A VAE includes the above, but also imposes some mathematical structure on the model. The means by which VAEs typically achieve this, is by something known to the ML community as reparameterization. To reparameterize, the parameters of the latent distribution are optimized by sampling from a normal distribution, and shifting the sample by the mean, and scaling by the standard deviation of the latent distribution. The result is a vast reduction in the noise introduced to the output signal.
- VAE and regression are combined by constructing the autoencoder, consisting of N layers with strided residual skip connections between each layer, for both the encoder and decoder, coupled via a bottleneck layer for the latent code, and finally decoding to N joints. In this case, the decoding to N joints is the regression.
- Semi-supervised learning is typically where there is a small amount of labelled data (supervised), and a large amount of unlabeled data (unsupervised). In the particular case of the present invention, the labelled data is the combined operator-captured input from OptiTrack, Rokoko, shoes,
HMD 14+controllers. There is also a rich source of data available online, however, labels for that on-line data are not available. Because the learning algorithm used here is a combination of these labelled data (supervised) and unlabeled data (unsupervised), it is a semi-supervised learning algorithm. - This semi-supervision takes the form of feeding the unlabeled data in intermittently and performing a reconstruction on the input, that is—have the network reconstruct the input given, and the use of reconstruction loss refers minimizing the mean-squared error between the source and generated output.
- Either a single frame of data to recover a skeleton is passed in or a sliding window for a temporally smoothed version of the input is maintained. If a sliding window is maintained, then in one embodiment, three frames would provide acceleration aware prediction, accomplished by adding an additional dimension to the network layers.
- The idea of a sliding window of multiple frames described above, is essentially that by looking at neighboring frames, a finite difference is computed in order to approximate the derivative of the function with respect to time. So instead of having just a set of values, there is a set of values and also their derivatives, and possibly second derivatives as well, with respect to time. Known from elementary mechanics, where the first derivative of position is velocity, and the second derivative is acceleration. Thus, an additional dimension to the network layers serves to account for this, since automatic differentiation [see Automatic differentiation in PyTorch, Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, Adam Lerer28 Oct. 2017 (modified: 28 Oct. 2017) NIPS 2017 Workshop Autodiff Submission, incorporated by reference, herein, for additional information] is a fundamental property that is useful in deep learning frameworks in order to leverage back-propagation [LECUN].
- The dataset used for training includes the following sources:
-
-
HMD 14/controllers - IMU-sensor based Mocap such as Rokoko provides body pose ground truth data
- Optical marker based Mocap such as OptiTrack provides body pose ground truth data
- Foot tracking via
IMU 32 optionally with pressure sensor and/or downward facingproximity sensor 40 - Common 3D pose datasets, such as Human 3.6m which are general and publicly available and wouldn't apply to every layer of the network, but the middle layers of the network of can be frozen and a reconstruction loss can be performed to see whether the model is capable of reconstructing itself and giving it a large source of unlabeled data to help the model generalize onto the pose manifold
-
- The ML training stage:
- For k epochs, where k is typically defined to be in the range of 100 to 200:
-
- Given some batched input, where input is:
- a. Pos{x,y,z}/Orientation {phi theta rho} for
HMD 14, right controller, left controller - b. IMU 32 {pitch, roll, yaw}+acceleration {x,y,z}
- c. Optional foot pressure and/or proximity values
- Feed input to model described above in
step 1, obtain output in the form of N target joints - (Optional) IK to impose realistic joint and limb constraints in order to scaffold the model
- Compute loss by comparing mean per joint position error (MPJPE) between source and ground truth
- for some constant number of iterations:
- d. Feed the network raw input, in order to generate a reconstruction of itself
- e. minimize the mean-squared error between the source and generated output
- Perform back-propagation
- The ML inference/run-time stage:
- 1. Provided time synced network stream of a sliding window for
HMD 14+controllers+IMU 32+optional pressure and/or proximity data - 2. Run input stream through the model
- 3. Recover outputs and send to client via wifi
- 4. use algorithm (optional) in
corresponding training step 3 - 5. Recover 3D pose
- At run-time, after the ML inference has been completed:
- During the run-time, after the 3D pose has been recovered for all users, then the 3D pose for all users is transmitted wirelessly from the
server computer 30 to the HMDs of all users, where the pose data is then used to construct, for each user, 3D computer animated representations of the avatars of all users. - Description of the Feet or Ankles Trackers
- For each of the user's two feet, a foot or ankle tracker is provided.
- Each foot or ankle tracker contains a 6DOF IMU 32 (three degrees of orientation as well as acceleration in x,y,z) and a Bluetooth or Wifi wireless transmitter, together with a
microprocessor 36 to process the tracked data and send it to aserver computer 30 via the wireless transmitter, as well as an electric battery or equivalent to power the foregoing. - Each foot or ankle tracker may also optionally contain a foot
pressure tracking insole 38 as described in U.S. patent application Ser. No. 16/514,597, which generates a pressure image, as shown inFIG. 8 . If this component is included, then the tracker is worn underneath each foot. The inclusion of this component allows the inference of the position of untracked body parts, such as the participant's waist, with a fidelity that would not be possible by tracking only the position and orientation of each foot. For example, if the pressure on either the left or right foot is mainly on the front of that foot, this indicates that the participant's weight is thrust forward in anticipation of walking forward, and therefore it can be inferred that the participant's waist is positioned more forward than the participant's head. Furthermore, if at the same time the other foot has pressure mainly on the rear of the foot, this would indicate the participant is walking and position the waist and knees and thighs and calves of each leg in the appropriate position for walking. Similarly, if there is no pressure anywhere on the other foot, this indicates also the participant is walking and adjust the waist and knees and thighs and calves are in mid-stride and position them accordingly to show the full body pose in mid-stride with the leg having no pressure on its insole being slightly bent and lifted off the ground being positioned to about to receive the weight of the participant as the participant's weight shifts while walking forward. All of this full body pose configuration would be further confirmed by reviewing the next few milliseconds of data to show the pressure changes on the two insoles are changing to correspond to what would be expected from pressure changes from walking, for instance, from data developed during the training phrase when the participant was wearing the motion capture suit and doing various movements including walking. Cases where neither foot is on the ground are examples of the power of combining the various sensing modalities of the foot tracking elements—the foot/ankle tracker's IMU data can be used in conjunction with the pressure data (or lack thereof) to determine user poses in situations such as if the user is jumping or running by the direction of the accelerometer directions. Again, body pose configuration would be confirmed by reviewing the previous and next few milliseconds of data from how the foot or feet left the ground to how the foot/ankle tracking accelerometer data changes and changes. If this component is included, then the tracker is worn underneath each foot. - In order reduce both the bandwidth of Bluetooth communication and the dimensionality (and therefore the runtime cost) of the ML computation, the
microprocessor 36 can optionally perform convolutions on zones of the sensor image before transmission, as described in U.S. patent application Ser. No. 16/514,597, U.S. patent application Ser. No. 16/514,597 and shown inFIG. 9 . This step can reduce the size of the data from on the order of 100-200 values to on the order of 5-10 values. - Alternatively, the tracker can be contained within a strap worn about the user's ankle.
- Each foot or ankle tracker may also optionally contain an
IR proximity sensor 40, a technology which is standard in the art, aiming downward so as to measure time-varying distance to the floor for that foot, as shown inFIG. 10 . In this embodiment, the tracker consists of a strap (1), on which are mounted a battery (2), microprocessor 36 (3), IMU 32 (4) and Bluetooth transmitter 34 (5). Optionally the tracker may also contain aproximity sensor 40 to measure time-varying distance to the floor, which consists of an infrared LED (6) and an infrared sensor (7). The data from theIMU 32 and the optional infrared sensor are sent to themicroprocessor 36, which then transmits the data to theserver computer 30 via Bluetooth. - Inventory of Parts
-
- 1.
HMD 14 - 2. Two VR controllers
- 3. Two feet or ankle trackers
- 4.
Motion capture suit 24 - 5. Optical motion capture system
- 6. Server computer
- 7. Wi-Fi router
- 8. Bluetooth transceiver
- 1.
- Step by Step Operation by User
-
- 1. Each user puts on feet or ankles trackers.
- 2. Each user launches VR app, puts on
HMD 14 and picks up the two controllers. - 3. Users walk around, seeing each other as full body avatars in a shared virtual world.
- Step by Step Internal Operation in Best Embodiment
- At ML (ML) training time:
-
- 1. Operator turns on mocap system and launches mocap program on server computer.
- 2. Operator puts on
motion capture suit 24. - 3. Operator puts on feet or ankles trackers.
- 4. Operator launches VR app, puts on
HMD 14 and picks up the two controllers. - 5. Operator walks around, performing a series of typical human movements.
- 6. During
step 5, mocap program onserver computer 30 analyzes optical data frommotion capture suit 24 and saves time stamped full body pose 12 data to a database. - 7. During
step 5, timestamped data fromHMD 14 and two controllers is sent to the database on theserver computer 30 wirelessly via Wi-Fi. - 8. During
step 5, timestamped data from feet or ankle trackers is sent to the database on theserver computer 30 wirelessly via Wi-Fi or Bluetooth. - 9. Time synchronized information from mocap,
HMD 14, hand controllers and feet or ankle trackers is now stored in a database on the server computer. - 10. ML (ML) training algorithm is applied, using mocap (motion capture) data in database as labeling data to train on mapping from input of
HMD 14, hand controllers and feet or ankle trackers data to output of full body pose 12.
- At run time:
-
- 1. Data from
HMD 14, hand controllers and feet or ankle trackers is sent from each user to server computer, which runs trained ML neural-net for that user to reconstruct full body pose 12 of that user. - 2. Full body pose 12 of all users is sent wirelessly to all HMDs, so that the VR app running in each user's
HMD 14 sees the full body pose 12 of all users.
- 1. Data from
- Example Uses of Invention
- Entertainment Use:
- A multi-participant immersive fantasy world, such as a re-creation of walking around in the Lord of the Rings universe. Each user can choose to appear as a different creature in this fictional universe, such as a Hobbit or an Elf or an Orc. A participant choosing to appear, for example, as a Hobbit, will have a head to foot appearance of a Hobbit to all other participants in the experience.
- Educational Use:
- A teacher is explaining physics to a group of students. Using shared VR, all participants maintain the visual illusion that they are full body avatars in an alternate version of reality. The teacher is able to walk up to a floating model of the solar system and use her controllers to adjust the position of the moon to explain the difference between a solar eclipse and a lunar eclipse. Students are able to walk around and see the simulation together from different points of view.
- Although the invention has been described in detail in the foregoing embodiments for the purpose of illustration, it is to be understood that such detail is solely for that purpose and that variations can be made therein by those skilled in the art without departing from the spirit and scope of the invention except as it may be described by the following claims.
Claims (13)
1. A system for reconstructing a VR avatar with full body pose from a participant comprising:
an inside-out tracked HMD that is to be worn by the participant;
a left-hand controller to be held by a left hand of the participant that produces position data of the left hand of the participant;
a right-hand controller to be held by a right hand of the participant that produces position data of the right hand of the participant;
a right tracker worn on a right foot or right ankle of the participant;
a left tracker worn on a left foot or left ankle of the participant;
a communication network in communication with the left-hand controller and the right-hand controller and the right tracker and the left tracker and the motion capture suit and the motion capture unit; and
a server computer in communication with the network that constructs a full body pose of the participant using real time data from only the right-hand controller and left-hand controller and right tracker and left tracker and HMD and from previously stored data about motion of the participant.
2. The system of claim 1 wherein the right tracker has a 6° of freedom IMU to produce tracked data from the right foot and a wireless transmitter which transmits the tracked data to the server computer through the communications network, the 6° of freedom are 3° of orientation and 3° of acceleration in an X, Y, Z coordinate system, and the left tracker has an imu.
3. The system of claim 2 wherein the right tracker has a microprocessor in communication with the wireless transmitter and the IMU, the microprocessor processes the tracked data and sends the tracked data that has been processed to the server computer through the wireless transmitter.
4. The system of claim 3 wherein the right tracker has a foot pressure tracking insole that is worn underneath the right foot, the foot pressure tracking insole produces pressure data which is provided to the microprocessor, and the left tracker has a foot pressure tracking insole that is worn underneath the left foot.
5. The system of claim 4 wherein the right tracker includes a proximity sensor which measures time-varying distance between the right foot and a floor upon which the participant is walking.
6. The system of claim 5 wherein the server computer receives input that includes position and orientation of each of the HMD, the left-hand controller and the right hand controller, gyroscopic orientation and accelerometer data from the IMU of the right tracker and left tracker, and six pressure values of the right foot and the left foot from the foot pressure tracking insole of the right tracker and the left tracker.
7. The system of claim 6 wherein the server computer receives a total number of input scalar values of 42, where six scalar values are for the position and orientation of the HMD and for each of the right-hand controller and the left-hand controller, six from each of the IMU of the right tracker and left tracker, and six for each of the foot pressure tracking insole of the right tracker and the left tracker, the server computer produces a body pose output of the participant from the 42 input scalar values.
8. The system of claim 7 wherein the server computer produces a total number of 48 output scalar values, where six scalar values for the position and orientation of each of the head, hands, feet, upper torso and pelvis, +2 scalar values for each of a right shoulder and left shoulder, +1 scalar value for each of the right elbow, left elbow, right knee and left knee of the participant, the computer produces a body pose output of the participant from the 48 scalar values.
9. The system of claim 8 wherein the body pose output of the participant includes position and orientation of the right hand and left hand, right foot and left foot, upper torso, pelvis, head, forward and upward rotation of each shoulder about the upper torso and inward to outward turning of elbows and knees of the participant.
10. The system of claim 9 including a motion capture suit to be worn by the participant; and
a motion capture unit which captures motion of the participant from the motion capture suit worn by the participant, the server computer in communication with the network that constructs the full body pose of the participant using real time data from only the right-hand controller and left-hand controller and right tracker and left tracker and HMD and from previously stored data from the motion capture unit.
11. A system for producing a shared virtual reality comprising:
a communication network;
a plurality of inside-out tracked HMDs that are worn by participants to view the shared virtual reality in communication with the network; and
a server computer in communication with the network which produces a full body pose of at least one of participant of the participants and transmits the full body pose via the network to all other participants HMD, so that each participant can see the full body pose in each participant's view into the shared virtual reality.
12. The system of claim 11 wherein the server produces the full body pose over time.
13. A method for producing a shared virtual reality of participants comprising the steps of:
sending data from an inside-out tracked HMD, hand controllers and feet or ankle trackers worn or held by each participant to a server computer;
reconstructing a full body pose for each participant from each participant's data with the server computer; and
sending the full body pose of all participants wirelessly to all HMDs, so that each participant sees the full body pose of all participants in the shared VR displayed in each participant's HMD.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/881,590 US20220374070A1 (en) | 2019-09-12 | 2022-08-04 | System and Method for Reconstructing a VR Avatar With Full Body Pose |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201962899493P | 2019-09-12 | 2019-09-12 | |
US17/016,884 US11409358B2 (en) | 2019-09-12 | 2020-09-10 | System and method for reconstructing a VR avatar with full body pose |
US17/881,590 US20220374070A1 (en) | 2019-09-12 | 2022-08-04 | System and Method for Reconstructing a VR Avatar With Full Body Pose |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/016,884 Division US11409358B2 (en) | 2019-09-12 | 2020-09-10 | System and method for reconstructing a VR avatar with full body pose |
Publications (1)
Publication Number | Publication Date |
---|---|
US20220374070A1 true US20220374070A1 (en) | 2022-11-24 |
Family
ID=74869514
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/016,884 Active US11409358B2 (en) | 2019-09-12 | 2020-09-10 | System and method for reconstructing a VR avatar with full body pose |
US17/881,590 Pending US20220374070A1 (en) | 2019-09-12 | 2022-08-04 | System and Method for Reconstructing a VR Avatar With Full Body Pose |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/016,884 Active US11409358B2 (en) | 2019-09-12 | 2020-09-10 | System and method for reconstructing a VR avatar with full body pose |
Country Status (1)
Country | Link |
---|---|
US (2) | US11409358B2 (en) |
Families Citing this family (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20220051779A1 (en) * | 2020-08-12 | 2022-02-17 | Amazon Technologies, Inc. | Dynamically altering an external geometry of body-wearable actuatable components |
US20220264079A1 (en) * | 2021-02-11 | 2022-08-18 | Kenneth Perlin | Multi-Person Mixed Reality Experience, Method and Apparatus |
CN113181619A (en) * | 2021-04-09 | 2021-07-30 | 青岛小鸟看看科技有限公司 | Exercise training method, device and system |
CN114119857A (en) * | 2021-10-13 | 2022-03-01 | 北京市应急管理科学技术研究院 | Processing method, system and storage medium for synchronizing position and limb of character avatar |
US20230259194A1 (en) * | 2022-02-16 | 2023-08-17 | Meta Platforms Technologies, Llc | Spatial Anchor Sharing for Multiple Virtual Reality Systems in Shared Real-World Environments |
US20240077933A1 (en) * | 2022-09-01 | 2024-03-07 | Daekun Kim | Co-located full-body gestures |
Citations (26)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5592401A (en) * | 1995-02-28 | 1997-01-07 | Virtual Technologies, Inc. | Accurate, rapid, reliable position sensing using multiple sensing technologies |
US5913727A (en) * | 1995-06-02 | 1999-06-22 | Ahdoot; Ned | Interactive movement and contact simulation game |
US5930741A (en) * | 1995-02-28 | 1999-07-27 | Virtual Technologies, Inc. | Accurate, rapid, reliable position sensing using multiple sensing technologies |
US20030030397A1 (en) * | 2000-09-20 | 2003-02-13 | Simmons John Castle | Natural robot control |
US20030120183A1 (en) * | 2000-09-20 | 2003-06-26 | Simmons John C. | Assistive clothing |
US20070003915A1 (en) * | 2004-08-11 | 2007-01-04 | Templeman James N | Simulated locomotion method and apparatus |
US20110009241A1 (en) * | 2009-04-10 | 2011-01-13 | Sovoz, Inc. | Virtual locomotion controller apparatus and methods |
US20110202306A1 (en) * | 2008-08-25 | 2011-08-18 | Universitat Zurich Prorektorat Mnw | Adjustable Virtual Reality System |
US20110306398A1 (en) * | 2010-06-11 | 2011-12-15 | Harmonix Music Systems, Inc. | Prompting a player of a dance game |
US20120116548A1 (en) * | 2010-08-26 | 2012-05-10 | John Goree | Motion capture element |
US20130278494A1 (en) * | 2012-04-18 | 2013-10-24 | Au Optronics Corp. | Three-dimensional interactive system |
US20140276242A1 (en) * | 2013-03-14 | 2014-09-18 | Healthward International, LLC | Wearable body 3d sensor network system and method |
US8979665B1 (en) * | 2010-03-22 | 2015-03-17 | Bijan Najafi | Providing motion feedback based on user center of mass |
US9381426B1 (en) * | 2013-03-15 | 2016-07-05 | University Of Central Florida Research Foundation, Inc. | Semi-automated digital puppetry control |
US9443521B1 (en) * | 2013-02-14 | 2016-09-13 | Sociometric Solutions, Inc. | Methods for automatically analyzing conversational turn-taking patterns |
US20170178272A1 (en) * | 2015-12-16 | 2017-06-22 | WorldViz LLC | Multi-user virtual reality processing |
US20180196505A1 (en) * | 2016-11-30 | 2018-07-12 | Seiko Epson Corporation | Head-mounted display device, computer program, and control method for head-mounted display device |
US20180322681A1 (en) * | 2017-03-02 | 2018-11-08 | Colopl, Inc. | Information processing method, program, virtual space delivering system and apparatus therefor |
US20180350136A1 (en) * | 2017-05-31 | 2018-12-06 | TeMAVR, LLC | Systems and associated methods for creating a viewing experience |
US10403050B1 (en) * | 2017-04-10 | 2019-09-03 | WorldViz, Inc. | Multi-user virtual and augmented reality tracking systems |
US20190354201A1 (en) * | 2018-05-16 | 2019-11-21 | Alcon Inc. | Foot controlled cursor |
US20200004349A1 (en) * | 2018-06-28 | 2020-01-02 | Dell Products L.P. | Information Handling System Touch Device with Automatically Orienting Visual Display |
US20200380784A1 (en) * | 2019-05-29 | 2020-12-03 | Dell Products, L.P. | Concealing loss of distributed simultaneous localization and mapping (slam) data in edge cloud architectures |
US20210149190A1 (en) * | 2019-11-14 | 2021-05-20 | Facebook Technologies, Llc | Co-located pose estimation in a shared artificial reality environment |
US11052288B1 (en) * | 2013-01-19 | 2021-07-06 | Bertec Corporation | Force measurement system |
US20210233304A1 (en) * | 2017-05-31 | 2021-07-29 | LiveCGI, Inc. | Systems and associated methods for creating a viewing experience |
-
2020
- 2020-09-10 US US17/016,884 patent/US11409358B2/en active Active
-
2022
- 2022-08-04 US US17/881,590 patent/US20220374070A1/en active Pending
Patent Citations (35)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5592401A (en) * | 1995-02-28 | 1997-01-07 | Virtual Technologies, Inc. | Accurate, rapid, reliable position sensing using multiple sensing technologies |
US5930741A (en) * | 1995-02-28 | 1999-07-27 | Virtual Technologies, Inc. | Accurate, rapid, reliable position sensing using multiple sensing technologies |
US6148280A (en) * | 1995-02-28 | 2000-11-14 | Virtual Technologies, Inc. | Accurate, rapid, reliable position sensing using multiple sensing technologies |
US5913727A (en) * | 1995-06-02 | 1999-06-22 | Ahdoot; Ned | Interactive movement and contact simulation game |
US20030030397A1 (en) * | 2000-09-20 | 2003-02-13 | Simmons John Castle | Natural robot control |
US20030120183A1 (en) * | 2000-09-20 | 2003-06-26 | Simmons John C. | Assistive clothing |
US6741911B2 (en) * | 2000-09-20 | 2004-05-25 | John Castle Simmons | Natural robot control |
US20070003915A1 (en) * | 2004-08-11 | 2007-01-04 | Templeman James N | Simulated locomotion method and apparatus |
US7542040B2 (en) * | 2004-08-11 | 2009-06-02 | The United States Of America As Represented By The Secretary Of The Navy | Simulated locomotion method and apparatus |
US8868373B2 (en) * | 2008-08-25 | 2014-10-21 | Universitat Zurich Prorektorat Mnw | Adjustable virtual reality system |
US20110202306A1 (en) * | 2008-08-25 | 2011-08-18 | Universitat Zurich Prorektorat Mnw | Adjustable Virtual Reality System |
US20110009241A1 (en) * | 2009-04-10 | 2011-01-13 | Sovoz, Inc. | Virtual locomotion controller apparatus and methods |
US9067097B2 (en) * | 2009-04-10 | 2015-06-30 | Sovoz, Inc. | Virtual locomotion controller apparatus and methods |
US9827478B1 (en) * | 2010-03-22 | 2017-11-28 | Boisensics Llc | Providing visual motion feedback based on sensor data |
US8979665B1 (en) * | 2010-03-22 | 2015-03-17 | Bijan Najafi | Providing motion feedback based on user center of mass |
US20120021833A1 (en) * | 2010-06-11 | 2012-01-26 | Harmonic Music Systems, Inc. | Prompting a player of a dance game |
US20110306398A1 (en) * | 2010-06-11 | 2011-12-15 | Harmonix Music Systems, Inc. | Prompting a player of a dance game |
US20120116548A1 (en) * | 2010-08-26 | 2012-05-10 | John Goree | Motion capture element |
US20130278494A1 (en) * | 2012-04-18 | 2013-10-24 | Au Optronics Corp. | Three-dimensional interactive system |
US11052288B1 (en) * | 2013-01-19 | 2021-07-06 | Bertec Corporation | Force measurement system |
US9443521B1 (en) * | 2013-02-14 | 2016-09-13 | Sociometric Solutions, Inc. | Methods for automatically analyzing conversational turn-taking patterns |
US20140276242A1 (en) * | 2013-03-14 | 2014-09-18 | Healthward International, LLC | Wearable body 3d sensor network system and method |
US9381426B1 (en) * | 2013-03-15 | 2016-07-05 | University Of Central Florida Research Foundation, Inc. | Semi-automated digital puppetry control |
US20170178272A1 (en) * | 2015-12-16 | 2017-06-22 | WorldViz LLC | Multi-user virtual reality processing |
US9990689B2 (en) * | 2015-12-16 | 2018-06-05 | WorldViz, Inc. | Multi-user virtual reality processing |
US20180196505A1 (en) * | 2016-11-30 | 2018-07-12 | Seiko Epson Corporation | Head-mounted display device, computer program, and control method for head-mounted display device |
US20180322681A1 (en) * | 2017-03-02 | 2018-11-08 | Colopl, Inc. | Information processing method, program, virtual space delivering system and apparatus therefor |
US10403050B1 (en) * | 2017-04-10 | 2019-09-03 | WorldViz, Inc. | Multi-user virtual and augmented reality tracking systems |
US20180350136A1 (en) * | 2017-05-31 | 2018-12-06 | TeMAVR, LLC | Systems and associated methods for creating a viewing experience |
US20210233304A1 (en) * | 2017-05-31 | 2021-07-29 | LiveCGI, Inc. | Systems and associated methods for creating a viewing experience |
US20190354201A1 (en) * | 2018-05-16 | 2019-11-21 | Alcon Inc. | Foot controlled cursor |
US20200004349A1 (en) * | 2018-06-28 | 2020-01-02 | Dell Products L.P. | Information Handling System Touch Device with Automatically Orienting Visual Display |
US10761618B2 (en) * | 2018-06-28 | 2020-09-01 | Dell Products L.P. | Information handling system touch device with automatically orienting visual display |
US20200380784A1 (en) * | 2019-05-29 | 2020-12-03 | Dell Products, L.P. | Concealing loss of distributed simultaneous localization and mapping (slam) data in edge cloud architectures |
US20210149190A1 (en) * | 2019-11-14 | 2021-05-20 | Facebook Technologies, Llc | Co-located pose estimation in a shared artificial reality environment |
Also Published As
Publication number | Publication date |
---|---|
US11409358B2 (en) | 2022-08-09 |
US20210081031A1 (en) | 2021-03-18 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11409358B2 (en) | System and method for reconstructing a VR avatar with full body pose | |
JP7001841B2 (en) | Image processing methods and equipment, image devices and storage media | |
US20220035443A1 (en) | Systems and methods for generating complementary data for visual display | |
US8786680B2 (en) | Motion capture from body mounted cameras | |
US20210349529A1 (en) | Avatar tracking and rendering in virtual reality | |
CN108564643B (en) | Performance capture system based on UE engine | |
Roth et al. | A simplified inverse kinematic approach for embodied VR applications | |
Chen et al. | Real-time human motion capture driven by a wireless sensor network | |
JP2023501079A (en) | Co-located Pose Estimation in a Shared Artificial Reality Environment | |
Liu et al. | A real-time interactive tai chi learning system based on vr and motion capture technology | |
US20230023609A1 (en) | Systems and methods for animating a simulated full limb for an amputee in virtual reality | |
CN115964933A (en) | Construction method of virtual and real training device based on digital twins | |
Yang et al. | Dance Posture Analysis Based on Virtual Reality Technology and Its Application in Dance Teac. | |
CN115546247A (en) | Motion capture method based on LightHouse positioning system | |
CN115798042A (en) | Escalator passenger abnormal behavior data construction method based on digital twins | |
Du et al. | Data fusion of multiple kinect sensors for a rehabilitation system | |
US11436806B1 (en) | Dual perspective rendering in virtual reality | |
US20220237874A1 (en) | Method, system, and apparatus for full-body tracking with magnetic fields in virtual reality and augmented reality applications | |
King et al. | Motion capture moves into new realms | |
Che | Optimization of interactive animation capture system for human upper limb movement based on XSENS sensor | |
Brownridge | Real-time motion capture for analysis and presentation within virtual environments | |
Tao et al. | Visual perception method based on human pose estimation for humanoid robot imitating human motions | |
Hristov et al. | Multi-view RGB-D System for Person Specific Activity Recognition in the context of holographic communication | |
Zhao et al. | 3D posture reconstruction and human animation from 2D feature points | |
Klimenko et al. | The Virtual Revival of the City of Saarlouis via a Multimodal Augmented/Mixed Reality Touristic HoloLens App |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |