KR20100084597A - Immersive collaborative environment using motion capture, head mounted display, and cave - Google Patents

Abstract

PURPOSE: A collaborative immersive environment using motion capture, an HMD and a CAVE is provided to evaluate a design while not constituting a prototype and to perform individual training without the prototype or location movement. CONSTITUTION: A motion capture system receives CAD(computer-aided design) data and is a VR simulator generating 3D(Dimensional) VR(Virtual Reality) simulation of a real size from the CAD data. The 3D VR simulation is indicated on a 3D HMD(Head Mount Display)(40) as a different projected figure. An observation system makes a user look at the VR simulation in three dimensions and in real time. The observation system makes the user form a collaborative immersive environment and makes other user evaluate a CAD design.

Description

Immersive Collaborative Environment Using Motion Capture, Head Mounted Display, and CAVE}

TECHNICAL FIELD The present invention relates generally to virtual reality and motion capture, and more particularly to systems and programs that enable human interaction in real and virtual environments using motion capture.

There are a variety of technologies that allow users to interact or analyze the environment. For example, motion capture technology is used in sports, medical and entertainment (especially video games and animation). In sports, for example, motion capture enables digital recording for golfer's swing analysis. In the medical field, motion capture can be used to provide feedback to patients during orthopedic rehabilitation training. For example, the movement of a walking patient is used to represent an accurate or inaccurate technique. In the field of animation, motion capture enables digital recording of actors' movements and facial expressions into computer models. The animator then uses the actor's recorded movement as the basis for the movement of the computer-generated character. As such, video games use motion capture to facilitate the animation of animated characters in the game.

Virtual reality technologies allow the user to react in a computer-simulated environment. Most virtual reality environments rely on computer screens or stereoscopic displays and are primarily visual experiences. A popular example of virtual reality technology is a flight simulator video game. Here the player controls a virtual aircraft in a computer simulation environment.

Telepresence refers to a technology that allows users to experience distance or feel as if they are remote. For example, if the display is of sufficient size and quality to make the user feel remote, telepresence includes a remote video camera that the user can use pan, tilt and zoom. .

None of these technologies alone provide a collaborative immersive environment for evaluating designs through interactions, virtual training of tasks, and simulation validation using live video. can not do it.

In order to solve the above problem, an embodiment of the present invention provides, for example, a collaborative visualization system. It combines motion capture and virtual reality with kinematics and CAD (computer aided design), especially for the purpose of evaluating designs.

Embodiments of the present invention are also simulated using, for example, portable motion capture systems that allow one or more people to respond in real and virtual environments, and design evaluation, virtual training of tasks, and, above all, live video. It provides a head mounted display (HMD) that verifies. Further embodiments of the present invention provide an object (object) to be tracked by the motion capture system and to be included in the virtual reality simulation. An example of a collaborative visualization system is described using one or more concurrent users, one or more external observers, real-time interaction and scaling (scaling), simulation and depresence views. It may include a sensational environment that enables a transition between them. The sensational environment may, for example, generate three-dimensional or three-dimensional images that appear to surround the viewer. That is, the viewer is "immersed" in the virtual environment.

Embodiments of the invention include, for example, a virtual reality simulator. The virtual reality simulator receives data from the CAD design and displays a simulation of the data, for example, to the user via a head mounted display (HMD), thereby producing a full-scale stereoscopic image (image). ) Is generated. For example, the image can be represented in detail so that the user can read the words on the working knob and switch in the simulation.

Embodiments of the present invention further include a motion capture system embedded in a virtual reality simulator. For example, a motion capture system can track a user's movements and reactions within a virtual reality simulation. This allows the head-mounted display (when the head of the user (wearing a suit of motion capture ensemble, including a flexible and adjustable level of detail device, from head to full bodysuit and gloves) rotates or tilts). The image of the simulation provided in HMD) changes correspondingly. The motion capture system can store tracked user movements and interactions for subsequent use, including, for example, for training or design evaluation purposes. In addition, the motion capture system tracks the movements and interactions of multiple users in the virtual reality simulation, allowing multiple users to be represented as avatars in real time within the simulation. Thus, several users can simulate coordinated activities (eg, performing routine maintenance on an aircraft) within a simulation. In addition, the motion capture system can track the movement and interaction of the object, including, for example, tools and props used by the user.

Embodiments of the present invention may include an immersive observation system that allows multiple observers to view, in real time, virtual reality simulations involving interaction of avatars in a common, immersive environment for evaluating CAD designs. have. An immersive viewing system can include a CAVE (Cave Automatic Virtual Environment), such as an 8-foot-height X 10-foot-width X 10-foot-length room, with a display on three walls and floors. have. Here, one or more observers view a shared environment that includes stereoscopic full-scale (actual size) images in a realistic and interactive manner. Effectively, CAVE enables designers of CAD designs to interact extensively with the proposed design in the simulator to analyze and evaluate the design without having to create costly prototypes. Can be observed. For example, an observer can see a user in a simulation performing regular maintenance operations on an aircraft. In addition, the trainer can observe the trainer performing various tasks, for example, by looking at the avatar of the trainer in response to the recorded motion capture data.

According to an embodiment of the present invention, the virtual reality simulator may adjust the size of each avatar in real time. In other words, the user of 5'4 "in the simulation can be adjusted in real time to be the avatar of 6'2" in the simulation. The image provided on the user's head mounted display (HMD) of 5'4 "corresponds to the perspective view anticipated by someone of 6'2". The observer of the simulation will see an avatar of 6'2 ". Scaling can be accomplished by adding a ratio for each aspect of the data upon change from motion capture data to simulation data. Scaling can be done in real time by placing the avatar's head, arms and legs in the correct position so that the kinematics software solves the problem of the remaining joints.Scaling can also be done by subsequent processing, as opposed to real time. .

According to an embodiment of the invention, the virtual reality simulator may include interaction with the simulated environment. In one such example, the virtual reality simulator may include collision detection software to provide feedback within the simulation. If the user pushes with his hands where the simulation is represented by the wall, for example a virtual collision is detected. The movement of the hand may be set to stop in a collision or disappear from the user's field of view, and the panel of the wall may be set to change color (or any other action) to provide feedback and indicate that a collision has occurred. . According to a preferred configuration, the wall turns red. Similarly, if the knee "collides" with the toolbox in the simulation, the toolbox turns red. In the configuration example, the sound is caused by the collision, and the sound may be a directional sound indicating the direction of the collision for the user or observer. Various types of sounds may further provide collision related information, such as the intensity of the collision or the object (or object) involved in the collision. For example, a user whose head hits a portion of the aircraft may make a different sound than objects that hit the floor. The simulation can also change behavior based on collisions detected, for example by opening a door or panel.

Embodiments of the invention may also include portable motion capture for capturing work in the field. Such systems include motion tracking markers (which may be present on suits, bodies, or other clothing), and multiple cameras mounted on tripods or fixed on rigid structures, so that the cameras can be moved by a user wearing a motion capture marker. Tracking, and the computer records the image from the camera. Portable motion capture systems allow remote procedures (eg field maintenace operations) to be recorded. Due to the inherent properties of the virtual reality simulator and motion capture system provided in accordance with an embodiment of the present invention, for the real-time evaluation of an existing design or for a sequence of motions for an existing design, or for interaction with a new design, a virtual reality simulator The data from the field preservation operation can then be studied. Further, according to an embodiment of the present invention, the portable motion capture system can be usefully used for real-time design evaluation, design representation, and training at remote locations in the field. Evaluation of a design may include, for example, an evaluation of the design of the environment, an analysis of the biotechnological aspects of the design, a study of work related to the design, and anything else that a person skilled in the art can do. have.

Furthermore, embodiments of the present invention include a method for verifying a simulation using live video using immersive techniques. For example, according to an embodiment of this method, a spherical camera captures live video or live still pictures at remote locations. Thereafter, a video or photo is provided to the head mounted display HMD. The motion capture system collects the user's head rotation information, which is used to control the pan, tilt, and zoom of the video. The user can then switch the display and simulation of the live video by verifying the simulation. Video can also be displayed on the desktop or CAVE. For example, using a spherical camera to capture live-action images from the tech of an aircraft carrier can be used to verify simulation of such an environment.

Embodiments of the invention, as will be appreciated by those skilled in the art, include, for example, multiple concurrent users and external observers, use real-time interaction and scaling (scaling), simulation and telepresence. Systems and methods that provide a sensory environment using the ability to switch between views.

According to this embodiment of the present invention, there is provided an improved method for evaluating a design without constructing a prototype, and for personal training without prototyping or repositioning.

Hereinafter, the present invention will be described in detail with the accompanying drawings and examples.

Although defined differently, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Although the same or similar methods as described herein can be used in practice or to test the invention, suitable methods are described herein.

The above-described embodiments of the present invention are for illustration and description only, and are not intended to limit the present invention to the described form. Accordingly, various changes and modifications can be made to those skilled in the art to which the present invention pertains. In addition, the detailed description of this specification does not limit the scope of the present invention. Like reference numerals denote like elements throughout the specification.

As shown in FIGS. 1 and 10, an embodiment of the present invention includes a collaborative visualization system 20. It integrates motion capture and virtual reality technology with kinematics and CAD, first and foremost, to validate simulations using design evaluations, virtual training of tasks, and live action video. In addition, embodiments of the present invention include, for example, a realistic viewing environment.

Embodiments of the present invention include, for example, a virtual reality simulator 58 for creating a virtual reality environment. The virtual reality simulator may receive data from the CAD program and generate a virtual reality simulation of the design projected to the user via the head mounted display 40 as shown in FIGS. 1, 2 and 5. In addition, the simulation can be displayed via the desktop or CAVE 44. Virtual reality simulation is three-dimensional and allows users to investigate, evaluate, and interact with CAD designs. Simulation environments are classified as realistic. Because the simulation is 3D and full-scale. The user's field of view can be rotated throughout the simulation, leaving the user immersed in the simulation.

Embodiments are provided for design evaluation of aircraft, space systems, spacecraft, ships, and missile systems. It often uses a costly evaluation process that typically requires, for example, expensive full-scale models and prototypes (models). As will be appreciated by those skilled in the art, the high cost evaluation process may include ergonomic analysis and task analysis for operation and maintenance work.

According to an embodiment of the present invention, the virtual reality software used comprises ENVISION D5 of DELMIA. As will be appreciated by those skilled in the art, this software creates a 3D environment for physics-based, 3D environments, in particular design, verification, and rapid modeling of conceptual designs involving structures, mechanical systems and humans. to provide. As will be appreciated by those skilled in the art, the software has the ability to evaluate system and subsystem level models, including physics-based motion, virtual reality, and highly accurate 3D simulation, and ergonomic evaluation of analysis and visualization. Enhance using. Further, according to an embodiment of the present invention, other software components used to create a virtual reality environment include PolyWorks of InnovMetric Software, a software tool used to convert point cloud data into polygons; NuGraf of Okino Computer Graphics, polygon transformation and reduction tool; Deep Exploration or Deep Server from Right Hemisphere, a polygon transform and collapse tool; And Microsoft Visula Studio from Microshoft, a code development suite. According to an embodiment of the present invention, the hardware supporting this software includes: 4 Dell Precision Workstations 670, 16 x DVD-ROM, 48/32 CDRW, Dual 3.0 Ghz Xeon, 2 MB L2 cache, 800 FSB 4 GB RAM, n Vidia Quadro FX3400 Includes 256 MB, 136 GB HD. As will be appreciated by those skilled in the art, a virtual reality simulator includes a computer program that is stored on one or more computer readable media and can be read by a computer, as described below. When read by a computer, the computer program operates to perform various instructions.

1, 2 and 3, in accordance with an embodiment of the present invention, motion capture system 30 may, for example, display markers at known locations on the suit (e.g., above the knee, waist and shoulders). 52, a user wearing a bodysuit 32, gloves 34, and a headgear 36. The motion capture system may further include a real object or prop with markers to be represented in the virtual reality simulator as well. Camera 54 (shown in FIG. 8B) digitally records the position of the marker and captures a data set as the user moves around or interacts with the simulation. This motion capture data can be used in the virtual reality simulator 58 in real time, such that the user and their movements are modeled into the avatar 56 in the simulation, and feedback is provided to the user. An avatar is an electronic image, which is represented by being realized in the form of a person, generally in a computer game or other virtual reality setting, and manipulated and driven by a computer user.

According to an embodiment of the present invention, motion capture system 30 is a camera 54 (eg, 12 Eagle-i and 12 Hawk-i from Motion Analysis Corporation, shown in FIG. 8B), head mounted display 40. Truss 38 (e.g., LxWxH is 20'x15'x10) used to move and shoot six co-users wearing gloves 34 (e.g., TALON gloves from Motion Analysis Corporation) and body suit 32 It includes up to 24 cameras mounted in 'im). According to an embodiment of the present invention, motion capture system 30 uses the software of Motion Analysis Corporation and includes the following plug-ins. That is, it includes Animation Plugins, RT2 Animation Plugins, Calcium 4, Talon Streaming 4, Talon Viewer 4, and EVaRT5. As will be appreciated by those skilled in the art, this software is specialized to allow templates and props to be created and to track everything from physical models to the human body. In addition, VRSim's SimIO module can be used to multicast data collected by EVaRT for use by simulation engine software ENVISION D5. As will be appreciated by those skilled in the art, other numbers of cameras may be included, for example, in embodiments.

According to an embodiment of the present invention, the virtual reality simulator may adjust the size of each avatar in real time, for example. That is, a 5'4 "user in the simulation can be scaled to a 6'2" avatar in the simulation in real time. An image provided on a 5'4 "user's head mounted display corresponds to a 6'2" perspective intended by someone. The observer of the simulation will see an avatar of 6'2 ", and the pose of the avatar matches the pose of the user. Scaling is applied to each aspect of the data when switching from motion capture data to simulation data. Optionally, scaling is accomplished by having the kinematic software address the problem with other joints (joints) by placing the avatar's head, hands, and feet in the correct position. It shows four avatars scaled to different sizes (all driven by the same user), and therefore all take the same pose (or pose). Be careful. That is, for example, to distance train a user (trainee) with a first size of 6'2 ", the first size (eg 6'2"). The avatar may be displayed in a first size and a second, different size (e.g., 5'4 ") user in response to the motion capture data from the (trainer) of.

1, 2 and 5, according to an embodiment of the present invention, the user visually experiences a virtual reality environment through the head mounted display 40, and each of the head mounted displays is viewed in different perspectives. It can include a simulation. It may include a stereo display helmet worn by the user for tangible visualization of the full scale data. As will be appreciated by those skilled in the art, Virtual Research's VR1280, a type of head mounted display 40, has a display performance of 60Hz, 1280x1204 in mono or stereo. As will be appreciated by those skilled in the art, the head mounted display individualizes the left eye and right eye displays into different images, allowing the user to view images in three dimensions on the head mounted display.

In one embodiment of the invention, the software used to display the 3D scene may be based on the OpenSceneGraph 3D graphics toolkit. VRSim's MiniViz is a viewer that allows users to see running ENVISION (D5) simulations. This viewer loads the model in the environment and references the ENVISION (D5) simulation and tracked viewpoints for the model's location.

Embodiments of the present invention provide a computer workstation that supports a head mounted display. In one embodiment of the invention, the hardware used to support four head mounted displays includes 4 Dell Precision Workstation 670, 16x DVD-ROM 48/32 CDRW, Dual 3.0 Ghz Xeon, 800 FSB, 4GB with 2MB L2 cache. RAM, nVidia Quadro FX3400 256 MB, 136 GB HD. For convenience and as will be appreciated by one of ordinary skill in the art, Hergo Easy Mount Mobile Computer Cart-Star Base 42 (shown in FIG. 4), according to an embodiment of the present invention, comprises a set of devices (computer, keyboard, mouse, head). Mount display 40, including Talon glove 34 and Hergo's flat panel monitor).

According to an embodiment of the present invention, the virtual reality simulator may include, for example, interaction with a simulated environment. In this example, the virtual reality simulator may include collision detection software to provide feedback within the simulation. If the user stabs his hand where the simulation indicates that the wall is present, a virtual collision is detected. The hand may stop at crash time or disappear from the user's field of view, and the panel color of the wall changes (or other actions take place) to provide feedback and indicate that a collision has occurred. In a preferred configuration, the walls turn red. Similarly, when the knee "collides" with the bomb in the simulation, the bomb turns red. In one embodiment, the appearance of the object is changed in response to the collision. In one embodiment, the sound is caused by the collision. Sound is a directional sound that indicates the direction of the collision for the user or observer. Various types of sounds may further provide information about the collision, such as the intensity, or information about the objects involved in the collision. For example, a user's head hitting a part of an aircraft sounds different from objects colliding with the floor. In addition, the simulation changes its behavior based on the detected collision, for example by salting a door or panel. This collision detection feature can also be used to facilitate grabbing the virtual object within the simulation, thereby allowing the object to be moved or manipulated in the virtual environment. Collisions can also occur between avatars and between simulated objects such as between simulated hand tools and simulated walls.

Embodiments of the invention may also include an immersive viewing environment. The viewing environment allows the designer to observe and interact with the virtual reality simulation including the avatar 56 (see FIG. 9) and may be driven in real time by motion capture data from the user. As shown in FIG. 1, the viewing environment is a Cave (Cave Automatic Virtual Environment) 44, using a display on three walls and floors, for example, 8 feet high x 10 feet wide x 10 feet long. Possible rooms may include. In this room, one or more observers can view a common environment in a realistic way, including stereoscopic and full-scale images. As will be appreciated by those skilled in the art, Mechdyne's CAVE 44 displays a resolution of 1280 x 1024 x 96 at 96 Hz. Thus, designers and other observers can see detailed interactions of users in the simulation in real time. In a configuration example, one wall of CAVE 44 can be used to display all individual view points with the other two walls and floor of CAVE 44 that will be used to display the entire scene. In this configuration example, the simulated immersive observer can simultaneously view the avatar 56 pose and avatar view point. Other available data (such as temperature, distance measurement, time, etc.) may be displayed or displayed on one wall of CAVE 44 according to this configuration example, whether viewed or not.

In one embodiment of the invention, the software used to display the 3D scene is based on OpenSceneGraph, a 3D graphics open source tool kit. Much like a standalone viewer for a head mounted display, CAVE 44 can use a MiniViz software program. However, the projection of each screen can be mapped using Mechdyne's application programmer's interface (API) called CAVELib. Operating CAVE 44 may also include other programs, such as Vega, Vega Prime, and Ensight Gold, in accordance with embodiments of the present invention.

Embodiments of the present invention may include head tracking devices 46 (see FIG. 6) and wand hardware 48 (see FIG. 7) that allow the observer to easily move around in the simulation. According to one embodiment of the invention, the InterSense Is-900 uses a combination of inertial and ultrasonic sensors to determine the position of the head tracking device and wand. The software that runs the head tracking device and wands includes a program called VRCO's Trackd 5.5. As will be appreciated by those skilled in the art, the Trackd application receives information from the head tracking device and wands and makes this information available to the CAVELib or ENVISION (D5) simulation. This tracking information can be applied to modify the projections on the walls and floor of the CAVE 44.

Embodiments of the present invention may also include a sensational environment having the ability to switch simulation and telepresence views, for example. Telepresence view is possible via the CAVE 44, the head mounted device 40 or the desktop display. Information about the telepresence view is collected through a spherical camera 50 (see FIG. 8A) at a remote location (eg, surface of the aircraft carrier). According to an embodiment of the invention, the spherical camera 50 comprises a set of six digital cameras built into one device that captures 75% of the sphere. Dynamic scenes may be displayed on the head mounted display 40 or CAVE 44. Here, the user can feel the live video information to help verify the simulation. As will be appreciated by those skilled in the art, Ladybug2 is a 4.7 MegaPixel video capture device using six 1024x768 CCDs. According to an embodiment of the present invention, Ladybug2 of Point Gray Reserach comes with a ladybug2 SDK, a software development environment, and a tool called LadybugCap, which enables content capture. Such software creates spherical avi files for playback, as will be appreciated by those skilled in the art. In one embodiment, Ladybug 3 is used to obtain better resolution but with a lower frame rate.

As will be appreciated by those skilled in the art, spherical camera 50 generates a 1GB incremental data file (which can be 2 seconds or 2 minutes). Thus, 30 seconds of video capture translates into 15 files per GB. These files need to be converted into a viewable format. Another solution uses a video editor, in which the first viewable file is pasted together to form a single video file, in a process of decreasing quality to produce a smaller second level video file. In contrast, an embodiment of the present invention reads a first level video file into a buffer and stores the first file, the last, current, the file before the current and the file after the current file. Provide an index such as (the file after the current). This allows a video group with many files to be played as if they were a single video file.

In accordance with an embodiment of the invention, a photorealistic object may be scanned to create a model for a virtual reality simulator. In one embodiment of the present invention, Creatform Handy Scan EXAscan may be used. In one embodiment of the invention, Lecica HDS 3000 is used. The Lecica HDS 3000 is a laser-based device that scans devices to quickly create high quality 3D models where 3D models do not yet exist. As will be appreciated by those skilled in the art, in practice, the actual resolution of a scanner is 1/1 for data modeled based on 1/4 "point precision for unmodeled data and multiple points from a point cloud. 8 "point precision. According to an embodiment of the invention, the software used to capture the area, set the resolution, and register the point cloud is Cyclone. PolyWorks is also used to register the cloud and generate polygonal shapes that will be inserted into the simulation. According to an embodiment of the invention, the hardware supporting the device may be a Dell Precision M70 Laptop, 1.86 GHz, 1 GB RAM and Nvidia Quadro FX Go 1400.

Embodiments of the invention may include, for example, a portable motion capture system for capturing work in the field. According to an embodiment of a portable motion capture system, the system includes a marker at a predetermined location, a motion capture suit, a plurality of cameras mounted on a tripod, so that the camera tracks the movement of a user wearing a motion capture suit, A computer or other storage medium that records an image from a camera is included. According to another embodiment of a portable motion capture system for capturing tasks in the field, the system includes a plurality of cameras including a marker at a predetermined location, a motion capture suit, and fixed to a rigid structure. A computer or other medium that tracks the movement of a user wearing the suit and records images from a camera and digitally records the position of a marker connected to the body suit in response to images recorded from the plurality of cameras. . In a training application, for example, the movement of the remote trainer present at the first location can be tracked and captured behind or in real time for the training of the trainee at the second location. In addition, the movement of one or more users can be tracked and captured for task analysis (including, for example, ergonomic and efficiency analysis). For example, in another embodiment, a portable motion capture system can be used for real-time design evaluation in the field and for example design presentations or demonstrations involving new designs.

Embodiments of the present invention include a method of evaluating engineering design, as shown in FIG. The method includes simulating a CAD design in a virtual reality environment (step 80) and driving one or more avatars within the virtual reality simulation using motion capture data obtained from a user interacting with the virtual reality simulation (step 81). Include. The method continuously displays a virtual reality simulation that includes the interaction of one or more avatars for multiple observers in the collaborative sensory environment to evaluate the CAD design (step 82), thereby implementing the product according to the CAD design. ) Can be performed with a user size of a predetermined range.

An embodiment of the present invention includes a method for verifying a simulation using live video using sensory techniques, as shown in FIG. 12. The method includes capturing live video with a spherical camera located remotely (step 90) and providing video to a head mounted display (step 91). The method continues by capturing the user's head rotation information by the motion capture system (step 92) and controlling the pan, tilt and zoom of the video by the user's head rotation information. Step 93 (step 93). The method may also include switching live video display and simulation (step 94), under the control of the user (step 94).

Embodiments of the invention also include computer programs (products) that are stored on actual computer memory media and that operate on a computer. This computer program includes a set of instructions that, when executed by a computer, cause the computer to perform various operations. For example, these actions may be provided to receive CAD data, generate video signals from the CAD data for simulation of a virtual reality design, to track each other and multiple users interacting with the simulation, and to track objects interacting with the simulation. Generating a scaled avatar in the simulation, generating a video signal for a common sensory environment, and receiving user input to select either video, graphics, or both.

In addition, an embodiment may include a computer program stored on one or more computer readable media and readable by a computer, such that when the computer program is read by the computer, it operates to perform the instructions described herein. The instructions include the operation of recording full-scale motion capture data at the first location for one or more trainers performing one or more tasks by the portable motion capture system. The instruction responds to motion capture data recorded for one or more trainers in the first position, thereby moving the one or more avatars in the virtual reality simulation by the virtual reality simulator in the second position, such that each of the one or more trainers has one or more. An action to correspond to one of the avatars. The command includes displaying a virtual reality simulation, including three or more moving avatars, as a three-dimensional image viewed around one or more trainees to define a common sensory environment using one or more head mounted displays. Accordingly, one or more trainees can analyze one or more tasks performed. The command may further include obtaining motion capture data for one or more trainees interacting with the virtual reality simulation via the motion capture system; In response to motion capture data for one or more trainees at the second location, moving the one or more avatars within the virtual reality simulation by the virtual reality simulator in real time; Detecting a collision between the avatar moving by the trainee and the simulation object of the virtual reality simulation by the virtual reality simulator; And changing the color of the simulation object of the virtual reality simulation by the virtual reality simulator to provide feedback on the detected collision.

Embodiments of the present invention include systems for training in remote locations, for example, tasks related to the operation or maintenance of an aircraft. Similarly, the work may relate to the operation or maintenance of an aircraft, space system, spacecraft, ship or missile system.

Embodiments of the present invention further include a method of simulating a task. The method includes recording full-scale motion capture data for one or more users performing one or more tasks by a portable motion capture system. The method includes the step of causing each of the one or more users to correspond to one of the one or more avatars by moving one or more avatars in the virtual reality simulation by the virtual reality simulator in response to motion capture data for the one or more users. . The method includes displaying a virtual reality simulation that includes one or more moving avatars as a three-dimensional image using one or more head mounted displays, such that each of the one or more head mounted displays provides different projections for the virtual reality simulation. .

The system of the present invention is a first location arranged to track movement of one or more users (eg trainers) and record life-size motion capture data for one or more users (eg trainers) performing one or more tasks. In the handheld motion capture system 30, 42. The system is arranged to receive recorded motion capture data from a first location and to move one or more avatars 56 in the three-dimensional virtual reality simulation at a second, different location in response to the recorded motion capture data. (58). The system may include an immersive observation system for displaying a virtual reality simulation that includes one or more moving avatars 56 as three-dimensional images, wherein the three-dimensional images appear to surround around one or more trainees, thereby providing one or more trainees. A common sensory environment 20 is defined using the head mounted display 40. Thus, each of the one or more head mounted displays 40 will have different virtual reality simulations, and one or more trainees can analyze one or more performed tasks.

Although embodiments in accordance with the present invention have been described as fully functional systems, those skilled in the art will appreciate that some or more mechanisms of the present invention and / or various aspects of the present invention may be written with various forms of instructions executed on a processor. It may be distributed in a computer readable medium format, and it is readily understood that the present invention applies equally regardless of the specific type of signal that produced the medium used to actually perform the distributed operation. Examples of computer readable media include, but are not limited to, nonvolatile, hard-coded media (such as ROM, CD-ROM, and DVDD-ROM) or electrically programmable read only memory (EEPROM), recordable media (such as floppy disk, hard disk). Drives, CD-R / RW, DVD-R / RW, DVD + R / RW, flash drives and other new types of memory), and transmission type media such as digital and analog communication links. However, the present invention is not limited thereto. For example, such media may include operational instructions and operational instructions related to the design and evaluation program (product) and the method steps described above.

The above-described embodiments of the present invention are for illustration and description only, and are not intended to limit the present invention to the described form. In addition, the detailed description of this specification does not limit the scope of the present invention. It is apparent to those skilled in the art that various changes and modifications can be made within the spirit and scope of the invention as described in the detailed description above and defined by the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS In order for the present invention to be fully understood and to have substantial effect, preferred embodiments of the present invention, but not limited thereto, are described below with reference to the accompanying drawings.

1 is a diagram illustrating a system that provides a collaborative sensory environment for evaluating engineering design in accordance with an embodiment of the present invention.

FIG. 2 illustrates an environment of four users wearing a motion capture device that interacts with a virtual reality simulation, in accordance with an embodiment of the invention.

3 is a diagram illustrating the periphery of a motion capture glove, in accordance with an embodiment of the invention.

4 is a perspective view illustrating a hergo easy mount mobile computer cart-star base in accordance with an embodiment of the present invention.

5 is a perspective view illustrating a head mounted display HMD according to an embodiment of the present invention.

6 is a perspective view showing a head tracker used in connection with a CAVE according to an embodiment of the present invention.

7 illustrates wand hardware to be used in connection with a CAVE according to an embodiment of the present invention.

8A is a perspective view illustrating a spherical camera according to an embodiment of the present invention.

8B is a perspective view of a motion capture camera in accordance with an embodiment of the invention.

9 is a perspective view illustrating an avatar in a virtual reality simulation in accordance with an embodiment of the present invention.

10 is a block diagram illustrating a collaborative visualization system according to an embodiment of the present invention.

11 is a flowchart illustrating a method of providing a collaborative sensory environment for evaluation of engineering design in accordance with an embodiment of the present invention.

12 is a flowchart illustrating a method of verifying a simulation using photorealistic video using immersive techniques in accordance with an embodiment of the present invention.

Claims (15)

In a system for evaluating engineering design, the system is: A virtual reality simulator arranged to receive CAD data for a computer-aided design (CAD) design, the virtual reality simulator arranged to generate a full-scale, three-dimensional virtual reality simulation from the CAD data, wherein the three-dimensional virtual reality simulation is one or more. A virtual reality simulator characterized by being displayed on one or more stereoscopic head mounted displays such that the stereoscopic head mounted displays each have a different degree of projection of the virtual reality simulation; A motion capture system inserted into the virtual reality simulator and arranged to simultaneously track movement of one or more users in the virtual reality simulation; And Wherein the one or more users view the virtual reality simulation in three dimensions and in real time to create a collaborative sensory environment, wherein the one or more users includes a sensory observation system for evaluating the CAD design, The virtual reality simulator is arranged such that the avatars in the simulation represent one or more users and that each of the one or more avatars is scaled independently and in real time, associated with the product configuration according to the CAD design by a predetermined range of users' sizes. Verifying that work can be performed, and The virtual reality simulator is arranged to detect a collision between a user and a simulation object in the virtual reality simulation and to provide feedback for the collision if detected. The method of claim 1, Feedback for the detected collision comprises modifying the appearance of the object and producing a directional sound. The method of claim 1, The motion capture system is further arranged to track the movement of one or more real objects used by the one or more users, and The virtual reality simulator further represents the one or more real objects in a simulation in response to the tracked movement of the one or more real objects, retrieves a collision between two or more simulation objects in the virtual reality simulation, and detects the detected Engineering design validation system, arranged to provide feedback about a collision. The system of claim 1, wherein the motion capture system is: One or more motion capture bodysuits, gloves and headgear having markers in predetermined locations; A plurality of cameras disposed in a preselected position to record an image capturing the movement of one or more users wearing the motion capture bodysuit, gloves and headgear; And One or more computers that digitally record the position of the marker relative to the motion capture bodysuit, gloves and headgear as the one or more users interact in the simulation in response to recorded images from the plurality of cameras. Engineering design verification system comprising a. The method of claim 1, The sensory observation system comprises a configurable display unit comprising three walls and a floor, The display unit provides a three-dimensional life-size image for the one or more users, the engineering design verification system, characterized in that to define the CAVE (Cave Automatic Virtual Environment). The method of claim 1, The CAD design data is an engineering design verification system, characterized in that for the aircraft. The method of claim 1, The CAD design data is an engineering design verification system, characterized in that for spacecraft. The method of claim 1, Engineering design verification system, characterized in that the CAD design data is for ships. The method of claim 1, Engineering design verification system, characterized in that the CAD design data is for a missile system. The method of claim 1, The motion capture system is a portable motion capture system disposed in a first position, The virtual reality simulator and sensory observation system are arranged in a second position, And the second position is remote from the first position. A computer readable medium having recorded thereon a computer program operative to execute the following instructions when read by a computer, the instructions comprising: Receiving computer-aided design (CAD) data by a virtual reality simulator; Generating a video signal by the virtual reality simulator to simulate a design from a CAD data display in virtual reality, wherein the video signal is generated such that one or more head mounted displays each have a different projection of virtual reality simulation. A video signal generation instruction characterized in that the head mounted display includes a separate left eye display and a right eye display each having a different signal, thereby allowing the user to view a stereoscopic image on the head mounted display. ; Instructions for tracking movement of one or more users interacting with the virtual reality simulation; Generating motion capture data for the one or more users in response to the tracked movement of the one or more users; And Wherein the one or more users each correspond to one of one or more avatars, the first sized user simulates a second sized avatar different from the first size, and wherein the one or more users evaluate the CAD design. Instructions for real time driving the one or more scaled avatars in a simulation by the virtual reality simulator using motion capture data for one or more users; Instructions for detecting, by the virtual reality simulator, a collision between a user and a simulation object in the virtual reality simulation; And Modify an object appearance in the virtual reality simulation by the virtual reality simulator to provide feedback for the detected collision And a computer readable medium having recorded thereon a computer program. The computer program of claim 11, wherein the computer program comprises: And further execute instructions to provide a directional sound in response to the detected collision. The computer program of claim 11, wherein the computer program comprises: Instructions for tracking movement of one or more real objects used by the one or more users; Generating motion capture data for one or more real objects in response to the tracked movement of the one or more real objects; Expressing the one or more real objects in a simulation by the virtual reality simulator in response to the motion capture data; Instructions for detecting a collision between two or more objects simulated in a virtual reality simulation by the virtual reality simulator; And And further execute, by the virtual reality simulator, instructions to change the appearance of an object in the virtual reality simulation to provide feedback for the detected collision. In a computer-implemented engineering design evaluation method, the method is: In response to CAD data from a computer-aided design (CAD) design, generating a three-dimensional virtual reality simulation by a virtual reality simulator; In response to motion capture data obtained from one or more users interacting simultaneously with the virtual reality simulation, the one or more users each move the one or more avatars in real time within the virtual reality simulation by the virtual reality simulator. Corresponding to one of the one or more avatars; A virtual reality simulation including interactions of the one or more avatars is displayed in real time with three-dimensional images visible around one or more users to form a common sensory environment using one or more head mounted displays, thereby enabling the one or more users. Has different projections of the virtual reality simulation in response to the motion capture data; Detecting a collision between a user and a simulation object in the virtual reality simulation by the virtual reality simulator; And Providing feedback on the detected collision Engineering design evaluation method comprising a. The method of claim 14, And wherein said feedback for the detected collision comprises at least one of an appearance condition of the object and a directional sound.

Images