RU54233U1 - VIRTUAL ENVIRONMENT SYSTEM - Google Patents

Abstract

The virtual environment system contains a cluster of personal computers with software for creating virtual scenes and synchronizing the operation of individual system components, a database with a description of virtual models and scenes, a rendering subsystem for graphically transforming the model description into a visual representation on the screen plane, configured to convert the model into two images from the point of view of the left and right eyes of the user, a projection subsystem with one or more screens for visual display of the model. The projection subsystem is stereoscopic and includes two projectors for the left and right video channels, and the optical system of each of the projectors is equipped with polarizing filters, respectively, for the left and right video channels. The system also contains a tracking subsystem for localizing and tracking the user's position, a synthesis subsystem for sound effects with an acoustic system, a power and tactile sensation generation subsystem with a tactile sensation generator to create the illusion of moving in space and touching virtual objects, a data manipulation subsystem, an input device and management, launch and control console system, a subsystem for modeling user interaction with virtual objects. In this case, the tracking subsystem includes an infrared camera and reflective markers mounted on the user's polarizing glasses. This implementation of the system allows for realistic reproduction of the virtual scene by creating stereoscopic images with high resolution and realistic dynamics of virtual objects.

Description

The utility model relates to the means of simulation technology and can be used to create a virtual environment with the ability to interact with three-dimensional models of the studied objects.

The use of virtual environment systems significantly enriches the presentation of large data arrays, allowing you to interact with three-dimensional models of the objects being studied and study their behavior and internal structure. The virtual environment provides the user “immersion” in the artificial world of the phenomenon under study and direct interaction with data in the model space. The virtual environment allows you to switch to a qualitatively higher level of data processing, modeling and designing experiments, creating complex machines and mechanisms, industrial facilities and processes.

It was experimentally established that when a user immerses himself in a virtual environment and initiates a stereoscopic vision of a scene that occurs on a mental level, a person has a more intense exchange of information between the left and right hemispheres of the brain, which stimulates intuitive thinking and the emergence of new ideas when analyzing any problem and / or a large amount of data (especially multidimensional). At the same time, the effect of the emergence of new ideas is enhanced by collective work (the effect of brainstorming).

There is a fairly wide range of fundamental problems of algebraic topology, theoretical mechanics, general and quantum physics, which do not have direct and easily presented analogies and carry a huge amount of information of complex structure, which is often difficult to interpret without involving any visual images. The virtual environment technology allows you to actively attract the intuition of the researcher and his ability to "understand" the image by placing it in the artificial world of the model of the phenomenon being studied, and also by providing the possibility of intuitively simple (direct) manipulation of data in the model space.

Modern computer systems make it possible to simulate complex natural phenomena and solve problems inaccessible to direct experimental verification. Supercomputers (currently the most powerful machines) and

cluster computing systems make it possible to sufficiently and accurately model various exotic experiments and extreme situations proposed by researchers. However, the huge amounts of data obtained by modeling complex phenomena cannot be analyzed without resorting to preliminary research using the active interaction of a person with a computer.

Until recently, only a few visualization techniques have been proposed for special applications. General concepts applicable for use in virtual environment technology to a wide variety of (including non-numerical) applications, as well as a complete mathematical analysis of existing methods, have not yet been developed.

So, visualization systems are used in various simulators, for example, in flight training complexes for the training of flight personnel. The known complex according to patent RU 2263973 C1, including visualization and information display systems with corresponding control systems and computers, pilot controls associated with a flight dynamics computer, automatic computer control systems, flight and navigation equipment, a main processor associated with them, the output of which is connected through the information support computer with control computers of the information display system and the visualization control system, initial task blocks Words and disturbances in flight display controller in the cockpit and the expert system to the database. This complex provides, in particular, visualization of the space surrounding the aircraft in real time, however, a flat (2D) image is formed. In addition, this complex is designed to solve only one problem - simulating the flight conditions of an aircraft.

The technical problem to which the utility model is directed is to expand the functionality and increase the realism of reproduction of a virtual scene and the dynamics of virtual objects.

This problem is solved due to the fact that the virtual environment system contains a cluster of personal computers with software for creating virtual scenes and synchronizing the operation of individual system components, a database with a description of virtual models and scenes, a subsystem

rendering for graphically transforming the description of the model into a visual representation on the screen plane, a projection subsystem with one or more screens for visual display of the model, a tracking subsystem for localizing and tracking the user's position, a synthesis subsystem for sound effects with an acoustic system, a subsystem for generating power and tactile sensations for creating the illusion of moving in space and touching virtual objects, data manipulation subsystem, input and control device control, the launch and control console of the system, the subsystem for modeling user interaction with virtual objects, while the rendering subsystem is capable of converting the model into two images from the point of view of the left and right eyes of the user, the stereoscopic projection subsystem includes two projectors for the left and right video channels, the optical system of each of the projectors is equipped with polarizing filters, respectively, for the left and right video channels, the tracking subsystem for localization and tracking of the user's position includes an infrared camera and reflective markers mounted on the user's polarized glasses, and the power and tactile sensation generation subsystem includes a tactile sensation generator.

The use of a cluster of personal computers in the system provides high computing power of the system, and in combination with a rendering subsystem that has the ability to convert the model into two images from the point of view of the user's left and right eyes, a stereoscopic projection subsystem with two projectors and polarizing filters, a tracking subsystem for localization and tracking the user's position with an infrared camera and reflective markers on the polarizing glasses of the user and subsystems th generation of power and tactile sensations enables realistic virtual scene playback (Stereoscopic image with high resolution), and the realism of the dynamics of virtual objects (believable animation); a sufficiently high frame rate when playing a virtual scene; accuracy of scene reproduction for the points of its “vision” with the left and right eyes; the identity of the scenes visible by the left and right eyes; separation of scenes visible by the left and right eyes, not allowing the mixing of images; overlapping the viewer's field of view with a virtual scene,

providing sufficient “zaporivanie" the viewer of the real scene; sufficiently bright and contrasting image of the scene; recounting the scene when moving the viewer in virtual space; accuracy and speed of creating "tactile sensations."

The virtual environment system is illustrated by its detailed description and the accompanying drawings.

Figure 1 shows a General schematic view of a system in axonometric projection;

figure 2 shows a General schematic view of another design of the system in axonometric projection;

figure 3 shows a functional block diagram of a system.

The virtual environment system (hereinafter - NWO) on a cluster of personal computers includes the following functional elements:

- open source software Avango to create virtual scenes and synchronize the operation of individual components of the system (hereinafter - Avango),

- a database containing a description of virtual models and scenes (hereinafter - the database),

- a rendering subsystem for graphically transforming the model description into a visual representation on the screen plane from the point of view of the user's left and right eyes (hereinafter - PR),

- a stereoscopic projection subsystem with one or more screens for visual display of the model (hereinafter - SPP),

- tracking subsystem for localization and tracking the user's position (hereinafter - PT),

- a subsystem for the synthesis of sound effects (hereinafter - the PZE),

- a subsystem for generating power and tactile sensations, creating the illusion of moving in space and touching virtual objects (hereinafter referred to as VET),

- data manipulation subsystem, input and control devices (hereinafter - PMD),

- a subsystem for modeling user interaction with virtual objects, replacing the user with his interaction with objects and characters in the real world (hereinafter - PMV),

- the console for launching and managing the system (hereinafter - KZU).

In figure 1 and 2 are indicated: the main graphic computer 1; auxiliary calculator 2 for the synthesis of sound effects; a screen 3 of material preserving the direction of the polarization vector upon reflection or transmission of light; projectors 4 and 5, equipped with filters of linear polarization of light; speakers 6 and 7 of the audio system; device 8 input and control. The differences between the systems depicted in figure 1 and figure 2, consist only in the use of the screen either in the lumen (figure 1) or in reflection (figure 2).

During the startup process, the system console initializes the Avango kernel, which deploys the remaining components of the system. Avango is a programming environment developed at the Fraunhofer Institute for Media Communications that is designed to form virtual environment applications. Avango uses the C ++ programming language to define two categories of objects - nodes and sensors. Nodes make up an object-oriented scene graph that represents and displays complex geometry. Sensors provide communication with the real world and are used to enter data from external devices into the application. All Avango objects are field containers that represent information about the state of the object as a set of fields. Avango objects support a universal stream interface, which allows you to write objects and information about their state to the stream, and subsequently reconstruct this data from the stream. Avanto defines the relationships between the fields, forming a graph of data flows, which is conceptually orthogonal to the scene graph and serves to enter real data into the scene, necessary for modeling interactive behavior. In addition to C ++, Avango supports the interpreted language Scheme. Scheme is a universal programming language derived from Algol and Lisp. It is a high-level language that can handle structural data such as strings, lists, and vectors. All Avango objects can be described on the Scheme. Avango uses the OpenGL Performer graphics system, which allows you to achieve the highest possible graphics acceleration and also has a number of

special properties required when developing virtual environment applications. Advanced tasks, such as erasing invisible edges, switching granularity, and communicating with hardware, are entirely performed by the OpenGL Performer system.

The software of the whole system as a whole is built on the basis of the Linux OS using the tools of the Avango virtual environment tool system and C ++ language.

In the process, the system console remains open, providing the user with monitoring tools and full control over the system. After launching, Avango uses the database to create a virtual scene and starts the main processes: PT and PR, as well as auxiliary processes: PZE, PTO and PMD. When using the cluster architecture, each process runs on its own cluster element and is controlled over the network.

The PT process determines the position of the user's head (eyes and ears), as well as the position and orientation of the interaction devices. The result is transferred to the core of Avanto. The PR process renders the virtual scene in accordance with the specified position of the user's eyes and transfers the result to the NGN subsystem for display on the screen. According to instructions from the core, the GES process synthesizes and reproduces sound effects in accordance with the position of the user's ears.

The VET process simulates force effects and transmits the result to a tactile sensation generator. The PMD process monitors discrete events, such as keystrokes on interaction devices, and transfers the result to the Avango core. Based on the signals from the PT and PMD, the PMV block models the user’s interaction with virtual objects, in particular, determines the speed and method of movement in the virtual scene, allows you to manipulate virtual objects, etc.

The essence of the utility model is illustrated in the block diagram (Fig. 3), which shows the functional diagram of the virtual environment system on a cluster of personal computers. The following is the content of the individual blocks of the system:

9. The console launch and control the system (KZU).

10. Avanto.

11. The core of the Avanto.

12. C ++ object component of Avanto.

13. SCM script component Avango.

14. Avanto graphical user interface.

15. A database containing a description of virtual models and scenes (DB).

16. Hierarchical representation of virtual models in the form of a scene graph.

17. Files with a description of the geometry of virtual models stored on an external medium (hard disk, DVD, flash-memory, etc.).

18. The rendering subsystem for the graphic transformation of the model description into a visual representation on the screen plane from the point of view of the user's left and right eyes (PR).

19. Graphic library OpenGL Performer.

20. Graphic library OpenGL.

21. A graphics card with 3D acceleration, having 2 video outputs.

22. Stereoscopic projection subsystem with one or more screens for visual display of the model (SPP).

23. Projector of the left video channel.

24. The projector is the right video channel.

25. Polarizing filter of the left video channel.

26. Polarizing filter of the right video channel.

27. Tracking subsystem for localization and tracking the user's position (PT).

28. Avanto sensor, interface to the driver of the tracking device.

29. Tracking device driver.

30. Tracking device (infrared camera).

31. Physical sensors (reflective markers).

32. The subsystem for the synthesis of sound effects (PZE).

33. Avanto sensor, an interface to a sound effects synthesizer.

34. Synthesizer sound effects jMax.

35. Sound card with two audio channels.

36. The speaker of the left audio channel.

37. Speaker of the right audio channel.

38. The subsystem for generating power and tactile sensations, creating the illusion of moving in space and touching virtual objects (VET).

39. Avanto sensor, interface to the tactile sensation driver.

40. Tactile sensation driver.

41. The generator of tactile sensations.

42. Subsystem of data manipulation, input and control devices (PMD).

43. Avanto sensor, interface to the driver of the interaction device.

44. Interaction device driver.

45. Interaction device (joystick).

46. Physical sensors (joystick buttons).

47. The subsystem for modeling user interaction with virtual objects, (PMV).

48. The model of navigation in the virtual space.

49. The virtual point of view.

50. A model of interaction with virtual objects.

51. Virtual manipulators (green beam, virtual hand, etc.).

The system operates as follows. After turning on all hardware components (computers running processes 9, 10, 18, 27, 32, 38, 42, projectors 23, 24, sound system 36, 37, devices 30, 45 for tracking and interaction, generator 41 tactile sensations) and starting the system from the control console 9, the Avango kernel 11 is initialized, which deploys three main software components: C ++ object component 12 Avango, SCM script component 13 Avango and graphical user interface 14 Avango. These components during the operation of the system closely interact with each other and with the core 11.

The control console 9 provides access to the core of the system through a script interface 13 using commands in the Scheme language, as well as through a graphical user interface 14. Next, the geometric representation of the virtual models is loaded from files 17 into RAM, forming a tree structure in it (scene graph 16) necessary for the optimal organization of the rendering process 18. Rendering is done graphically

OpenGL Performer library 19, which optimizes graphic processes using hierarchical data organization, algorithms for erasing invisible lines and surfaces. The result is transmitted to the OpenGL graphics library 20, which renders primitives (triangles, lines, dots) taking into account graphic modes (lighting, textures, transparency), using 21 lower-level techniques implemented in the graphics card (Z-buffering, masking, shading).

Rendering is done for the points of view of the left and right eyes, the received images are transmitted through two separate video channels to a stereo projection system 22. The system consists of two identical projectors 23, 24 equipped with polarizing filters 25, 26 that are complementary to each other. A stereoscopic image is observed by the user on a special a screen that preserves the polarization vector using polarizing glasses, the filters of which are oriented parallel to the filters of the projectors.

To determine the position of the points of view of the user's eyes required during rendering 18, the tracking subsystem 27 is used. This subsystem measures the coordinates of reflective markers 31 attached to polaroid glasses, as well as to interaction devices 45, for which their position is recorded by infrared camera 30 and converted to coordinates a special program - the driver of the tracking device 29, executed in the background. The resulting data is transmitted through the interface 28 to the Avango core 11 for further use in the rendering process 18.

According to the instructions of the kernel 11, the interface 33 starts the synthesis of sound effects 34 reproduced by the sound card 35 using the stereo sound system 36, 37. Also, the commands of the kernel 11 are transmitted through the interface 39 to the driver 40, which simulates the force effects reproduced by the tactile sensation generator 41. When pressed Buttons 46 on the interaction device 45, the signal is transmitted through the driver 44 and interface 43 to the Avango core 11. This data is used by block 47 for modeling user interaction with virtual objects, in particular, block 48 determines the speed and method of moving the virtual point of view 49, block 50 determines the interaction with virtual objects by means of manipulators 51. The simulation results are transmitted to rendering process 18.

This system can be implemented using, for example, the following hardware-software configuration.

1. Computing system - a cluster of 4 personal computers. Personal computer configuration (3 pcs.):

- motherboard class ASUS P5AD2;

- processor class Pentium IV 3.6 Ghz;

- RAM 2 Gb Dual channel;

- two hard drives with a volume of at least 120 Gb;

- housing with good ventilation.

The configuration of the terminal personal computer (1 pc):

- motherboard class ASUS P5AD2 Premium;

- processor class Pentium IV 3.6 Ghz;

- video card PNY PCI-E on a chip Nvidia Quadro FX1300 256 Mb;

- RAM 2 Gb Dual channel;

- two hard drives with a volume of at least 120 Gb;

- housing with good ventilation;

- LCD monitor 15-17 '', CD-ROM, keyboard, mouse - standard.

Switch

- 4-port Gigabit Ethernet switch with at least 4 Gb / s bus performance.

2. Tracking subsystem

The tracking system includes a set of infrared emitters, video cameras and reflective marks on tracking objects. Reflective tags are mounted on stereo glasses and input and control devices. Emitters and video cameras are mounted in the space near the screen, so that radiation does not directly enter the cameras. After processing the images from the reflectors, their coordinates in the physical space are calculated and, if necessary, translated into the coordinates of the virtual space. The minimum number of cameras needed to restore tracking coordinates is two, but for reliability, more cameras are usually used. The system does not require a clock signal. The calculated user coordinates are used by the graphics subsystem to represent the scene from the user's point of view.

The proposed localization and tracking system provides localization accuracy of at least 5 mm in space with a frequency of at least 100 hertz.

3. Stereoscopic projection subsystem

- Two Panasonic PT-L735E class projectors (luminosity of at least 3000 ANSI lumens).

- an adjustment platform with three angular degrees of freedom,

- a screen of the Silver Screen type with a system of smooth (without wrinkles) tissue tension, which preserves the polarization vector of reflected light.

4. Software subsystem

The software toolkit for creating distributed interactive VE applications is the Avango package. Avango, a programming environment developed at the Fraunhofer Institute for Media Communications and adapted for personal computers, is designed to create distributed, interactive applications in a virtual environment.

The use of generally available high-performance personal computers, elements of a virtual environment, free software, as well as the inclusion of developed software modules in the system made it possible to create a hardware-software complex that provides solutions to problems previously available only to complex and expensive complexes.

Claims (1)

A virtual environment system containing a cluster of personal computers with software for creating virtual scenes and synchronizing the operation of individual system components, a database with a description of virtual models and scenes, a rendering subsystem for graphically transforming a model description into a visual representation on screen planes, a projection subsystem with one or multiple screens for visual display of the model, tracking subsystem for localization and tracking the user's position, subscription him synthesis of sound effects with an acoustic system, a subsystem for generating power and tactile sensations to create the illusion of moving in space and touching virtual objects, a data manipulation subsystem, an input and control device, a launch and control console for the system, a subsystem for modeling user interaction with virtual objects, characterized the fact that the rendering subsystem is capable of converting the model into two images from the point of view of the left and right eyes of the user While fading, the projection subsystem is stereoscopic and includes two projectors for the left and right video channels, the optical system of each of the projectors equipped with polarizing filters, respectively, for the left and right video channels, while the tracking subsystem for localization and tracking the user's position includes an infrared camera and reflective markers mounted on the polarizing glasses of the user, and the subsystem for generating power and tactile sensations includes a tactile sensing generator puppies.