Realize distributed virtual reality and visualization method and system on the mobile device
Technical field
The present invention relates to a kind ofly realize distributed virtual reality and visualization method at mobile device, belong to virtual reality and technology, figure and visualization technique, network communications technology field.
Background technology
Society, the various application software on mobile device and the mobile device are more and more universal.The latest edition Cortex-A9 version of the mobile computing chip on the present mobile device such as up-to-date embedded-type ARM framework, but the internal memory peak capacity is 512M, core frequency is up to 2GHZ, but its static storage and dynamic capability and calculated performance and tradition all greatly differ from each other based on computer and the server of X86 framework, especially performance the most powerful dedicated version in GPU aspect is Mali-T604 under this framework, its parameter is for supporting OpenGL ES 1.1/2.0, the geometric manipulations capacity limit is per second ten million rank, the pixel filling rate limit is per second 100,000,000 ranks, only can support the 3D digital entertainment of some lightweights, but a width of cloth virtual scene may comprise the dough sheet of 1,000,000 millions in virtual reality and visualization system, also may include a large amount of physical models, the vector line drawing, or other virtual reality content such as illumination, wave, sky etc., finish the calculating of these huge data and draw head and shoulders above calculating and the drawing ability of mobile device, and how to store these mass datas and also face the challenge.
Therefore present mobile device is fully helpless to some senior 3 D image drawings and computing application, lets alone for the reality environment with method of massive graphic data (comprising how much, texture, image etc.) to carry out real-time rendering and carry out relevant visual application.
Summary of the invention
For carrying out 3 D image drawing, virtual reality and visual restriction to computing capability and storage capacity, 3-D graphic disposal ability on the mobile device, the method and system of virtual reality and visualization function have been the object of the invention is to propose realize on a kind of mobile device.
Virtual reality on mobile device and the computing function of visualization system, memory function and three-dimensional image processing function are by being distributed in virtual reality and visual high-performance GPU cluster server is realized, and mobile device itself only provides the functions such as receiving the decode of the parsing of user interaction graphical interface and interactive function, user interactions instruction and transmission, virtual reality and the visual Streaming Media of drawing out, this requirement to the mobile device calculated performance is very low, almost only need possess the function that shows image and get final product.The computational load of whole like this virtual reality and visual application is just almost completely born by the GPU cluster server, the GPU cluster server is responsible for finishing whole virtual reality and is calculated and all functions of drawing, and be responsible for receiving and carrying out user's control command of being sent by mobile device, and result and visual as a result compressed encoding that three-dimensional virtual scene is drawn feed back to mobile device take Streaming Media as carrier.According to this framework we develop prototype system, test result shows, this system has realized carrying out nimbly and freely efficient virtual reality and visualization system function at mobile device, surmount single mobile device far away and can possess and finish the function of calculating and storing, and mutual control also can reach real-time.Medical imaging is visual in addition: digital tomosynthesis, geoscience is visual: the oil and gas exploration, calculation biology: a series of and visual relevant application such as molecular dynamics simulation also can be in the application that realizes under this framework on the mobile device.The leading person Google of digital earth system research and development and application has attempted according to current schema Google Map being transplanted on the mobile device, released GoogleMap Mobile (Google Maps mobile phone version), the user can use online browse whole world image under mobile device such as the PocketPC platform, but it realizes that framework is fully different from our method.
Technical scheme of the present invention is:
Realize distributed virtual reality and visualization method on a kind of mobile device, the steps include:
1) sets up the communication connection of mobile device and virtual reality and visualization server; Wherein, mobile device comprises a graphical interface of user and the client of carrying out virtual reality and visualization display, and virtual reality and visualization server comprise a virtual reality and visualization system;
2) mobile device receives the interactive operation request command by described graphical interface of user, and it is resolved to the interactive command of virtual reality and visualization system;
3) mobile device is packaged into Packet Generation to virtual reality and visualization server with the interactive command of resolving;
4) virtual reality and visualization server parse interactive command from packet, and send it to described virtual reality and visualization system;
5) virtual reality and visualization system are carried out corresponding calculating according to interactive command, then carry out virtually drawing and are that Streaming Media sends to mobile device with the execution result compressed encoding;
6) mobile device shows virtual scene and visualization result to receiving decoding data.
Further, if the interactive operation request command is the order outside the three-dimensional main window view of described graphical interface of user, then virtual reality and visualization server are resolved this order according to the graphical interfaces command messages response function of setting.
Further, if the interactive operation request command is the order within the three-dimensional main window view of described graphical interface of user, its command format is the coordinate of two dimension view, then virtual reality and visualization server with this two dimension view coordinate format command mapping be in the corresponding three-dimensional space object and according to action forward-backward correlation within this window, parse the operational motion order to three dimensional object.
Further, described virtual reality and visualization system comprise a parallel virtually drawing subsystem; After these all virtual realities and Visual calculation task were finished, described parallel virtually drawing subsystem carried out the 3-D graphic virtually drawing.
Further, described parallel virtually drawing subsystem has the GPU cluster to make up; Described parallel virtually drawing subsystem adopts the sort-first strategy to carry out 3 D image drawing.
Further, described user images interface comprises: the container of interactive means in menu, button, toolbar, dialog box, shortcut, three-dimensional navigation graphical interfaces and the three-dimensional scenic.
Realize distributed virtual reality and visual system on a kind of mobile device, it is characterized in that comprising the server end on client, virtual reality and the visualization server on the mobile device; Described mobile device be connected virtual reality and be connected by communication network with visualization server; Wherein, described client comprises a graphical interface of user, a command analysis and sending module, stream media decoding module, Streaming Media drafting module; Described server end comprises a virtual reality and visualization system, orders receiver module, Streaming Media compressed encoding module;
Described graphical interface of user is used for receiving the interactive operation request command;
Described command analysis and sending module are used for the interactive operation request command is resolved to the interactive command of virtual reality and visualization system, and it is packaged into Packet Generation to described order receiver module;
Described order receiver module is used for the packet that receives is stored and resolved, and the interactive command that parses is sent to described virtual reality and visualization system;
Described virtual reality and visualization system are used for carrying out corresponding calculating and virtually drawing according to interactive command, and execution result are sent to Streaming Media compressed encoding module;
Described Streaming Media compressed encoding module, being used for execution result is carried out compressed encoding is Streaming Media, and sends it to described stream media decoding module;
Described stream media decoding module is used for receiving decoding data;
Described Streaming Media drafting module is used for decoded data are drawn demonstration virtual scene and visualization result.
Further, described virtual reality and visualization system comprise a parallel virtually drawing subsystem, are used for that these all virtual realities and Visual calculation task are finished data afterwards and carry out the 3-D graphic virtually drawing.
Further, described parallel virtually drawing subsystem has the GPU cluster to make up; Described parallel virtually drawing subsystem adopts the sort-first strategy to carry out 3 D image drawing.
Further, described server end adopts separate server framework or distributed server architecture.
Compared with prior art, good effect of the present invention is:
Google Maps mobile phone version is only supported general GIS function, map view for example, the functions such as point searching navigation, do not possess the height interaction of similar virtual reality system, to the operability of three-dimensional scenic and controlled, the function (comprising editor's structure, the modification of three-dimensional virtual scene, virtual emulation etc.) that does not more possess the three-dimension virtual reality system.And its three-dimensional function very a little less than, only support very limited pseudo-three-dimensional browse mode, can't carry out browsing and roaming of high accuracy three-dimensional scene, the true sensitivity of the drafting of three-dimensional scenic is poor.
Novelty of the present invention is embodied in: solved the mobile computing device of hardware performance relative mistake finished with existing ordinary individual's computer and high performance graphics work station, server on the real-time virtual reality that can the realize function the same with visualization system, and the backstage adopt the GPU cluster realize virtual reality with visualization function as the server of high-performance calculation so that the user at more perfect on common personal computer and high performance graphics work station of the virtual reality functional experience effect on the mobile device even ratio.
Description of drawings
The operational flow diagram of Fig. 1, moving distributing virtual reality of the present invention and visualization system;
Distributed virtual reality and visualization system structure chart on Fig. 2, the mobile device of the present invention.
Embodiment
Below in conjunction with accompanying drawing, the present invention is further described in detail:
The operational process of moving distributing virtual reality of the present invention and visualization system is as shown in Figure 1:
1) the overall general flow of system is as follows: the client on the mobile device is connected with visual application server with virtual reality, to calculate and operation requests sends to virtual reality applications system on virtual reality and the visual application server, application system will be calculated accordingly with execution result and send to client on the mobile device, and a virtual reality and visual application server can provide support and serve for a plurality of mobile devices;
2) user according to the client on the mobile device to and the result that shows, by the various functions in the mutual whole virtual reality of control of graphic user interface and the visualization system, user interface is divided into two parts: the 1) interactive interface within the three-dimensional main window view of graphical interfaces (perhaps being called the window main frame) (2) outside the three-dimensional main window views such as menu, button, tool bar.Above interface is generated by the client on the mobile device, and server end is not have the interactive graphics (IG) interface.Interactive interface is comprising the three-dimensional virtual scene that passes through in the interactive mode browsing virtual reality systems such as keyboard, mouse, menu graphical interface, select, pick up the three-dimensional object, add, deletion, revise three dimensional object with and correlation behavior and attribute, carry out the various forms of motion controls of three-dimensional body in the virtual scene, adjust various rendering attributes (for example illumination of three-dimensional environment, material, shade, the Realistic Rendering grade), carry out a series of various functions relevant with virtual reality system and the operation such as visual of the set of three-dimensional mass data;
Native system is different from Long-distance Control (for example Remote desk process), the interactive graphics (IG) interface of native system is responsible for by the client on the mobile device fully, server end only provides the service on backstage, can not provide any interactive interface to remote mobile user, and server end can not generate any graphic user interface yet, more can not show any interface content at server, user on the server can freely use server to carry out other various calculating and demonstration task, and the at all existence of imperceptible remote mobile client.Long-distance Control belongs to system-level technology in addition, and the invention belongs to the technology of application layer.
3) the every calculating in the virtual reality system and drafting function are finished jointly by the various server clusters on virtual reality and visual application server and backstage.
● wherein data server is responsible for transmitting data to application server, and the cooperation between the data server and scheduling are to realize by coordination server.In the distributed virtual reality platform, data server is responsible for preservation and the distribution of massive terrains data, visualized data, model data etc.The task of data server is heavier, so in the platform a plurality of data servers can be arranged, and data server comes balanced load by coordination server.The virtual environment data of asking when application server are not during at the current some servers that connect, coordination server can be browsed or the request of the data such as visual figure, image, multimedia calculates the position of data in distributed environment (namely which number of units according to server on) according to user's needs, connected by application server and the data server that stores these data, and carry out obtaining and transmitting of data.
● owing to can have simultaneously a plurality of clients that whole virtual reality is conducted interviews and control in the Distributed Virtual Reality System, it is collaborative that client is moved in reality environment is to be responsible for by central server (collaborative, synchronously, load-balanced server).
● virtual reality and visual application server are mainly finished the drafting function of management, calculating and the three-dimensional environment of virtual reality system, when finishing above function, if the data that need to use are not on current server, then can send request to data server and ask for related data, calculate again after the data to be obtained and the processing such as drafting.Drafting for three-dimensional environment, the hardware configuration of application server can be configuration polylith GPU on the main frame, also can be that multiple host (configuration one or polylith GPU on every main frame) is as drawing asynchronous GPU cluster of the common formation of node, software environment adopts the parallel drawing strategy, generally adopts sort-first parallel drawing strategy.The drafting action of carrying out at virtual reality and visual application server is " virtually drawing ", that is to say this drafting do not have any image result output display on display terminal (such as display, projecting apparatus etc.), but output to virtual image space (such as FBO frame buffer object etc.), the image result that virtually drawing goes out is through making up at application server, and then compressed encoding is sent to mobile client by application server.
The system module structure of distributed virtual reality and visualization system as shown in Figure 2 on the mobile device of the present invention.Whole system is made of two parts, is respectively mobile device client and some virtual realities and visual application server end.The mobile device client runs on the mobile device, and virtual reality and visualization server end run on virtual reality and the visual application server.
1) graphical interface of user: graphical interfaces provides partly that the user carries out mutual various interfaces on mobile client, the container that comprises the various interactive meanses in menu, button, toolbar, dialog box, shortcut, three-dimensional navigation graphical interfaces and the three-dimensional scenic, the user can freely control the various objects in virtual reality and visualization system and the three-dimensional virtual scene by above various interactive interface instruments.(this graphical interfaces is User Exploitation, can utilize any mobile phone graphic user interface developing instrument exploitation).
2) user interaction commands is resolved and sending module: this module at first is responsible for receiving user's operational motion and is resolved the interactive command that becomes whole virtual reality system, because mobile client itself is not really carried out these orders, the execution reality of the operational order in these virtual reality systems all is to be finished by virtual reality and visual application server, is sent to virtual reality and visual application server so this module is responsible for that these interactive commands are broken into packet by wireless network.
3) user command receiver module: at server end special-purpose data buffering memory block is arranged, be used for receiving the mutual control command that mobile client sends.
4) window main frame user command parsing module: according to the difference of the type of user command, if this order is the order on the window main frame of graphical interfaces of mobile client, corresponding graphical interfaces command messages response function is then arranged within this module, resolve this order, and executing the task of will ordering is given to distributed virtual reality and visualization system.
5) three-dimensional main window two dimension is shone upon and parsing module to three-dimensional: if user's mutual control command is within three-dimensional main window view, the command format that then transmits be two dimension view coordinate (for example, if what click is some points within the main window view, then transmit the coordinate of the Pixel-level of this point), utilize this two dimension to the function of three-dimensional inverse mapping, can obtain the object in the corresponding three-dimensional space, and the forward-backward correlation according to the action within window, the practical operation action that parses three dimensional object (for example pulls, mobile etc.), parse after the corresponding action, pass through message passing mechanism, by distributed virtual reality and visualization system response message, and carry out relevant operational order.
5) distributed virtual reality and visualization system: carry out relevant instruction by this Real-time running system, finish every calculation task (comprising scene management, DDM, system resource management, IO management, physical engine, collision detecting system, script engine, computer animation, artificial intelligence, network engine etc.) relevant in the virtual reality system
6) the parallel virtually drawing subsystem of GPU cluster:
The drafting subsystem is the part in virtual reality and the visualization system, among the present invention the GPU cluster is drawn acceleration and processes.
Because the task amount of 3 D image drawing all is the heaviest in virtual reality and the visualization system, often all can become the bottleneck of system's operation, so will drawing this step, we from virtual reality and visualization system, separate, after waiting for that these all virtual realities and Visual calculation task are finished, finished the work of whole drafting by the parallel virtually drawing subsystem of GPU cluster structure.Parallel virtually drawing subsystem adopts the sort-first strategy, namely first with scene according to the difference in corresponding display window zone be divided into 4 sub-window areas (size of 4 subwindows be not all with divide, dynamically adjust according to the size that shows task amount), give respectively 4 GPU with the drafting task of 4 subwindows and carry out parallel drawing, the drafting task amount born of each GPU can reduce like this, has improved the efficient of whole drafting.Simultaneously according to the difference of 4 performances of GPU hardware own, also can be drawn by how much in the partition window zone task amounts and carry out the drafting task according to the corresponding GPU of tactful dynamic assignment of self adaptation adjustment.
7) parallel drawing synthetic image post-processing module: after treating that 4 GPU synthetic images are complete, need in virtual reality and visualization system, carry out recovery operation, 4 width of cloth images that disperse all be fetched and spliced and synthesize a complete image.(known can the realization)
8) Streaming Media compression and sending module: it is that (H.263 compression standard can adopt for a frame of Streaming Media that this module is responsible for image is carried out compressed encoding, H.264 or the multiple standards coding form such as MPEG-4), after through compression and interframe compression in the frame, be sent to the mobile device client via the communication connection.
9) Streaming Media receives and decompression module: this module of mobile device end is after receiving frame of video, and the decoding of establishing criteria codec format reverts to a two field picture.
10) Streaming Media drafting module: the two field picture utilization means of drawing are shown in real time, and the user has just seen and the duplicate virtual scene of virtual reality system and the visualization result of utilizing the calculating of high-performance calculation machine platform to demonstrate in real time like this.