Realize distributed virtual reality and visualization method and system on the mobile device
Technical field
The present invention relates to a kind of distributed virtual reality and visualization method on mobile device, realized, belong to virtual reality and technology, figure and visualization technique, network communications technology field.
Background technology
Current society, the various application software on mobile device and the mobile device are more and more universal.The latest edition Cortex-A9 version of mobile computing chip on the present mobile device such as up-to-date embedded-type ARM framework, but the internal memory peak capacity is 512M, core frequency is up to 2GHZ, but its static storage and dynamic capability and calculated performance and tradition all greatly differ from each other based on the computer and the server of X86 framework, especially performance the most powerful dedicated version in GPU aspect is Mali-T604 under this framework, its parameter is for supporting OpenGL ES 1.1/2.0, the geometric manipulations capacity limit is per second ten million rank, the pixel filling rate limit is per second 100,000,000 ranks, only can support the 3D digital entertainment of some lightweights, but a width of cloth virtual scene may comprise the dough sheet of 1,000,000 millions in virtual reality and visualization system, also may include a large amount of physical models, the vector line drawing, or other virtual reality content such as illumination, wave, sky etc., finish these huge data computing and draw the calculating and the drawing ability of mobile device head and shoulders above, and how to store these mass datas and also face the challenge.
Therefore present mobile device is powerless fully to some senior 3 D image drawings and computing application, lets alone at the reality environment with magnanimity graph data (comprising how much, texture, image etc.) to carry out real-time rendering and carry out relevant visual application.
Summary of the invention
At carrying out 3 D image drawing, virtual reality and visual restriction on the mobile device, the method and system of virtual reality and visualization function have been the objective of the invention is to propose realize on a kind of mobile device to computing capability and storage capacity, 3-D graphic disposal ability.
Computing function, memory function and three-dimensional image processing function at virtual reality on the mobile device and visualization system are realized by being distributed on virtual reality and the visual high-performance GPU cluster server, and mobile device itself only provides functions such as receiving the decode of the parsing of user interaction graphical interface and interactive function, user interactions instruction and transmission, virtual reality and the visual Streaming Media of drawing out, this requirement to the mobile device calculated performance is very low, and the function that almost only need possess display image gets final product.The computational load of whole like this virtual reality and visual application is just almost completely born by the GPU cluster server, the GPU cluster server is responsible for finishing whole virtual reality and is calculated and all functions of drawing, and be responsible for receiving and carrying out user's control command of sending, and result and visual compressed encoding as a result that three-dimensional virtual scene is drawn are that carrier feeds back to mobile device with the Streaming Media by mobile device.According to this framework we develop prototype system, test result shows, this system has realized carrying out efficient virtual reality and visualization system function nimbly and freely on mobile device, surmount far away single mobile device can possess and finish and calculate and the function of storage, and mutual control also can reach real-time.Medical imaging is visual in addition: digital tomosynthesis, geoscience is visual: oil and natural gas exploration, calculation biology: a series of and visual relevant application such as molecular dynamics simulation also can be in the application that realizes under this framework on the mobile device.The leading person Google of digital earth system research and development and application has attempted according to current schema Google Map being transplanted on the mobile device, released GoogleMap Mobile (Google Maps mobile phone version), the user can use online browse whole world image under mobile device such as the PocketPC platform, but it realizes that framework is different fully with our method.
Technical scheme of the present invention is:
Realize distributed virtual reality and visualization method on a kind of mobile device, the steps include:
1) sets up communicating to connect of mobile device and virtual reality and visualization server; Wherein, mobile device comprises a graphical interface of user and carries out the client of virtual reality and visualization display that virtual reality and visualization server comprise a virtual reality and visualization system;
2) mobile device receives the interactive operation request command by described graphical interface of user, and it is resolved to the interactive command of virtual reality and visualization system;
3) mobile device is packaged into packet with the interactive command of resolving and sends to virtual reality and visualization server;
4) virtual reality and visualization server parse interactive command from packet, and send it to described virtual reality and visualization system;
5) virtual reality and visualization system are carried out corresponding calculated according to interactive command, carry out virtually drawing then and are that Streaming Media sends to mobile device with the execution result compressed encoding;
6) mobile device is decoded to receiving data, shows virtual scene and visualization result.
Further, if the interactive operation request command is the order outside the three-dimensional main window view of described graphical interface of user, then virtual reality and visualization server are resolved this order according to the graphical interfaces command messages response function of setting.
Further, if the interactive operation request command is the order within the three-dimensional main window view of described graphical interface of user, its command format is the coordinate of two dimension view, then virtual reality and visualization server with this two dimension view coordinate format command mapping be in the corresponding three-dimensional space object and according to action forward-backward correlation within this window, parse operational motion order to three dimensional object.
Further, described virtual reality and visualization system comprise a parallel virtually drawing subsystem; After these all virtual realities and Visual calculation task were finished, described parallel virtually drawing subsystem carried out the 3-D graphic virtually drawing.
Further, described parallel virtually drawing subsystem has the GPU cluster to make up; Described parallel virtually drawing subsystem adopts the sort-first strategy to carry out 3 D image drawing.
Further, described user images interface comprises: the container of interactive means in menu, button, toolbar, dialog box, shortcut, three-dimensional navigation graphical interfaces and the three-dimensional scenic.
Realize distributed virtual reality and visual system on a kind of mobile device, it is characterized in that comprising the server end on client, virtual reality and the visualization server on the mobile device; Described mobile device is connected by communication network with visualization server with described virtual reality; Wherein, described client comprises a graphical interface of user, a command analysis and sending module, stream media decoding module, Streaming Media drafting module; Described server end comprises a virtual reality and visualization system, orders receiver module, Streaming Media compressed encoding module;
Described graphical interface of user is used to receive the interactive operation request command;
Described command analysis and sending module are used for the interactive operation request command is resolved to the interactive command of virtual reality and visualization system, and it is packaged into packet sends to described order receiver module;
Described order receiver module is used for the packet that receives is stored and resolved, and the interactive command that parses is sent to described virtual reality and visualization system;
Described virtual reality and visualization system are used for carrying out corresponding calculated and virtually drawing according to interactive command, and execution result are sent to Streaming Media compressed encoding module;
Described Streaming Media compressed encoding module, being used for execution result is carried out compressed encoding is Streaming Media, and sends it to described stream media decoding module;
Described stream media decoding module is used for decoding to receiving data;
Described Streaming Media drafting module is used for decoded data are drawn demonstration virtual scene and visualization result.
Further, described virtual reality and visualization system comprise a parallel virtually drawing subsystem, are used for that these all virtual realities and Visual calculation task are finished data afterwards and carry out the 3-D graphic virtually drawing.
Further, described parallel virtually drawing subsystem has the GPU cluster to make up; Described parallel virtually drawing subsystem adopts the sort-first strategy to carry out 3 D image drawing.
Further, described server end adopts separate server framework or distributed server architecture.
Compared with prior art, good effect of the present invention is:
Google Maps mobile phone version is only supported general GIS function, map view for example, functions such as point searching navigation, do not possess the height interaction of similar virtual reality system, to the operability of three-dimensional scenic and controlled, the function (comprising editor's structure, the modification of three-dimensional virtual scene, virtual emulation etc.) that does not more possess the three-dimension virtual reality system.And its three-dimensional function very a little less than, only support very limited pseudo-three-dimensional browse mode, can't carry out browsing and roaming of high accuracy three-dimensional scene, the true sensitivity of the drafting of three-dimensional scenic is poor.
Novelty of the present invention is embodied in: solved finished on the mobile computing device of hardware performance relative mistake with existing ordinary individual's computer and high performance graphics work station, server on the real-time virtual reality function the same that can realize with visualization system, and the backstage adopts the GPU cluster to realize that as the server of high-performance calculation virtual reality and visualization function make the user at the virtual reality function experience effect on the mobile device even than the perfection more on common personal computer and high performance graphics work station.
Description of drawings
The operational flow diagram of Fig. 1, moving distributing virtual reality of the present invention and visualization system;
Distributed virtual reality and visualization system structure chart on Fig. 2, the mobile device of the present invention.
Embodiment
Below in conjunction with accompanying drawing, the present invention is further described in detail:
The operational process of moving distributing virtual reality of the present invention and visualization system is as shown in Figure 1:
1) the overall general flow of system is as follows: the client on the mobile device is connected with visual application server with virtual reality, to calculate and operation requests sends to virtual reality applications system on virtual reality and the visual application server, application system sends to client on the mobile device with corresponding calculated and execution result, and a virtual reality and visual application server can provide support and serve for a plurality of mobile devices;
2) user receives and result displayed according to the client on the mobile device, by the every function in mutual whole virtual reality of control of graphic user interface and the visualization system, user interface is divided into two parts: the 1) interactive interface within the three-dimensional main window view of graphical interfaces (perhaps being called the window main frame) (2) outside the three-dimensional main window views such as menu, button, tool bar.Above interface is generated by the client on the mobile device, and server end is not have the interactive graphics (IG) interface.Interactive interface is comprising the three-dimensional virtual scene that passes through in the interactive mode browsing virtual reality systems such as keyboard, mouse, menu graphical interface, select, pick up the three-dimensional object, add, deletion, revise three dimensional object with and correlation behavior and attribute, carry out the various forms of motion controls of three-dimensional body in the virtual scene, adjust various rendering attributes (for example illumination of three-dimensional environment, material, shade, the sense of reality is drawn grade), carry out a series of every function and the operations relevant such as visual of three-dimensional mass data set with virtual reality system;
Native system is different with Long-distance Control (for example remote desktop connection), the interactive graphics (IG) interface of native system is responsible for by the client on the mobile device fully, server end only provides the service on backstage, can not provide any interactive interface to long-range mobile subscriber, and server end can not generate any graphic user interface yet, more can on server, not show any interface content, user on the server can freely use server to carry out other various calculating and demonstration task, and the existence of imperceptible long-range mobile client at all.Long-distance Control belongs to system-level technology in addition, and the invention belongs to the technology of application layer.
3) every calculating in the virtual reality system and drafting function are finished jointly by the various server clusters on virtual reality and visual application server and backstage.
● wherein data server is responsible for to the application server Data transmission, and cooperation between the data server and scheduling are to realize by coordination server.In the distributed virtual reality platform, data server is responsible for the preservation and the distribution of massive terrains data, visualized data, model data etc.The task of data server is heavier, so in the platform a plurality of data servers can be arranged, and data server comes balanced load by coordination server.The virtual environment data of asking when application server are not when current some servers that connect, coordination server can be browsed or the request of data such as visual figure, image, multimedia calculates the position (promptly on which platform data server) of data in distributed environment according to user's needs, connect the obtaining and transmit of the line data of going forward side by side by application server and the data server that stores these data.
● owing to can have a plurality of clients that whole virtual reality is conducted interviews simultaneously and control in the Distributed Virtual Reality System, it is collaborative that client is moved in reality environment is to be responsible for by central server (collaborative, synchronously, load-balanced server).
● virtual reality and visual application server are mainly finished the drafting function of management, calculating and the three-dimensional environment of virtual reality system, when finishing above function, the data that use if desired are not on current server, then can send request and ask for related data, wait to obtain to calculate again after the data and processing such as drafting to data server.Drafting at three-dimensional environment, the hardware configuration of application server can be configuration polylith GPU on the main frame, also can be that multiple host (configuration one or polylith GPU on every main frame) is as drawing asynchronous GPU cluster of the common formation of node, software environment adopts the parallel drawing strategy, generally adopts sort-first parallel drawing strategy.The drafting action of carrying out at virtual reality and visual application server is " virtually drawing ", that is to say that this drafting does not have any image result output and is presented on the display terminal (as display, projecting apparatus etc.), but output to virtual image space (as FBO frame buffer object etc.), the image result that virtually drawing goes out is through making up on application server, and compressed encoding is sent to mobile client by application server then.
The system module structure of distributed virtual reality and visualization system as shown in Figure 2 on the mobile device of the present invention.Whole system is made of two parts, is respectively mobile device client and some virtual realities and visual application server end.The mobile device client runs on the mobile device, and virtual reality and visualization server end run on virtual reality and the visual application server.
1) graphical interface of user: graphical interfaces provides partly that the user carries out mutual various interfaces on mobile client, the container that comprises the various interactive meanses in menu, button, toolbar, dialog box, shortcut, three-dimensional navigation graphical interfaces and the three-dimensional scenic, the user can freely control the various objects in virtual reality and visualization system and the three-dimensional virtual scene by above various interactive interface instruments.(this graphical interfaces is user's exploitation, can utilize any mobile phone graphic user interface developing instrument exploitation).
2) user interaction commands is resolved and sending module: this module at first is responsible for receiving user's operational motion and is resolved the interactive command that becomes whole virtual reality system, because mobile client itself is not really carried out these orders, the execution reality of the operational order in these virtual reality systems all is to be finished by virtual reality and visual application server, is sent to virtual reality and visual application server so this module is responsible for that these interactive commands are broken into packet by wireless network.
3) user command receiver module: at server end special-purpose data buffering memory block is arranged, be used to receive the mutual control command that mobile client sends.
4) window main frame user command parsing module: according to the difference of the type of user command, if this order is the order on the window main frame of graphical interfaces of mobile client, corresponding graphical interfaces command messages response function is then arranged within this module, resolve this order, and executing the task of will ordering is given to distributed virtual reality and visualization system.
5) three-dimensional main window two dimension is shone upon and parsing module to three-dimensional: if user's mutual control command is within three-dimensional main window view, then the command format of Chuan Diing be two dimension view coordinate (for example, if what click is some points within the main window view, then transmit the coordinate of the Pixel-level of this point), utilize the function of this two dimension to three-dimensional inverse mapping, can obtain the object in the corresponding three-dimensional space, and forward-backward correlation according to the action within window, the practical operation action that parses three dimensional object (for example pulls, move etc.), parse after the corresponding action, pass through message passing mechanism, by distributed virtual reality and visualization system response message, and carry out relevant operational order.
5) distributed virtual reality and visualization system: carry out relevant instruction by this real time execution system, finish every calculation task (comprising scene management, DDM, system resource management, IO management, physical engine, collision detecting system, script engine, computer animation, artificial intelligence, network engine etc.) relevant in the virtual reality system
6) the parallel virtually drawing subsystem of GPU cluster:
The drafting subsystem is the part in virtual reality and the visualization system, among the present invention the GPU cluster is drawn acceleration and handles.
Because the task amount of 3 D image drawing all is the heaviest in virtual reality and the visualization system, often all can become the bottleneck of system's operation, so will drawing this step, we from virtual reality and visualization system, separate, after waiting for that these all virtual realities and Visual calculation task are finished, finish the work of whole drafting by the parallel virtually drawing subsystem of GPU cluster structure.Parallel virtually drawing subsystem adopts the sort-first strategy, promptly earlier with scene according to the difference in institute corresponding display window zone be divided into 4 sub-window areas (size of 4 subwindows be not all with division, dynamically adjust according to the size that shows task amount), give 4 GPU respectively with the drafting task of 4 subwindows and carry out parallel drawing, the drafting task amount born of each GPU can reduce like this, has improved the efficient of whole drafting.Simultaneously according to the difference of 4 performances of GPU hardware own, also can draw task amounts and carry out the drafting task by dividing in the window area how much according to the corresponding GPU of tactful dynamic assignment of self adaptation adjustment.
7) parallel drawing generates the post processing of image module: after treating that 4 GPU generation images finish, need carry out recovery operation in virtual reality and visualization system, 4 width of cloth images that disperse all be fetched and spliced and synthesize a complete image.(known can the realization)
8) Streaming Media compression and sending module: it is that (H.263 compression standard can adopt for a frame of Streaming Media that this module is responsible for image is carried out compressed encoding, H.264 or multiple standards coding form such as MPEG-4), after through compression and interframe compression in the frame, be sent to the mobile device client via the communication connection.
9) Streaming Media receives and decompression module: this module of mobile device end is after receiving frame of video, and the decoding of establishing criteria codec format reverts to a two field picture.
10) Streaming Media drafting module: the two field picture utilization means of drawing are shown in real time, and the user has just seen and duplicate virtual scene of virtual reality system and the visualization result of utilizing the calculating of high-performance calculation machine platform to demonstrate in real time like this.