WO2017113718A1 - 基于虚拟现实的多界面统一显示系统及方法 - Google Patents

基于虚拟现实的多界面统一显示系统及方法 Download PDF

Info

Publication number
WO2017113718A1
WO2017113718A1 PCT/CN2016/089237 CN2016089237W WO2017113718A1 WO 2017113718 A1 WO2017113718 A1 WO 2017113718A1 CN 2016089237 W CN2016089237 W CN 2016089237W WO 2017113718 A1 WO2017113718 A1 WO 2017113718A1
Authority
WO
WIPO (PCT)
Prior art keywords
virtual reality
remote desktop
interface
intelligent electronic
desktop proxy
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2016/089237
Other languages
English (en)
French (fr)
Inventor
聂林
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Le Holdings Beijing Co Ltd
Leshi Zhixin Electronic Technology Tianjin Co Ltd
Original Assignee
Le Holdings Beijing Co Ltd
Leshi Zhixin Electronic Technology Tianjin Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Le Holdings Beijing Co Ltd, Leshi Zhixin Electronic Technology Tianjin Co Ltd filed Critical Le Holdings Beijing Co Ltd
Priority to US15/242,204 priority Critical patent/US20170192734A1/en
Publication of WO2017113718A1 publication Critical patent/WO2017113718A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/012—Head tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/0346—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/038—Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G06F3/04815—Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/1454—Digital output to display device ; Cooperation and interconnection of the display device with other functional units involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/147—Digital output to display device ; Cooperation and interconnection of the display device with other functional units using display panels
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/451—Execution arrangements for user interfaces
    • G06F9/452—Remote windowing, e.g. X-Window System, desktop virtualisation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/80—Geometric correction
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • G09G5/003—Details of a display terminal, the details relating to the control arrangement of the display terminal and to the interfaces thereto
    • G09G5/005—Adapting incoming signals to the display format of the display terminal
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/2866—Architectures; Arrangements
    • H04L67/2876—Pairs of inter-processing entities at each side of the network, e.g. split proxies
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2210/00—Indexing scheme for image generation or computer graphics
    • G06T2210/32—Image data format
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2360/00—Aspects of the architecture of display systems
    • G09G2360/12—Frame memory handling
    • G09G2360/127—Updating a frame memory using a transfer of data from a source area to a destination area
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2360/00—Aspects of the architecture of display systems
    • G09G2360/18—Use of a frame buffer in a display terminal, inclusive of the display panel
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2370/00—Aspects of data communication
    • G09G2370/02—Networking aspects
    • G09G2370/022—Centralised management of display operation, e.g. in a server instead of locally
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2370/00—Aspects of data communication
    • G09G2370/20—Details of the management of multiple sources of image data
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services

Definitions

  • the present invention relates to the field of virtual reality technology, and in particular to a multi-interface unified display system and method based on virtual reality.
  • the technical problem to be solved by the present invention is to provide a multi-interface unified display system based on virtual reality, so as to uniformly display a user interface of a plurality of intelligent electronic devices.
  • Another technical problem to be solved by the present invention is to provide a multi-interface unified display method based on virtual reality, so as to uniformly display a user interface of a plurality of intelligent electronic devices.
  • a virtual reality based multi-interface unified display system comprising:
  • a plurality of remote desktop proxy servers are respectively built in the corresponding plurality of intelligent electronic devices to obtain a current screen of the corresponding intelligent electronic device and externally transmitted;
  • the virtual reality integrated machine further includes:
  • the virtual reality 3D engine is configured to convert the current screen image of the intelligent electronic device transmitted by the different remote desktop proxy client into a map that can be recognized by the graphic program interface, and then bind the texture to the surface of the corresponding window in the virtual scene, and also Rendering the corresponding screens of the left and right eyes to a pair of defined buffers through the graphics program interface, and performing inverse distortion processing on the contents of the buffer;
  • a service module is displayed to display the processed pictures in the buffer.
  • the intelligent electronic device is at least one of a personal computer and a smart phone.
  • the virtual reality integrated machine is a virtual reality helmet.
  • the present invention also provides a multi-interface unified display method based on virtual reality, comprising the following steps:
  • Step S1 the remote desktop proxy server intercepts the current screen image and transmits the same to the remote desktop proxy client of the VR all-in-one through the network;
  • Step S2 the remote desktop proxy client of the VR integrated machine receives the current screen of the intelligent electronic device and transmits it to the VR 3D engine;
  • Step S3 The 3D engine converts the current screen image of the intelligent electronic device transmitted by the different proxy client into a map format recognizable by the graphic program interface;
  • Step S4 the 3D engine binds the texture to the surface of the corresponding window in the virtual scene, and respectively renders the corresponding image of the left and right eyes to a pair of defined buffers through the graphic program interface;
  • Step S5 in order to cooperate with the image distortion caused by the optical lens of the helmet, the 3D engine performs anti-distortion processing on the content in the buffer;
  • step S6 the processed screen in the buffer is submitted to the display service module for display.
  • graphics program interface is OpenGL.
  • the method further includes the following steps:
  • step S7 displacement control of the mouse pointer in the displayed screen is realized by virtual reality integrated machine simulation.
  • step S7 specifically includes:
  • Step S71 the virtual reality integrated machine gyroscope obtains the rotational angular velocity of the user's head extending the x, y, and z axes;
  • Step S72 calculating a corresponding rotation angle according to the current rotational angular velocity and the time interval when the current distance was last sampled;
  • Step S73 fixing the mouse pointer at the center of the screen coordinates, and the 3D engine reversely rotates the current scene to the upper angle and recalculates the mouse pointer coordinates;
  • step S74 the new mouse pointer coordinates are transmitted to the server through the remote desktop proxy client.
  • step S72 a data fusion algorithm is used to calculate and obtain a corresponding rotation angle.
  • step S6 the processed screen in the buffer is submitted to the display service module through the application programming interface of the EGL.
  • the present invention utilizes a virtual 360 ⁇ 180 degree omnidirectional vision function of the VR integrated machine, and through multiple remote desktop proxy clients and a remote desktop agent built in the corresponding intelligent electronic device.
  • the servers are connected one by one to obtain the current screen of the corresponding intelligent electronic device, and after being processed by the 3D engine 32, the virtual reality scene is presented in a centralized manner, thereby facilitating the user.
  • Viewing the user interfaces of multiple intelligent electronic devices simultaneously in a single interface such as a virtual reality scene is more conducive to centrally and efficiently viewing and managing these user interfaces.
  • the simple operation of the intelligent electronic device such as mouse pointer displacement control, is realized in combination with the control function of the virtual reality integrated machine.
  • the embodiment of the present invention further provides a computer storage medium, wherein the computer storage medium can store a program, and when the program is executed, the virtual reality-based multi-interface unified display executed by the remote desktop proxy server can be implemented.
  • the embodiment of the present invention further provides a computer storage medium, wherein the computer storage medium can store a program, and when the program is executed, the virtual reality-based multi-interface unified display executed by the remote desktop proxy client can be implemented.
  • FIG. 1 is a block diagram showing the system configuration of a multi-interface unified display system based on virtual reality according to the present invention.
  • FIG. 2 is a schematic flow chart of a multi-interface unified display method based on virtual reality according to the present invention.
  • FIG. 3 is a schematic flow chart of implementing mouse index control according to the virtual reality-based multi-interface unified display method of the present invention.
  • the present invention provides a virtual reality based multi-interface unified display system, which includes:
  • a plurality of remote desktop proxy server terminals 1 respectively built in a corresponding plurality of intelligent electronic devices to obtain a current screen of the corresponding intelligent electronic device and externally transmitted;
  • the virtual reality integrated machine (VR integrated machine) 3, the virtual reality integrated machine 3 further includes:
  • a plurality of remote desktop proxy clients 30 are connected in one-to-one correspondence with the remote desktop proxy server 1 to obtain a current screen of the corresponding smart electronic device;
  • the virtual reality 3D engine 32 is configured to convert the images transmitted by the different remote desktop proxy clients 30 into textures recognizable by the image rendering program, and then bind the textures to the surface of the corresponding window in the virtual scene, and also Rendering the corresponding images of the left and right eyes to a pair of defined buffers through the application programming interface of the image rendering program, and performing anti-distortion processing on the contents of the buffer;
  • the display service module 34 is configured to display the processed screen in the buffer.
  • the smart electronic device usable in the present invention may be at least one of a personal computer (Personal Computer, abbreviated as PC) and a smart phone.
  • a personal computer PC
  • a smart phone In the embodiment shown in FIG. 1, a personal computer (PC) is simultaneously used. ) 20 and smartphone 22.
  • PC Personal Computer
  • smartphone 22 smartphone 22.
  • the number of intelligent electronic devices that can establish a connection with the virtual reality integrated device 3 can be multiple, for example, three, four, or even more, and is not limited to two shown in FIG.
  • the virtual reality integrated machine is preferably a virtual reality helmet.
  • the present invention further provides a multi-interface unified display method based on virtual reality, which includes the following process steps:
  • Step S1 the remote desktop proxy server intercepts the current screen image and transmits the screen to the remote desktop proxy client of the VR all-in-one through the network;
  • Step S2 the remote desktop proxy client of the VR machine receives the image and transmits the image to the VR 3D engine;
  • Step S3 the 3D engine converts the images sent by the different proxy clients into a map format that the image rendering program can recognize;
  • step S4 the 3D engine binds the texture to the surface of the corresponding window in the virtual scene. And the image corresponding to the left and right eyes is respectively rendered into a pair of defined buffers through an API (Application Programming Interface) of the image rendering program.
  • an API Application Programming Interface
  • OpenGL is preferably used as the Picture rendering program
  • Step S5 in order to cooperate with the image distortion caused by the optical lens of the helmet, the 3D engine performs anti-distortion processing on the content in the buffer;
  • step S6 the processed picture in the buffer is submitted to the display service module through the EGL API for display.
  • the present invention may further include the following steps: uniformly displaying the multiple interfaces based on the multiple interfaces of the virtual reality above.
  • step S7 displacement control of the mouse pointer in the displayed screen is realized by virtual reality integrated machine simulation.
  • step S7 specifically includes the following steps:
  • Step S71 the virtual reality integrated machine gyroscope obtains the rotational angular velocity of the user's head extending the x, y, and z axes;
  • Step S72 multiplying the current speed by the time interval of the current distance from the last sampling to obtain a corresponding rotation angle
  • Step S73 fixing the mouse pointer at the center of the screen coordinates, and the 3D engine reversely rotates the current scene to the upper angle and recalculates the mouse pointer coordinates;
  • step S74 the new mouse pointer coordinates are transmitted to the server through the remote desktop proxy client.
  • a data fusion algorithm may also be adopted to obtain a corresponding rotation angle.
  • the invention utilizes the virtual 360 ⁇ 180 degree omnidirectional vision function of the VR integrated machine, and connects the remote desktop proxy server 30 in the corresponding intelligent electronic device through the plurality of remote desktop proxy clients 30 to obtain the corresponding intelligent electronic
  • the current screen of the device is processed in the virtual reality scene after being processed by the 3D engine 32, so that the user can view the user interfaces of multiple intelligent electronic devices simultaneously in a single interface such as a virtual reality scene, and can also be combined with the virtual interface.
  • the control function of the real machine realizes simple operation of the intelligent electronic device, for example: mouse pointer displacement control.
  • the embodiment of the present invention further provides a computer storage medium, wherein the computer storage medium can store a program, and when the program is executed, the virtual desktop based on the remote desktop proxy server can be implemented.
  • the interface unifies some or all of the steps in each implementation of the method.
  • the embodiment of the present invention further provides a computer storage medium, wherein the computer storage medium can store a program, and when the program is executed, the virtual desktop based client can be implemented by the remote desktop proxy client.
  • the interface unifies some or all of the steps in each implementation of the method.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Software Systems (AREA)
  • Computer Hardware Design (AREA)
  • Signal Processing (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Computer Graphics (AREA)
  • User Interface Of Digital Computer (AREA)
  • Information Transfer Between Computers (AREA)
  • Digital Computer Display Output (AREA)
  • Geometry (AREA)

Abstract

一种基于虚拟现实的多界面统一显示系统及方法，所述系统包括：多个远程桌面代理服务器端(1)，分别内置于多个智能电子装置中以获得智能电子装置当前屏幕画面；以及虚拟现实一体机(3)，又包括：多个远程桌面代理客户端(30)，与远程桌面代理服务器端(1)对应连接以获取智能电子装置当前屏幕画面；虚拟现实3D引擎(32)，把智能电子装置当前屏幕画面转化为图形程序接口可识别的贴图，再将贴图绑定到虚拟场景当中对应窗口的表面，还分别渲染左右眼对应的画面到一对缓冲区中，并对缓冲区中的内容进行反畸变处理；以及显示服务模块(34)，用以显示缓冲区中经处理过的画面。该系统及方法在虚拟现实场景中集中显现多个智能电子装置用户界面，更有利于用户查看和管理。

Description

基于虚拟现实的多界面统一显示系统及方法
本申请要求于2015年12月31日提交中国专利局、申请号为201511034715X、发明名称为“基于虚拟现实的多界面统一显示系统及方法”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本发明涉及虚拟现实技术领域，尤其是指基于虚拟现实的多界面统一显示系统及方法。
背景技术
在日常工作和生活中，人们时常会用到诸如智能手机、电脑等各种具有用户界面(UI User Interface)的智能电子装置。在使用时，人们只能分别查看这些智能产品的用户界面，相当繁琐。发明人在实现本发明的过程中发现：如能在一个界面中同时查看多个智能电子装置的用户界面，将会极大地方便用户使用这些智能电子装置。
发明内容
本发明所要解决的技术问题在于，提供一种基于虚拟现实的多界面统一显示系统，以便于统一显示多个智能电子装置的用户界面。
本发明另一个所要解决的技术问题在于，提供一种基于虚拟现实的多界面统一显示方法，以便于统一显示多个智能电子装置的用户界面。
为解决上述技术问题，本发明提供如下技术方案：一种基于虚拟现实的多界面统一显示系统，包括：
多个远程桌面代理服务器端，分别内置于对应的多个智能电子装置中以获得对应的智能电子装置当前屏幕画面并对外传送；以及
虚拟现实一体机，所述虚拟现实一体机又包括：
多个远程桌面代理客户端，与所述远程桌面代理服务器端一一对应连接以获取对应的智能电子装置当前屏幕画面；
虚拟现实3D引擎，用于把不同的远程桌面代理客户端传来的智能电子装置当前屏幕画面转化为图形程序接口可以识别的贴图，再将贴图绑定到虚拟场景当中对应窗口的表面，并且还通过图形程序接口分别渲染左右眼对应的画面到一对制定的缓冲区中，并且对缓冲区中的内容进行反畸变处理；以及
显示服务模块，用以显示缓冲区中经处理过的画面。
进一步地，所述智能电子装置是个人电脑和智能手机两者中的至少一种。
进一步地，所述虚拟现实一体机是虚拟现实头盔。
另一方面，本发明还提供一种基于虚拟现实的多界面统一显示方法，包括如下步骤：
步骤S1，远程桌面代理服务器端截取当前屏幕画面并通过网络传输给VR一体机的远程桌面代理客户端；
步骤S2，VR一体机的远程桌面代理客户端收到智能电子装置当前屏幕画面并传递给VR 3D引擎；
步骤S3，3D引擎把不同代理客户端传来的智能电子装置当前屏幕画面转化为图形程序接口可识别的贴图格式；
步骤S4，3D引擎把贴图绑定到虚拟场景当中的对应窗口的表面上，并通过图形程序接口分别渲染左右眼对应的画面到一对制定的缓冲区中；
步骤S5，为配合头盔的光学透镜造成的图像畸变，3D引擎对缓冲区中的内容进行反畸变处理；
步骤S6，把缓冲区中的处理过的画面提交给显示服务模块进行显示。
进一步地，所述图形程序接口为OpenGL。
进一步地，所述方法还包括以下步骤：
步骤S7，通过虚拟现实一体机模拟实现对显示的画面中的鼠标指针进行位移控制。
进一步地，所述步骤S7具体又包括：
步骤S71，虚拟现实一体机陀螺仪获取用户头部延x,y,z轴的转动角速度；
步骤S72，根据当前转动角速度和当前距离上次取样时的时间间隔计算获得对应的旋转角度；
步骤S73，将鼠标指针固定在屏幕坐标的中心，由3D引擎将当前场景反向转动上面的角度并重新计算鼠标指针坐标；
步骤S74，将新的鼠标指针坐标通过远程桌面代理客户端传递给服务器端。
进一步地，步骤S72中，采用数据融合算法计算获得对应的旋转角度。
进一步地，步骤S6中，通过EGL的应用程序编程接口把缓冲区中的处理过的画面提交给显示服务模块。
采用上述技术方案后，本发明至少具有如下有益效果：本发明利用VR一体机虚拟360×180度全方位视野功能，通过多个远程桌面代理客户端与对应的智能电子装置中内置的远程桌面代理服务器端一一对应连接以获取对应的智能电子装置当前屏幕画面，经过3D引擎32的处理后，在虚拟现实场景中予以集中呈现，从而方便用户 在虚拟现实场景这样单一的界面中同时查看多个智能电子装置的用户界面，更有利于集中高效地查看和管理这些用户界面。此外，结合虚拟现实一体机的控制功能实现对智能电子装置的简单操作，例如：鼠标指针位移控制。
本发明实施例还提供一种计算机存储介质，其中，该计算机存储介质可存储有程序，该程序执行时可实现前述实施例提供的,远程桌面代理服务器端执行的基于虚拟现实的多界面统一显示方法的各实现方式中的部分或全部步骤。
本发明实施例还提供一种计算机存储介质，其中，该计算机存储介质可存储有程序，该程序执行时可实现前述实施例提供的,远程桌面代理客户端执行的基于虚拟现实的多界面统一显示方法的各实现方式中的部分或全部步骤。
应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，并不能限制本发明。
附图说明
图1是本发明基于虚拟现实的多界面统一显示系统的系统构成方框图。
图2是本发明基于虚拟现实的多界面统一显示方法的流程示意图。
图3是本发明基于虚拟现实的多界面统一显示方法的实现鼠标指标控制的流程示意图。
具体实施方式
需要说明的是，在不冲突的情况下，本申请中的实施例及实施例中的特征可以相互结合，下面结合附图和具体实施例对本发明作进一步详细说明。
如图1所示，本发明提供一种基于虚拟现实的多界面统一显示系统，其包括：
多个远程桌面代理服务器端1，分别内置于对应的多个智能电子装置中以获得对应的智能电子装置当前屏幕画面并对外传送；以及
虚拟现实一体机(VR一体机)3，所述虚拟现实一体机3又包括：
多个远程桌面代理客户端30，与所述远程桌面代理服务器端1一一对应连接以获取对应的智能电子装置当前屏幕画面；
虚拟现实3D引擎32，用于把不同的远程桌面代理客户端30传来的图片转化为图片渲染程序可识别的贴图，再将贴图绑定到虚拟场景当中对应窗口的表面，并且还 通过图片渲染程序的应用程序编程接口分别渲染左右眼对应的画面到一对制定的缓冲区中，并且对缓冲区中的内容进行反畸变处理；以及
显示服务模块34，用以显示缓冲区中经处理过的画面。
可用于本发明的智能电子装置可以是个人电脑(Personal Computer，缩写为PC)和智能手机两者中的至少一种，在如图1所示的实施例中，即同时采用了个人电脑(PC)20和智能手机22。可以理解的是，可以与虚拟现实一体机3建立连接的智能电子装置可以为多台，例如：3台、4台，甚至更多，而不局限于图1中所示的2台。
所述虚拟现实一体机优选为虚拟现实头盔。
结合图2所示，本发明还提供一种基于虚拟现实的多界面统一显示方法，包括如下流程步骤：
步骤S1，远程桌面代理服务器端截取当前屏幕画面并通过网络传输画面给VR一体机的远程桌面代理客户端；
步骤S2，VR一体机的远程桌面代理客户端收到图片并传递给VR 3D引擎；
步骤S3，3D引擎把不同代理客户端传来的图片转化为图片渲染程序可以识别的贴图格式；
步骤S4，3D引擎把贴图绑定到虚拟场景当中的对应窗口的表面上。并通过图片渲染程序的API(Application Programming Interface，应用程序编程接口)分别渲染左右眼对应的画面到一对制定的缓冲区中，在如图2所示的实施例中，优选地采用了OpenGL作为图片渲染程序；
步骤S5，为配合头盔的光学透镜造成的图像畸变，3D引擎对缓冲区中的内容进行反畸变处理；
步骤S6，把缓冲区中的处理过的画面通过EGL的API提交给显示服务模块进行显示。
基于以上虚拟现实的多界面统一显示多界面的，本发明还可以进一步包括以下步骤：
步骤S7，通过虚拟现实一体机模拟实现对显示的画面中的鼠标指针进行位移控制。
结合图3所示，所述步骤S7具体又包括如下步骤：
步骤S71，虚拟现实一体机陀螺仪获取用户头部延x,y,z轴的转动角速度；
步骤S72，将当前速度乘以当前距离上次取样时的时间间隔得到对应的旋转角度；
步骤S73，将鼠标指针固定在屏幕坐标的中心，由3D引擎将当前场景反向转动上面的角度并重新计算鼠标指针坐标；
步骤S74，新的鼠标指针坐标通过远程桌面代理客户端传递给服务器端。
在具体实施步骤S72时，还可以采用数据融合算法来获得对应的旋转角度。
本发明利用VR一体机虚拟360×180度全方位视野功能，通过多个远程桌面代理客户端30与对应的智能电子装置中内置的远程桌面代理服务器端1一一对应连接以获取对应的智能电子装置当前屏幕画面，经过3D引擎32的处理后，在虚拟现实场景中予以集中呈现，从而方便用户在虚拟现实场景这样单一的界面中同时查看多个智能电子装置的用户界面，而且还可以结合虚拟现实一体机的控制功能实现对智能电子装置的简单操作，例如：鼠标指针位移控制。
本发明实施例还提供一种计算机存储介质，其中，该计算机存储介质可存储有程序，该程序执行时可实现图2所示实施例提供的,远程桌面代理服务器端执行的基于虚拟现实的多界面统一显示方法的各实现方式中的部分或全部步骤。
本发明实施例还提供一种计算机存储介质，其中，该计算机存储介质可存储有程序，该程序执行时可实现图2所示实施例提供的,远程桌面代理客户端执行的基于虚拟现实的多界面统一显示方法的各实现方式中的部分或全部步骤。
尽管已经示出和描述了本发明的实施例，对于本领域的普通技术人员而言，可以理解在不脱离本发明的原理和精神的情况下可以对这些实施例进行多种变化、修改、替换和变型，本发明的范围由所附权利要求及其等同范围限定。

Claims (9)

  1. 一种基于虚拟现实的多界面统一显示系统，其特征在于，包括：
    多个远程桌面代理服务器端，分别内置于对应的多个智能电子装置中以获得对应的智能电子装置当前屏幕画面并对外传送；以及虚拟现实一体机，所述虚拟现实一体机又包括：
    多个远程桌面代理客户端，与所述远程桌面代理服务器端一一对应连接以获取对应的智能电子装置当前屏幕画面；
    虚拟现实3D引擎，用于把不同的远程桌面代理客户端传来的智能电子装置当前屏幕画面转化为图形程序接口可以识别的贴图，再将贴图绑定到虚拟场景当中对应窗口的表面，并且还通过图形程序接口分别渲染左右眼对应的画面到一对制定的缓冲区中，并且对缓冲区中的内容进行反畸变处理；以及显示服务模块，用以显示缓冲区中经处理过的画面。
  2. 如权利要求1所述的基于虚拟现实的多界面统一显示系统，其特征在于，所述智能电子装置是个人电脑和智能手机两者中的至少一种。
  3. 如权利要求1所述的基于虚拟现实的多界面统一显示系统，其特征在于，所述虚拟现实一体机是虚拟现实头盔。
  4. 一种基于虚拟现实的多界面统一显示方法，其特征在于，包括如下步骤：
    步骤S1，远程桌面代理服务器端截取当前屏幕画面并通过网络传输给VR一体机的远程桌面代理客户端；
    步骤S2，VR一体机的远程桌面代理客户端收到智能电子装置当前屏幕画面并传递给VR 3D引擎；
    步骤S3，3D引擎把不同代理客户端传来的智能电子装置当前屏幕画面转化为图形程序接口可识别的贴图格式；
    步骤S4，3D引擎把贴图绑定到虚拟场景当中的对应窗口的表面上，并通过图形程序接口分别渲染左右眼对应的画面到一对制定的缓冲区中；
    步骤S5，为配合头盔的光学透镜造成的图像畸变，3D引擎对缓冲区中的内容进行反畸变处理；
    步骤S6，把缓冲区中的处理过的画面提交给显示服务模块进行显示。
  5. 如权利要求4所述的基于虚拟现实的多界面统一显示方法，其特征在于，所述图形程序接口为OpenGL。
  6. 如权利要求4所述的基于虚拟现实的多界面统一显示方法，其特征在于，所述方法还包括以下步骤：
    步骤S7，通过虚拟现实一体机模拟实现对显示的画面中的鼠标指针进行 位移控制。
  7. 如权利要求6所述的基于虚拟现实的多界面统一显示方法，其特征在于，所述步骤S7具体又包括：
    步骤S71，虚拟现实一体机陀螺仪获取用户头部延x,y,z轴的转动角速度；
    步骤S72，根据当前转动角速度和当前距离上次取样时的时间间隔计算获得对应的旋转角度；
    步骤S73，将鼠标指针固定在屏幕坐标的中心，由3D引擎将当前场景反向转动上面的角度并重新计算鼠标指针坐标；
    步骤S74，将新的鼠标指针坐标通过远程桌面代理客户端传递给服务器端。
  8. 如权利要求7所述的基于虚拟现实的多界面统一显示方法，其特征在于，步骤S72中，采用数据融合算法计算获得对应的旋转角度。
  9. 如权利要求4所述的基于虚拟现实的多界面统一显示方法，其特征在于，步骤S6中，通过EGL的应用程序编程接口把缓冲区中的处理过的画面提交给显示服务模块。
PCT/CN2016/089237 2015-12-31 2016-07-07 基于虚拟现实的多界面统一显示系统及方法 Ceased WO2017113718A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US15/242,204 US20170192734A1 (en) 2015-12-31 2016-08-19 Multi-interface unified displaying system and method based on virtual reality

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201511034715.X 2015-12-31
CN201511034715.XA CN105892643A (zh) 2015-12-31 2015-12-31 基于虚拟现实的多界面统一显示系统及方法

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US15/242,204 Continuation US20170192734A1 (en) 2015-12-31 2016-08-19 Multi-interface unified displaying system and method based on virtual reality

Publications (1)

Publication Number Publication Date
WO2017113718A1 true WO2017113718A1 (zh) 2017-07-06

Family

ID=57002287

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/089237 Ceased WO2017113718A1 (zh) 2015-12-31 2016-07-07 基于虚拟现实的多界面统一显示系统及方法

Country Status (3)

Country Link
US (1) US20170192734A1 (zh)
CN (1) CN105892643A (zh)
WO (1) WO2017113718A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TWI775397B (zh) * 2021-04-21 2022-08-21 宏碁股份有限公司 3d顯示系統與3d顯示方法
CN115225883A (zh) * 2021-04-21 2022-10-21 宏碁股份有限公司 3d显示系统与3d显示方法

Families Citing this family (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10652303B2 (en) * 2016-04-28 2020-05-12 Rabbit Asset Purchase Corp. Screencast orchestration
CN106354256B (zh) * 2016-08-28 2019-05-17 杭州勺子网络科技有限公司 一种虚拟现实的移动控制方法
CN106502641A (zh) * 2016-09-18 2017-03-15 北京小鸟看看科技有限公司 Vr设备的开机画面的显示方法和头戴式3d显示器
CN108111874B (zh) * 2016-11-16 2020-01-31 腾讯科技（深圳）有限公司 一种文件处理方法、终端及服务器
CN106851240A (zh) * 2016-12-26 2017-06-13 网易（杭州）网络有限公司 图像数据处理的方法及装置
CN107220925B (zh) * 2017-05-05 2018-10-30 珠海全志科技股份有限公司 一种实时虚拟现实加速方法及装置
CN107358659B (zh) * 2017-07-21 2021-06-22 福建星网视易信息系统有限公司 基于3d技术的多画面融合显示方法及存储设备
CN108090946A (zh) * 2017-12-14 2018-05-29 苏州蜗牛数字科技股份有限公司 一种材质球的构建方法及装置
CN110321187B (zh) * 2018-03-30 2022-08-30 合肥杰发科技有限公司 基于代理模式的多媒体显示方法、装置及设备
CN111176451B (zh) * 2019-12-30 2023-06-02 上海曼恒数字技术股份有限公司 一种虚拟现实多通道沉浸式环境的控制方法及系统
EP4369075A4 (en) 2021-09-29 2024-11-20 Samsung Electronics Co., Ltd. ELECTRONIC DEVICE FOR PROVIDING AUGMENTED REALITY SERVICE, AND METHOD OF OPERATING THE SAME
CN114706936B (zh) * 2022-05-13 2022-08-26 高德软件有限公司 地图数据处理方法及基于位置的服务提供方法
CN118037923A (zh) * 2024-04-09 2024-05-14 北京趋动智能科技有限公司 图像渲染方法、装置、存储介质及电子设备
CN121166241A (zh) * 2024-06-17 2025-12-19 云智能资产控股（新加坡）私人股份有限公司 一种云桌面显示方法、设备、系统、计算机程序产品及存储介质

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102129361A (zh) * 2010-01-13 2011-07-20 宏正自动科技股份有限公司 多分割画面的集中显示系统与方法
US20110246950A1 (en) * 2010-03-30 2011-10-06 Michael Luna 3d mobile user interface with configurable workspace management
US8108791B2 (en) * 2009-02-27 2012-01-31 Microsoft Corporation Multi-screen user interface
CN102426829A (zh) * 2011-09-30 2012-04-25 冠捷显示科技（厦门）有限公司 双画面显示装置及实现方法
CN103577163A (zh) * 2012-07-19 2014-02-12 中兴通讯股份有限公司 移动终端多用户界面的实现方法及装置
CN104035760A (zh) * 2014-03-04 2014-09-10 苏州天魂网络科技有限公司 跨移动平台实现沉浸式虚拟现实的系统
CN104915979A (zh) * 2014-03-10 2015-09-16 苏州天魂网络科技有限公司 跨移动平台实现沉浸式虚拟现实的系统

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160098095A1 (en) * 2004-01-30 2016-04-07 Electronic Scripting Products, Inc. Deriving Input from Six Degrees of Freedom Interfaces
CN104216533B (zh) * 2014-08-28 2017-06-06 东华大学 一种基于DirectX9的头戴式虚拟现实显示器
US9898864B2 (en) * 2015-05-28 2018-02-20 Microsoft Technology Licensing, Llc Shared tactile interaction and user safety in shared space multi-person immersive virtual reality

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8108791B2 (en) * 2009-02-27 2012-01-31 Microsoft Corporation Multi-screen user interface
CN102129361A (zh) * 2010-01-13 2011-07-20 宏正自动科技股份有限公司 多分割画面的集中显示系统与方法
US20110246950A1 (en) * 2010-03-30 2011-10-06 Michael Luna 3d mobile user interface with configurable workspace management
CN102426829A (zh) * 2011-09-30 2012-04-25 冠捷显示科技（厦门）有限公司 双画面显示装置及实现方法
CN103577163A (zh) * 2012-07-19 2014-02-12 中兴通讯股份有限公司 移动终端多用户界面的实现方法及装置
CN104035760A (zh) * 2014-03-04 2014-09-10 苏州天魂网络科技有限公司 跨移动平台实现沉浸式虚拟现实的系统
CN104915979A (zh) * 2014-03-10 2015-09-16 苏州天魂网络科技有限公司 跨移动平台实现沉浸式虚拟现实的系统

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TWI775397B (zh) * 2021-04-21 2022-08-21 宏碁股份有限公司 3d顯示系統與3d顯示方法
CN115225883A (zh) * 2021-04-21 2022-10-21 宏碁股份有限公司 3d显示系统与3d显示方法
US11962743B2 (en) 2021-04-21 2024-04-16 Acer Incorporated 3D display system and 3D display method

Also Published As

Publication number Publication date
CN105892643A (zh) 2016-08-24
US20170192734A1 (en) 2017-07-06

Similar Documents

Publication Publication Date Title
WO2017113718A1 (zh) 基于虚拟现实的多界面统一显示系统及方法
US10726625B2 (en) Method and system for improving the transmission and processing of data regarding a multi-user virtual environment
US20140320592A1 (en) Virtual Video Camera
CN107274469A (zh) 面向虚拟现实的协同渲染方法
CN109743626B (zh) 一种图像显示方法、图像处理方法和相关设备
CN107248194A (zh) 一种基于云计算的cae数据三维显示交互方法
JP5295416B1 (ja) 画像処理装置、画像処理方法及び画像処理プログラム
CN111984114A (zh) 基于虚拟空间的多人交互系统及其多人交互方法
CN105825499A (zh) 基准平面的确定方法和确定系统
CN107111427A (zh) 修改视频通话数据
WO2018188479A1 (zh) 基于增强现实的导航方法及装置
CN116057577A (zh) 用于增强现实的地图
CN106445129A (zh) 全景图像信息显示方法、装置及系统
WO2023207379A1 (zh) 图像处理方法、装置、设备及存储介质
CN108022305A (zh) 一种基于ar技术的看房体验系统
EP4485357A2 (en) Image processing method and apparatus, electronic device, and storage medium
CN110662015A (zh) 用于显示图像的方法及装置
CN107943301A (zh) 一种基于ar技术的购房观赏体验系统
CN106454012A (zh) 视频数据显示方法、装置及系统
CN115826740A (zh) 设备交互的显示方法、系统、电子设备、介质、产品
WO2020253342A1 (zh) 3d视频的全景渲染方法、计算机设备和可读存储介质
CN118827950A (zh) 控制方法、装置、头戴显示设备及介质
CN115761190A (zh) 基于场景映射的多用户增强现实相片浏览方法及系统
WO2023030091A1 (zh) 控制运动物体运动的方法、装置、设备及介质
CN119537630B (zh) 一种信息展示方法、装置、设备、存储介质及程序产品

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16880518

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16880518

Country of ref document: EP

Kind code of ref document: A1