WO2024012459A1 - 一种vr端端云结合的虚拟演唱会渲染方法及系统 - Google Patents

一种vr端端云结合的虚拟演唱会渲染方法及系统 Download PDF

Info

Publication number
WO2024012459A1
WO2024012459A1 PCT/CN2023/106828 CN2023106828W WO2024012459A1 WO 2024012459 A1 WO2024012459 A1 WO 2024012459A1 CN 2023106828 W CN2023106828 W CN 2023106828W WO 2024012459 A1 WO2024012459 A1 WO 2024012459A1
Authority
WO
WIPO (PCT)
Prior art keywords
virtual
cloud
concert
data
local
Prior art date
Application number
PCT/CN2023/106828
Other languages
English (en)
French (fr)
Inventor
匡昊
费元华
郭建君
Original Assignee
北京蔚领时代科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京蔚领时代科技有限公司 filed Critical 北京蔚领时代科技有限公司
Publication of WO2024012459A1 publication Critical patent/WO2024012459A1/zh

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/111—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
    • H04N13/117—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/156—Mixing image signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/194—Transmission of image signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/296—Synchronisation thereof; Control thereof

Definitions

  • This application relates to the field of VR technology, and in particular to a virtual concert rendering method and system that combines VR end-end and cloud.
  • VR virtual reality technology
  • the cloud rendering (cloudrender) model is similar to conventional cloud computing. That is, the 3D program is placed on a remote server for rendering. The user terminal clicks a "cloud rendering" button through Web software or directly in the local 3D program and uses a high-speed Internet connection. To access resources, instructions are issued from the user terminal, the server performs corresponding rendering tasks according to the instructions, and the rendering result screen is sent back to the user terminal for display.
  • This application aims to technically solve the above problems and provide concert performers and viewers with a solution for virtual concert rendering with high efficiency, low latency, high response speed and low threshold. solution.
  • the purpose of this application is to provide a virtual concert rendering method and system that combines VR end-end and cloud, which has high rendering efficiency, low latency, high response speed and low threshold.
  • this application provides a virtual concert rendering method that combines VR end-end and cloud. It is characterized in that the method includes:
  • the driven virtual idol images and virtual concert scene images are collected in real time
  • the local audience picture is obtained by rendering the interaction between the audience in the auditorium and the audience on the local computing device.
  • the performance data includes the motion expression data and sound data of the motion capture actors; the motion capture actors' own movements and dances, the expressions and stage styles of the motion capture actors; the sound data includes the songs and background music sung by the motion capture actors.
  • the virtual idol is driven based on the motion expression data of the motion capture actor, and the virtual camera is adjusted to an angle consistent with the visual angle of the local VR device according to the visual angle of the local VR device, and the adjusted virtual camera captures the object in real time.
  • the driven virtual idol screen and virtual concert scene screen are examples of the virtual cameras.
  • the local computing device After the local computing device obtains the audience interaction data, it sends it to the clients of other viewers.
  • the clients of other viewers render the audience interaction data on their local computing devices, and all the audience clients render the audience interaction effect. .
  • the method of creating a virtual idol and a virtual concert scene in the cloud and integrating the virtual idol into the virtual concert scene includes:
  • motion capture actor's motion expression data is collected through motion capture equipment; wherein the motion expression data includes motion data and expression data;
  • motion capture equipment includes inertial motion capture sensors and optical motion capture sensors.
  • the action data is transmitted to the model engine in real time through the virtual engine plug-in of the model engine, the action data is associated with the body movements of the virtual idol, and the virtual idol is driven to execute the action data through the bone binding data and skin data;
  • the virtual engine plug-in of the model engine is used to associate the expression data of the motion capture actor with the facial expressions of the virtual idol, and the expression control data is used to drive the virtual idol to execute the expression data.
  • This application also provides a virtual concert rendering system that combines VR end-end and cloud.
  • the system includes:
  • Data collection equipment used to collect visual angle data and performance data of local VR equipment in real time
  • the driving module uses performance data to drive virtual idols
  • the picture collection device is used to collect the driven virtual idol pictures and virtual concert scene pictures in real time according to the visual angle of the local VR device;
  • the rendering module is used to render the collected virtual idol images and virtual concert scene images in real time, and insert sound data to obtain cloud 360VR video streams;
  • the video synthesis module is used to synthesize the cloud 360VR video stream and local audience images into a virtual concert video stream;
  • the data transmission module is used to send the synthesized virtual concert video stream to the local viewing client.
  • the image collection device is a cloud virtual camera set up in the cloud. After the cloud virtual camera adjusts its visual angle to be consistent with the visual angle of the collected local VR device, it collects the virtual idol image and the virtual concert scene image.
  • the stage performance content output from the cloud rendering server in this application is a 360VR video stream.
  • This processing method is different from the traditional two-dimensional video stream and provides a more three-dimensional and realistic picture effect.
  • the audience uses the local VR device to rotate the viewing angle at will, the visual angle of the 360VR video stream and the local VR device will remain consistent, and there will be no picture distortion or distortion. The situation of wearing gangs.
  • This application chooses to render the concert content in the cloud and render the auditorium images locally. Since the content of the concert is complex and the graphics requirements are high, it requires high-performance rendering computing power as support. Therefore, the computing power advantage of the cloud server improves the rendering efficiency and rendering effect of the concert content. Since the interaction in the auditorium requires real-time, low latency and high response speed, but the rendering computing power is not high, it is chosen to render locally, taking advantage of the delay advantage of local computing and improving the response speed of audience interaction.
  • This application enables performance activities such as concerts to be held online; motion capture actors/artists do not need to gather offline and can jointly access the concert through cloud collaboration; because the parts that require the most rendering computing power are transferred from the local Moving to the cloud reduces the device threshold for users to watch cloud concerts; due to sufficient computing power in the cloud, the concert director does not need to consider user-side performance issues in terms of effect, and has lower requirements for user-side performance and a lower threshold, thus better Give full play to artistic imagination; audiences can participate in concerts and other interpretation activities at any time and anywhere as long as they have smooth access to the Internet.
  • Figure 1 is a flow chart of a virtual concert rendering method that combines VR end-device and cloud according to an embodiment of the present application.
  • Figure 2 is a flow chart of a method for creating virtual idols and virtual concert scenes in the cloud according to an embodiment of the present application.
  • Figure 3 is a flow chart for real-time collection of visual angle data and performance data of a local VR device according to an embodiment of the present application.
  • Figure 4 is a flow chart of real-time collection of driven virtual idol images and virtual concert scene images according to an embodiment of the present application.
  • Figure 5 is a schematic structural diagram of a virtual concert rendering system integrating VR end-end and cloud according to an embodiment of the present application.
  • this application provides a virtual concert rendering method that combines VR end-end and cloud.
  • the method includes:
  • Step S1 Create a virtual idol and a virtual concert scene in the cloud, and integrate the virtual idol into the virtual concert scene.
  • step S1 includes the following sub-steps:
  • Step S110 Create a corresponding number of virtual idol models and corresponding virtual concert scene models in the cloud based on the required live concert program.
  • the virtual idol model uses a character model creation tool to obtain the three-dimensional character model of the virtual idol; the three-dimensional character model is imported into the model engine, and the three-dimensional character model is analyzed through the model engine to obtain the model data of the virtual idol.
  • Step S120 Integrate the virtual concert scene model and the virtual idol model to obtain an integrated model file.
  • Step S130 Import the integrated model file into the virtual engine to generate the integrated virtual idol and virtual concert scene.
  • the integrated virtual idol takes the virtual concert scene as the background.
  • Step S2 Collect the visual angle data and performance data of the local VR device in real time, and send the collected data to the cloud server.
  • step S2 includes the following sub-steps:
  • Step S210 Collect the visual angle of the local VR device in real time.
  • every angle change of the local VR device (local viewing device) will be collected and transmitted to the cloud virtual camera in real time.
  • the cloud virtual camera adjusts the shooting angle synchronously to keep the angles of the local VR device and the cloud virtual camera consistent, and then in The cloud renders the images captured by the cloud virtual camera in real time.
  • Step S220 Collect performance data in real time.
  • the performance data includes the motion capture actor's movement expression data and sound data; the motion capture actor's own movement and dance, the motion capture actor's expression and stage style; the sound data includes the motion capture actor's songs and background music.
  • Performance data is used to drive the virtual idol to obtain stage content: the stage content includes the virtual idol’s own movement and dance driven by the motion capture actor’s expression data, virtual idol expression and stage style, etc., sound data (songs and background music of the motion capture actor) etc.) and environmental special effects, etc.
  • Action expression data includes action data, expression data, etc.
  • Motion capture equipment is motion capture data acquisition equipment used by motion capture actors, including but not limited to inertial motion capture sensors (collecting motion data of motion capture actors through inertial sensors, and arranging inertial sensors on specific parts of the motion capture actor's body, such as Ankles, knees, thighs, arms, hands, chest, waist and shoulders, head, etc.
  • Inertial sensors can include accelerometers, gyroscopes and magnetometers, etc.), optical motion capture sensors (RGB cameras, depth cameras).
  • motion capture actors are the drivers of the movements of digital human virtual idols in the cloud.
  • the target position of the virtual idol in the virtual concert scene is determined, and the motion capture actor's movements collected by the inertial sensor
  • the data determines the moving distance and direction of the motion capture actor in the real scene of the concert; according to the distance mapping ratio in the scene mapping rules, determine the virtual idol corresponding to the moving distance of the motion capture actor in the real scene in the virtual concert scene. distance value; according to the direction mapping ratio in the scene mapping rules, determine the direction value of the virtual idol in the virtual concert scene corresponding to the moving direction of the motion capture actor in the real scene; according to the direction value of the virtual idol in the live concert scene
  • the distance and direction values determine the target location.
  • Step S230 transmit the collected visual angle and performance data of the local VR device to the cloud server in real time.
  • Step S3 Use the performance data to drive the virtual idol, and collect the driven virtual idol images and virtual concert scene images in real time according to the visual angle of the local VR device.
  • the virtual idol is driven based on the motion expression data of the motion capture actor, and based on the visual angle of the local VR device, the virtual camera is adjusted to an angle consistent with the visual angle of the local VR device, and the driven virtual idol is collected in real time.
  • the idol screen and the virtual concert scene screen, the virtual idol screen has the virtual concert scene screen as the background screen.
  • the position of the virtual camera is adjusted to be consistent with the position.
  • step S3 includes the following sub-steps:
  • Step S310 Drive the virtual idol based on the real-time collected motion and expression data of the motion capture actor.
  • the inertial sensor is set at the action part of the motion capture actor; the action data of the motion capture actor is collected through the inertial sensor, and the action data is transmitted to the model engine in real time through the virtual engine plug-in of the model engine, and the action data is combined with the body movements of the virtual idol. Association is made, and the virtual idol is driven to perform action data through bone binding data and skinning data.
  • the expression data of the motion capture actor is collected through the depth camera; the virtual engine plug-in of the model engine is used to associate the expression data of the motion capture actor with the facial expression of the virtual idol, and the expression control data is used to drive the virtual idol to execute the expression data.
  • Step S320 Synchronously adjust the shooting angle of the cloud virtual camera according to the visual angle of the local VR device.
  • the visual angle of the local VR device is transmitted to the cloud virtual camera, and the shooting angle of the cloud virtual camera is adjusted to be consistent with the visual angle of the local VR device to ensure that the angles of the local VR device and the cloud virtual camera are consistent.
  • Step S330 The cloud virtual camera that is consistent with the visual angle of the local VR device captures the virtual idol and virtual concert scene in real time.
  • Step S4 Render the collected virtual idol images and virtual concert scene images in real time, and insert sound data to obtain a cloud 360 VR video stream.
  • the method of rendering virtual idol images and virtual concert scene images includes mapping, adding materials, and adding textures to virtual idols and virtual concert scenes.
  • Rendering the virtual concert scene also includes inserting environmental special effects.
  • Step S5 Combine the cloud 360VR video stream and the local audience image into a virtual concert video stream.
  • the cloud 360VR video stream and the local audience picture are combined into a virtual concert video stream on the local computing device.
  • the existing Google's standard field of view relationship algorithm is used to synthesize the virtual concert video stream based on the field of view relationship between the cloud 360VR video stream and the local audience picture.
  • the method of synthesizing the virtual concert video stream according to the field of view relationship is: when the locally rendered audience interacts in the game, the picture of the cloud 360VR video stream is adjusted in real time according to the audience's position and field of view angle, and then the cloud 360VR video stream is obtained Virtual concert video stream synthesized with local audience footage.
  • the 360VR video stream is used as the HDR (high dynamic range image) environment background of the game in the existing game engine.
  • the local audience picture is a locally rendered auditorium picture.
  • the auditorium picture includes the audience Interaction between the audience and the audience: giving gifts to the host, changing looks, using props, communicating with other viewers, etc.
  • the audience is rendered by the local computing device (or local server) and does not need to be captured.
  • the existing virtual handle joystick operation technology is used to move the audience position.
  • the rendering method of the local audience picture includes: the local computing device obtains the audience data and the audience interaction data; the local computing device renders the auditorium based on the audience data and the audience interaction data to obtain the local audience picture, in which the local audience picture is obtained according to the number and location of the audience. Render the audience and then send the audience interaction data to other audience clients, which render the audience interaction data on their local computing devices.
  • the audience's interactive rendering method is: the audience gives the host a "fireworks" gift, and the effect is that a firework is released in the virtual scene of the concert. If rendering in the cloud, there will be a certain delay, so the fireworks rendering effect will appear directly in the audience's local area, and the local server will be notified at the same time. The local server will notify other audience clients, and all clients will render the fireworks effects locally.
  • the interaction in the concert may require the user to wave the light stick according to the music rhythm or accent point to make a call.
  • the behavior of each user is different, and the local server will be used to render the glow stick effect.
  • Step S6 Send the synthesized virtual concert video stream to the local viewing client.
  • the local viewing client usually refers to downloading a dedicated app (application) on a mobile phone, a PC browser, or installing an Android app on a VR all-in-one machine.
  • the local viewing client receives the virtual concert video stream and plays it.
  • this application provides a virtual concert rendering system 100 that combines VR end-end and cloud.
  • the system includes:
  • Building module 10 used to create virtual idols and virtual concert scenes in the cloud, and integrate virtual idols into the virtual concert scene;
  • the data collection device 20 is used to collect the visual angle data and performance data of the local VR device in real time, and send the collected data to the cloud server;
  • the driving module 30 uses performance data to drive the virtual idol
  • the picture collection device 40 is used to collect the driven virtual idol pictures and virtual concert scene pictures in real time according to the visual angle of the local VR device;
  • the rendering module 50 is used to render the collected virtual idol images and virtual concert scene images in real time, and insert sound data to obtain a cloud 360 VR video stream;
  • the video synthesis module 60 is used to synthesize the cloud 360 VR video stream and the local audience picture into a virtual concert video stream;
  • the data transmission module 70 is used to send the synthesized virtual concert video stream to the local viewing client.
  • the picture collection device 40 is a cloud virtual camera set up in the cloud. After the cloud virtual camera adjusts its visual angle to be consistent with the collected visual angle of the local VR device, it collects virtual idol pictures and virtual concert scene pictures.
  • the stage performance content output from the cloud rendering server in this application is a 360VR video stream.
  • This processing method is different from the traditional two-dimensional video stream and provides a more three-dimensional and realistic picture effect.
  • the audience uses the local VR device to rotate the viewing angle at will, the visual angle of the 360VR video stream and the local VR device will remain consistent, and there will be no picture distortion or blurring.
  • This application chooses to render the concert content on the cloud and place the auditorium images on Local rendering. Since the content of the concert is complex and the graphics requirements are high, it requires high-performance rendering computing power as support. Therefore, the computing power advantage of the cloud server improves the rendering efficiency and rendering effect of the concert content. Since the interaction in the auditorium requires real-time, low latency and high response speed, but the rendering computing power is not high, it is chosen to render locally, taking advantage of the delay advantage of local computing and improving the response speed of audience interaction.
  • This application enables performance activities such as concerts to be held online; motion capture actors/artists do not need to gather offline and can jointly access the concert through cloud collaboration; because the parts that require the most rendering computing power are transferred from the local Moving to the cloud reduces the device threshold for users to watch cloud concerts; due to sufficient computing power in the cloud, the concert director does not need to consider user-side performance issues in terms of effect, and has lower requirements for user-side performance and a lower threshold, thus better Give full play to artistic imagination; audiences can participate in concerts and other interpretation activities at any time and anywhere as long as they have smooth access to the Internet.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Processing Or Creating Images (AREA)

Abstract

本申请提供一种VR端端云结合的虚拟演唱会渲染方法及系统，该方法包括：在云端创建虚拟偶像和虚拟演唱会场景，并将虚拟偶像融合在虚拟演唱会场景中；实时采集本地VR设备的视觉角度数据和表演数据，并将采集的数据发送至云端服务器；利用表演数据驱动虚拟偶像；根据本地VR设备的视觉角度，实时采集被驱动后的虚拟偶像画面和虚拟演唱会场景画面；实时渲染采集的虚拟偶像画面和虚拟演唱会场景画面，并插入声音数据，获得云端360VR视频流；将云端360VR视频流与本地观众画面合成一个虚拟演唱会视频流；将合成的虚拟演唱会视频流发送给本地观看客户端。本申请渲染效率高、低延迟、高响应速度和低门槛。

Description

一种VR端端云结合的虚拟演唱会渲染方法及系统 技术领域
本申请涉及VR技术领域，尤其涉及一种VR端端云结合的虚拟演唱会渲染方法及系统。
背景技术
虚拟现实技术(VR)是一种综合应用计算机图形学，人机接口技术，传感器技术以及人工智能等技术，制造出逼真的人工模拟环境，并能有效地模拟人在自然环境中的各种感知的高级的人机交互技术。
云渲染(cloudrender)的模式与常规的云计算类似，即将3D程序放在远程的服务器中渲染，用户终端通过Web软件或者直接在本地的3D程序中点击一个"云渲染"按钮并借助高速互联网接入访问资源，指令从用户终端中发出，服务器根据指令执行对应的渲染任务，而渲染结果画面则被传送回用户终端中加以显示。
通过VR技术，观众无需线下聚集也可以获得无限接近现场演出的真实感和沉浸感。VR画面越细腻写实，所需要的计算机图形算力就越高。这就意味着，观众想要获得极致演唱会体验，就需要在本地观看设备和运算设备上投入重金。如此高的门槛，阻碍了观众的体验，也不利于行业发展。
本申请旨在从技术上解决以上问题，为演唱会的演绎方和观看方提供一种虚拟演唱会渲染效率高、低延迟、高响应速度和低门槛的解 决方案。
发明内容
本申请的目的在于提供一种VR端端云结合的虚拟演唱会渲染方法及系统，该方法渲染效率高、低延迟、高响应速度和低门槛。
为达到上述目的，本申请提供一种VR端端云结合的虚拟演唱会渲染方法，其特征在于，该方法包括：
在云端创建虚拟偶像和虚拟演唱会场景，并将虚拟偶像融合在虚拟演唱会场景中；
实时采集本地VR设备的视觉角度数据和表演数据；
利用表演数据驱动虚拟偶像；
根据本地VR设备的视觉角度，实时采集被驱动后的虚拟偶像画面和虚拟演唱会场景画面；
实时渲染采集的虚拟偶像画面和虚拟演唱会场景画面，并插入声音数据，获得云端360VR视频流；
将云端360VR视频流与本地观众画面合成一个虚拟演唱会视频流；
将合成的虚拟演唱会视频流发送给本地观看客户端。
如上的，其中，本地观众画面为在本地计算设备对观众席的观众和观众的互动进行渲染获得。
如上的，其中，表演数据包括动捕演员的动作表情数据和声音数据；动捕演员的自身动作舞蹈、动捕演员的表情和舞美造型；声音数据包括动捕演员的演唱歌曲和背景音乐。
如上的，其中，根据动捕演员的动作表情数据驱动虚拟偶像，并根据本地VR设备的视觉角度，将虚拟摄像机调整为与本地VR设备视觉角度一致的角度，通过调整后的虚拟摄像机实时采集被驱动后的虚拟偶像画面和虚拟演唱会场景画面。
如上的，其中，本地计算设备获得观众的互动数据后，发送给其他观众的客户端，其他观众的客户端在其本地计算设备上渲染观众互动数据，所有的观众客户端都渲染出观众互动效果。
如上的，其中，在云端创建虚拟偶像和虚拟演唱会场景，并将虚拟偶像融合在虚拟演唱会场景中的方法包括：
在云端根据所需直播的演唱会节目创建对应的数量的虚拟偶像模型及对应的虚拟演唱会场景模型；
将虚拟演唱会场景模型及虚拟偶像模型进行整合处理，得到整合模型文件；
将整合模型文件导入至虚拟引擎，生成融合后的虚拟偶像和虚拟演唱会场景。
如上的，其中，通过动捕设备采集动捕演员的动作表情数据；其中，动作表情数据包括动作数据和表情数据；
其中，动捕设备包括惯性动捕传感器和光学动捕传感器。
如上的，其中，通过模型引擎的虚拟引擎插件将动作数据实时传输至模型引擎，将动作数据与虚拟偶像的肢体动作进行关联，并通过骨骼绑定数据和蒙皮数据驱动虚拟偶像执行动作数据；利用模型引擎的虚拟引擎插件将动捕演员的表情数据与虚拟偶像的面部表情进行关联，并通过表情控制数据驱动虚拟偶像执行表情数据。
本申请还提供一种VR端端云结合的虚拟演唱会渲染系统，该系统包括：
构建模块，用于在云端创建虚拟偶像和虚拟演唱会场景，并将虚拟偶像融合在虚拟演唱会场景中；
数据采集设备，用于实时采集本地VR设备的视觉角度数据和表演数据；
驱动模块，利用表演数据驱动虚拟偶像；
画面采集设备，用于根据本地VR设备的视觉角度，实时采集被驱动后的虚拟偶像画面和虚拟演唱会场景画面；
渲染模块，用于实时渲染采集的虚拟偶像画面和虚拟演唱会场景画面，并插入声音数据，获得云端360VR视频流；
视频合成模块，用于将云端360VR视频流与本地观众画面合成一个虚拟演唱会视频流；
数据传输模块，用于将合成的虚拟演唱会视频流发送给本地观看客户端。
如上的，其中，画面采集设备为在云端设置的云端虚拟摄像机，云端虚拟摄像机调整其视觉角度与采集的本地VR设备的视觉角度一致后，采集虚拟偶像画面和虚拟演唱会场景画面。
本申请实现的有益效果如下：
(1)本申请云端渲染服务器上输出的舞台表演内容为360VR视频流，这种处理方式区别于传统二维视频流，提供了更加立体真实的画面效果。另外，当观众使用本地VR设备随意旋转观看角度时，360VR视频流画面与本地VR设备视觉角度将保持一致，不会出现画面畸变、 穿帮的的情况。
(2)本申请选择将演唱会内容放在云端渲染，将观众席画面放在本地渲染。由于演唱会内容复杂、画面要求高，需要高性能的渲染算力作为支撑，因此，云端服务器的算力优势提高了演唱会内容的渲染效率和渲染效果。由于观众席互动要求实时，需要低延迟、高响应速度，但是对于渲染算力要求不高，因此选择放在本地渲染，发挥了本地计算的延迟优势，提高观众互动响应速度。
(3)本申请使得演唱会等演绎表演活动可以在线上举行；动捕演员/艺术家无需在线下聚集，可以通过云端协作的方式共同接入演唱会；由于将最需要渲染算力的部分从本地移到云端进行，降低用户观看云端演唱会的设备门槛；由于云端算力充足，演唱会导演在效果上无需考虑用户端性能问题，对用户端性能要求较低，门槛较低，从而更好的发挥艺术想象空间；观众可以在任何时间任何地点，只要能够流畅的接入网络即可参与演唱会等演绎活动。
附图说明
为了更清楚地说明本申请实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请中记载的一些实施例，对于本领域技术人员来讲，还可以根据这些附图获得其他的附图。
图1为本申请实施例的一种VR端端云结合的虚拟演唱会渲染方法的流程图。
图2为本申请实施例的在云端创建虚拟偶像和虚拟演唱会场景的方法流程图。
图3为本申请实施例的实时采集本地VR设备的视觉角度数据和表演数据的流程图。
图4为本申请实施例的实时采集被驱动后的虚拟偶像画面和虚拟演唱会场景画面的流程图。
图5为本申请实施例的一种VR端端云结合的虚拟演唱会渲染系统的结构示意图。
附图标记：10-构建模块；20-数据采集设备；30-驱动模块；40-画面采集设备；50-渲染模块；60-视频合成模块；70-数据传输模块；100-虚拟演唱会渲染系统。
具体实施方式
下面结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
实施例一
如图1所示，本申请提供一种VR端端云结合的虚拟演唱会渲染方法，该方法包括：
步骤S1，在云端创建虚拟偶像和虚拟演唱会场景，并将虚拟偶像融合在虚拟演唱会场景中。
如图2所示，步骤S1包括如下子步骤：
步骤S110，在云端根据所需直播的演唱会节目创建对应的数量的虚拟偶像模型及对应的虚拟演唱会场景模型。
具体的，虚拟偶像模型是利用角色模型创建工具获取虚拟偶像的三维角色模型；将三维角色模型导入模型引擎，并通过模型引擎对三维角色模型进行解析，获得虚拟偶像的模型数据。
步骤S120，将虚拟演唱会场景模型及虚拟偶像模型进行整合处理，得到整合模型文件。
步骤S130，将整合模型文件导入至虚拟引擎，生成融合后的虚拟偶像和虚拟演唱会场景。
具体的，融合后的虚拟偶像以虚拟演唱会场景为背景。
步骤S2，实时采集本地VR设备的视觉角度数据和表演数据，并将采集的数据发送至云端服务器。
如图3所示，步骤S2包括如下子步骤：
步骤S210，实时采集本地VR设备的视觉角度。
具体的，本地VR设备(本地观看设备)的每次角度变化都会被采集，并实时传输给云端虚拟摄像机，云端虚拟摄像机同步调整拍摄角度，保持本地VR设备和云端虚拟摄像机的角度一致，然后在云端实时渲染出云端虚拟摄像机拍摄到的画面。
步骤S220，实时采集表演数据。
其中，表演数据包括动捕演员的动作表情数据和声音数据；动捕演员的自身动作舞蹈、动捕演员的表情和舞美造型；声音数据包括动捕演员的歌曲和背景音乐。
表演数据用于驱动虚拟偶像获得舞台内容：舞台内容包括根据动捕演员的动作表情数据驱动的虚拟偶像自身动作舞蹈、虚拟偶像表情和舞美造型等、声音数据(动捕演员的演唱歌曲和背景音乐等)和环境特效等。
其中，通过动捕设备采集动捕演员的动作表情数据(虚拟偶像自身动作舞蹈、虚拟偶像表情和舞美造型等)和声音数据(歌曲和背景音乐等)。动作表情数据包括动作数据和表情数据等。
动捕设备是动捕演员使用的动作捕捉数据获取设备，包括但不限于惯性动捕传感器(通过惯性传感器采集动捕演员的动作数据，将惯性传感器配置于动捕演员的身体的特定部位，比如脚踝、膝盖、大腿、胳膊、手、胸部、腰部和肩部、头部等等，惯性传感器可以包括加速度计、陀螺仪和磁力计等)、光学动捕传感器(RGB摄像头、深度摄像头)。
其中，动捕演员是云端数字人虚拟偶像动作的驱动者。
根据演唱会真实场景与虚拟演唱会场景的映射关系，根据动捕演员在演唱会真实场景中的位置，确定在虚拟演唱会场景中虚拟偶像的目标位置，通过惯性传感器采集的动捕演员的动作数据确定动捕演员在演唱会真实场景中的移动距离和移动方向；按照场景映射规则中的距离映射比例，确定动捕演员在真实场景中的移动距离所对应的虚拟偶像在虚拟演唱会场景中的距离值；按照场景映射规则中的方向映射比例，确定动捕演员在真实场景中的移动方向所对应的虚拟偶像在虚拟演唱会场景中的方向值；根据虚拟偶像在演唱会直播场景中的距离值和方向值确定目标位置。
步骤S230，将采集的本地VR设备的视觉角度和表演数据实时传入云端服务器。
步骤S3，利用表演数据驱动虚拟偶像，并根据本地VR设备的视觉角度，实时采集被驱动后的虚拟偶像画面和虚拟演唱会场景画面。
具体的，在云端服务器，根据动捕演员的动作表情数据驱动虚拟偶像，并根据本地VR设备的视觉角度，将虚拟摄像机调整为与本地VR设备视觉角度一致的角度，实时采集被驱动后的虚拟偶像画面和虚拟演唱会场景画面，虚拟偶像画面以虚拟演唱会场景画面为背景画面。优选的，根据携带本地VR设备的观众在虚拟演唱会场景中的位置，调整虚拟摄像机的位置与该位置保持一致。
如图4所示，步骤S3包括如下子步骤：
步骤S310，根据实时采集的动捕演员的动作表情数据驱动虚拟偶像。
具体的，惯性传感器设置在动捕演员的动作部位；通过惯性传感器采集动捕演员的动作数据，通过模型引擎的虚拟引擎插件将动作数据实时传输至模型引擎，将动作数据与虚拟偶像的肢体动作进行关联，并通过骨骼绑定数据和蒙皮数据驱动虚拟偶像执行动作数据。通过深度摄像头采集动捕演员的表情数据；利用模型引擎的虚拟引擎插件将动捕演员的表情数据与虚拟偶像的面部表情进行关联，并通过表情控制数据驱动虚拟偶像执行表情数据。
步骤S320，根据本地VR设备的视觉角度，同步调整云端虚拟摄像机的拍摄角度。
具体的，将本地VR设备的视觉角度传输给云端虚拟摄像机，调整云端虚拟摄像机的拍摄角度与本地VR设备的视觉角度一致，保证本地VR设备和云端虚拟摄像机的角度保持一致。
步骤S330，与本地VR设备的视觉角度保持一致的云端虚拟摄像机实时拍摄虚拟偶像和虚拟演唱会场景画面。
步骤S4，实时渲染采集的虚拟偶像画面和虚拟演唱会场景画面，并插入声音数据，获得云端360VR视频流。
具体的，渲染虚拟偶像画面和虚拟演唱会场景画面的方法包括为虚拟偶像和虚拟演唱会场景贴图、添加材质和添加纹理等。渲染虚拟演唱会场景画面还包括插入环境特效。将获得的云端360VR视频流发送至本地计算设备。
步骤S5，将云端360VR视频流与本地观众画面合成一个虚拟演唱会视频流。
具体的，在本地计算设备上将云端360VR视频流与本地观众画面合成一个虚拟演唱会视频流。其中，采用现有的Google的视场角标准关系算法，将云端360VR视频流与本地观众画面按照视场关系合成虚拟演唱会视频流。其中，按照视场关系合成虚拟演唱会视频流的方法为：本地渲染的观众在游戏中互动时，根据观众所属的位置和视场角度实时调整云端360VR视频流的画面，进而获得云端360VR视频流与本地观众画面合成的虚拟演唱会视频流。
作为本发明的一个具体实施例，在现有的游戏引擎中将360VR视频流作为游戏的HDR(高动态范围图像)环境背景。
本地观众画面为在本地渲染的观众席画面，观众席画面包括观 众和观众的互动：给主播送礼物、更换造型、使用道具、与其他观众交流等。观众是本地计算设备(或本地服务器)渲染的，不需要捕捉，采用现有的虚拟手柄摇杆操作技术移动观众位置。
本地观众画面的渲染方法包括：本地计算设备获取观众数据和观众互动数据；在本地计算设备根据观众数据和观众互动数据对观众席进行渲染获得本地观众画面，其中根据观众的数量和位置在观众席渲染观众，然后将观众互动数据发送给其他观众的客户端，其他观众的客户端在其本地计算设备渲染观众互动数据。
观众的互动渲染方法例如：观众给主播赠送一个“烟花”礼物，效果是在演唱会虚拟场景放出一个烟花。如果在云端渲染会存在一定的延迟，所以会在观众本地直接出现烟花渲染效果，同时通知本地服务器，本地服务器通知其他观众的客户端，所有客户端本地渲染烟花效果。
观众的互动渲染方法再例如：演唱会中的互动可能会出现用户需要根据音乐节奏或重音点挥动荧光棒打call。每个用户的行为不相同，会采取本地服务器渲染荧光棒效果。
步骤S6，将合成的虚拟演唱会视频流发送给本地观看客户端。
其中，本地观看客户端通常是指在手机上下载专属的app(应用)、PC端浏览器、或者VR一体机上安装安卓app。本地观看客户端接收虚拟演唱会视频流后进行播放。
实施例二
如图5所示，本申请提供一种VR端端云结合的虚拟演唱会渲染系统100，该系统包括：
构建模块10，用于在云端创建虚拟偶像和虚拟演唱会场景，并将虚拟偶像融合在虚拟演唱会场景中；
数据采集设备20，用于实时采集本地VR设备的视觉角度数据和表演数据，并将采集的数据发送至云端服务器；
驱动模块30，利用表演数据驱动虚拟偶像；
画面采集设备40，用于根据本地VR设备的视觉角度，实时采集被驱动后的虚拟偶像画面和虚拟演唱会场景画面；
渲染模块50，用于实时渲染采集的虚拟偶像画面和虚拟演唱会场景画面，并插入声音数据，获得云端360VR视频流；
视频合成模块60，用于将云端360VR视频流与本地观众画面合成一个虚拟演唱会视频流；
数据传输模块70，用于将合成的虚拟演唱会视频流发送给本地观看客户端。
画面采集设备40为在云端设置的云端虚拟摄像机，云端虚拟摄像机调整其视觉角度与采集的本地VR设备的视觉角度一致后，采集虚拟偶像画面和虚拟演唱会场景画面
本申请实现的有益效果如下：
(1)本申请云端渲染服务器上输出的舞台表演内容为360VR视频流，这种处理方式区别于传统二维视频流，提供了更加立体真实的画面效果。另外，当观众使用本地VR设备随意旋转观看角度时，360VR视频流画面与本地VR设备视觉角度将保持一致，不会出现画面畸变、穿帮的的情况。
(2)本申请选择将演唱会内容放在云端渲染，将观众席画面放在 本地渲染。由于演唱会内容复杂、画面要求高，需要高性能的渲染算力作为支撑，因此，云端服务器的算力优势提高了演唱会内容的渲染效率和渲染效果。由于观众席互动要求实时，需要低延迟、高响应速度，但是对于渲染算力要求不高，因此选择放在本地渲染，发挥了本地计算的延迟优势，提高观众互动响应速度。
(3)本申请使得演唱会等演绎表演活动可以在线上举行；动捕演员/艺术家无需在线下聚集，可以通过云端协作的方式共同接入演唱会；由于将最需要渲染算力的部分从本地移到云端进行，降低用户观看云端演唱会的设备门槛；由于云端算力充足，演唱会导演在效果上无需考虑用户端性能问题，对用户端性能要求较低，门槛较低，从而更好的发挥艺术想象空间；观众可以在任何时间任何地点，只要能够流畅的接入网络即可参与演唱会等演绎活动。
以上所述仅为本发明的实施方式而已，并不用于限制本发明。对于本领域技术人员来说，本发明可以有各种更改和变化。凡在本发明的精神和原理内所做的任何修改、等同替换、改进等，均应包括在本发明的权利要求范围之内。

Claims (10)

  1. 一种VR端端云结合的虚拟演唱会渲染方法，其特征在于，该方法包括：
    在云端创建虚拟偶像和虚拟演唱会场景，并将虚拟偶像融合在虚拟演唱会场景中；
    实时采集本地VR设备的视觉角度数据和表演数据；
    利用表演数据驱动虚拟偶像；
    根据本地VR设备的视觉角度，实时采集被驱动后的虚拟偶像画面和虚拟演唱会场景画面；
    实时渲染采集的虚拟偶像画面和虚拟演唱会场景画面，并插入声音数据，获得云端360VR视频流；
    将云端360VR视频流与本地观众画面合成一个虚拟演唱会视频流；
    将合成的虚拟演唱会视频流发送给本地观看客户端。
  2. 根据权利要求1所述的VR端端云结合的虚拟演唱会渲染方法，其特征在于，本地观众画面为在本地计算设备对观众席的观众和观众的互动进行渲染获得。
  3. 根据权利要求1所述的VR端端云结合的虚拟演唱会渲染方法，其特征在于，表演数据包括动捕演员的动作表情数据和声音数据；动捕演员的自身动作舞蹈、动捕演员的表情和舞美造型；声音数据包括动捕演员的演唱歌曲和背景音乐。
  4. 根据权利要求3所述的VR端端云结合的虚拟演唱会渲染方法，其特征在于，根据动捕演员的动作表情数据驱动虚拟偶像，并根 据本地VR设备的视觉角度，将虚拟摄像机调整为与本地VR设备视觉角度一致的角度，通过调整后的虚拟摄像机实时采集被驱动后的虚拟偶像画面和虚拟演唱会场景画面。
  5. 根据权利要求2所述的VR端端云结合的虚拟演唱会渲染方法，其特征在于，本地计算设备获得观众的互动数据后，发送给其他观众的客户端，其他观众的客户端在其本地计算设备上渲染观众互动数据，所有的观众客户端都渲染出观众互动效果。
  6. 根据权利要求1所述的VR端端云结合的虚拟演唱会渲染方法，其特征在于，在云端创建虚拟偶像和虚拟演唱会场景，并将虚拟偶像融合在虚拟演唱会场景中的方法包括：
    在云端根据所需直播的演唱会节目创建对应的数量的虚拟偶像模型及对应的虚拟演唱会场景模型；
    将虚拟演唱会场景模型及虚拟偶像模型进行整合处理，得到整合模型文件；
    将整合模型文件导入至虚拟引擎，生成融合后的虚拟偶像和虚拟演唱会场景。
  7. 根据权利要求3所述的VR端端云结合的虚拟演唱会渲染方法，其特征在于，通过动捕设备采集动捕演员的动作表情数据；其中，动作表情数据包括动作数据和表情数据；
    其中，动捕设备包括惯性动捕传感器和光学动捕传感器。
  8. 根据权利要求7所述的VR端端云结合的虚拟演唱会渲染方法，其特征在于，通过模型引擎的虚拟引擎插件将动作数据实时传输至模型引擎，将动作数据与虚拟偶像的肢体动作进行关联，并通过骨 骼绑定数据和蒙皮数据驱动虚拟偶像执行动作数据；利用模型引擎的虚拟引擎插件将动捕演员的表情数据与虚拟偶像的面部表情进行关联，并通过表情控制数据驱动虚拟偶像执行表情数据。
  9. 一种VR端端云结合的虚拟演唱会渲染系统，其特征在于，该系统包括：
    构建模块，用于在云端创建虚拟偶像和虚拟演唱会场景，并将虚拟偶像融合在虚拟演唱会场景中；
    数据采集设备，用于实时采集本地VR设备的视觉角度数据和表演数据；
    驱动模块，利用表演数据驱动虚拟偶像；
    画面采集设备，用于根据本地VR设备的视觉角度，实时采集被驱动后的虚拟偶像画面和虚拟演唱会场景画面；
    渲染模块，用于实时渲染采集的虚拟偶像画面和虚拟演唱会场景画面，并插入声音数据，获得云端360VR视频流；
    视频合成模块，用于将云端360VR视频流与本地观众画面合成一个虚拟演唱会视频流；
    数据传输模块，用于将合成的虚拟演唱会视频流发送给本地观看客户端。
  10. 根据权利要求9所述的VR端端云结合的虚拟演唱会渲染系统，其特征在于，画面采集设备为在云端设置的云端虚拟摄像机，云端虚拟摄像机调整其视觉角度与采集的本地VR设备的视觉角度一致后，采集虚拟偶像画面和虚拟演唱会场景画面。
PCT/CN2023/106828 2022-07-15 2023-07-11 一种vr端端云结合的虚拟演唱会渲染方法及系统 WO2024012459A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210830431.5 2022-07-15
CN202210830431.5A CN114900678B (zh) 2022-07-15 2022-07-15 一种vr端端云结合的虚拟演唱会渲染方法及系统

Publications (1)

Publication Number Publication Date
WO2024012459A1 true WO2024012459A1 (zh) 2024-01-18

Family

ID=82730157

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/106828 WO2024012459A1 (zh) 2022-07-15 2023-07-11 一种vr端端云结合的虚拟演唱会渲染方法及系统

Country Status (2)

Country Link
CN (1) CN114900678B (zh)
WO (1) WO2024012459A1 (zh)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114900678B (zh) * 2022-07-15 2022-09-30 北京蔚领时代科技有限公司 一种vr端端云结合的虚拟演唱会渲染方法及系统
CN115494962A (zh) * 2022-11-18 2022-12-20 清华大学深圳国际研究生院 一种虚拟人实时互动系统及方法
CN115657862B (zh) * 2022-12-27 2023-03-14 海马云（天津）信息技术有限公司 虚拟ktv场景画面自动切换的方法、装置、存储介质及设备

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112235585A (zh) * 2020-08-31 2021-01-15 江苏视博云信息技术有限公司 一种虚拟场景的直播方法、装置及系统
JP2021041241A (ja) * 2020-12-10 2021-03-18 株式会社バンダイナムコエンターテインメント プログラム及び電子機器
CN113822970A (zh) * 2021-09-23 2021-12-21 广州博冠信息科技有限公司 直播控制方法、装置、存储介质与电子设备
CN114900678A (zh) * 2022-07-15 2022-08-12 北京蔚领时代科技有限公司 一种vr端端云结合的虚拟演唱会渲染方法及系统

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109416842B (zh) * 2016-05-02 2023-08-29 华纳兄弟娱乐公司 在虚拟现实和增强现实中的几何匹配
CN106843507B (zh) * 2017-03-24 2024-01-05 苏州创捷传媒展览股份有限公司 一种虚拟现实多人互动的方法及系统
US10438414B2 (en) * 2018-01-26 2019-10-08 Microsoft Technology Licensing, Llc Authoring and presenting 3D presentations in augmented reality
CN109841196B (zh) * 2018-12-24 2021-09-28 武汉西山艺创文化有限公司 一种基于透明液晶显示屏的虚拟偶像演播系统
CN112019921A (zh) * 2020-09-01 2020-12-01 北京德火科技有限责任公司 应用于虚拟演播室的肢体动作数据处理方法
CN112558761A (zh) * 2020-12-08 2021-03-26 南京航空航天大学 一种面向移动端的远程虚拟现实交互系统及交互方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112235585A (zh) * 2020-08-31 2021-01-15 江苏视博云信息技术有限公司 一种虚拟场景的直播方法、装置及系统
JP2021041241A (ja) * 2020-12-10 2021-03-18 株式会社バンダイナムコエンターテインメント プログラム及び電子機器
CN113822970A (zh) * 2021-09-23 2021-12-21 广州博冠信息科技有限公司 直播控制方法、装置、存储介质与电子设备
CN114900678A (zh) * 2022-07-15 2022-08-12 北京蔚领时代科技有限公司 一种vr端端云结合的虚拟演唱会渲染方法及系统

Also Published As

Publication number Publication date
CN114900678A (zh) 2022-08-12
CN114900678B (zh) 2022-09-30

Similar Documents

Publication Publication Date Title
WO2024012459A1 (zh) 一种vr端端云结合的虚拟演唱会渲染方法及系统
WO2022062678A1 (zh) 虚拟直播方法、装置、系统及存储介质
US9939887B2 (en) Avatar control system
KR102077108B1 (ko) 콘텐츠 체험 서비스 제공 장치 및 그 방법
CN108986189B (zh) 基于三维动画中实时多人动作捕捉并直播的方法和系统
CN107170030A (zh) 一种虚拟主播直播方法及系统
TWI752502B (zh) 一種分鏡效果的實現方法、電子設備及電腦可讀儲存介質
JP2021525911A (ja) マルチサーバクラウド仮想現実（ｖｒ）ストリーミング
CN107197385A (zh) 一种实时虚拟偶像直播方法及系统
CN109069934A (zh) 对虚拟现实环境(vr)中的vr用户的观众视图跟踪
CN107248185A (zh) 一种虚拟仿真偶像实时直播方法及系统
CN111862348B (zh) 视频显示方法、视频生成方法、装置、设备及存储介质
CN113822970A (zh) 直播控制方法、装置、存储介质与电子设备
CN110568923A (zh) 基于Unity3D的虚拟现实交互方法、装置、设备及存储介质
WO2023045637A1 (zh) 视频数据生成方法、装置、电子设备及可读存储介质
JP2023504609A (ja) ハイブリッドストリーミング
WO2023236656A1 (zh) 互动画面的渲染方法、装置、设备、存储介质及程序产品
JP6559375B1 (ja) コンテンツ配信システム、コンテンツ配信方法、およびコンテンツ配信プログラム
WO2024027063A1 (zh) 直播方法、装置、存储介质、电子设备及产品
Zhu et al. Integrated Co-Designing Using Building Information Modeling and Mixed Reality with Erased Backgrounds for Stock Renovation
Zipor Integrating Motion Capture Technology into Theatrical Performance
US20240163528A1 (en) Video data generation method and apparatus, electronic device, and readable storage medium
Matos et al. The visorama system: A functional overview of a new virtual reality environment
US20240048780A1 (en) Live broadcast method, device, storage medium, electronic equipment and product
Gomide Motion capture and performance

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23838950

Country of ref document: EP

Kind code of ref document: A1