CN117411979A

CN117411979A - Virtual guide system and virtual guide method

Info

Publication number: CN117411979A
Application number: CN202311323485.3A
Authority: CN
Inventors: 薛萌; 张超彦; 陈安
Original assignee: Bizhen Technology Shanghai Co ltd
Current assignee: Bizhen Technology Shanghai Co ltd
Priority date: 2023-10-12
Filing date: 2023-10-12
Publication date: 2024-01-16

Abstract

The invention provides a virtual guide system, which comprises: a plurality of guide clients, a guide server, a guide table, a logic server and a plurality of user clients; the guiding client at least comprises a plurality of virtual cameras, a plurality of camera service units and a plurality of WebRTC plug flow modules, wherein the plurality of camera service units are used for reading the position parameters of the plurality of virtual cameras and registering the virtual cameras with the plurality of WebRTC plug flow modules; the WebRTC streaming pushing modules send the registration information of the registered virtual cameras to the guide server; and the guiding and broadcasting station is used for acquiring the camera list of the registered virtual cameras from the guiding and broadcasting server, and carrying out signaling exchange with the WebRTC streaming modules through the guiding and broadcasting server so as to establish P2P connection between the guiding and broadcasting station and the WebRTC streaming modules. The invention fully utilizes hardware resources, simultaneously processes a plurality of virtual camera pictures on one guide client, and expands the number of the guide virtual camera pictures to the greatest extent on the basis of reducing design and development work.

Description

Virtual guide system and virtual guide method

Technical Field

The invention relates to the technical field of virtual reality and meta universe, in particular to a virtual guide system and a virtual guide method.

Background

With the rapid development of graphic technology and network technology, the variety and frequency of virtual performance are rapidly increasing, and the novel virtual performance adopts a cloud rendering mode to enable audiences to directly enter a 3D scene in the form of an avatar without downloading any client to interact with other audiences and even performers to become meta-space (Metaverse) performance in the virtual performance.

Like traditional shows, such activities often also require the provision of a director's view that allows the audience to enjoy the entire show in a "pure" manner. The view angle of the pure-sharing mode of the meta-space (Metaverse) performance activity is usually controlled by a director, and similar to the traditional offline performance, the director is usually a team rather than an individual, and how to enable the director team to complete the director task with high quality in the activity process for the brand-new category of the meta-space performance becomes a necessary requirement in the near future, but the director is still an unmanned field at present.

The conventional guiding of virtual performance still resembles the conventional video rebroadcasting in flow, because the content of the performance is mostly prerecorded into video and then played in the conventional manner, but the meta-universe performance has a large number of 3D and real-time interactive components, so that a great deal of design and development work is required to achieve a satisfactory guiding effect, the current industry belongs to the field of unmanned technology, and along with the gradual maturation of the meta-universe performance form, the intervention of guiding team becomes necessary.

Disclosure of Invention

The invention provides a virtual guide system and a virtual guide method, which are used for solving the technical problem that a great deal of design and development work is required for guide in meta-space performance in the prior art.

One aspect of the present invention provides a virtual director system, comprising: a plurality of guide clients, a guide server, a guide table, a logic server and a plurality of user clients;

the guide client is in communication connection with the guide server, the guide server is in communication connection with a guide table, the guide client is in communication connection with the logic server, and the logic server is in communication connection with the user client;

the guiding client at least comprises a plurality of virtual cameras, a plurality of camera service units and a plurality of WebRTC plug flow modules, wherein one virtual camera corresponds to one camera service unit, and one camera service unit corresponds to one WebRTC plug flow module;

the camera service units are used for reading the position parameters of the virtual cameras and registering the virtual cameras to the WebRTC plug flow modules;

the WebRTC streaming pushing modules send registration information of the registered virtual cameras to the guide server;

the guiding and broadcasting station is used for acquiring camera lists of a plurality of registered virtual cameras from the WebRTC push modules, and carrying out signaling exchange with the WebRTC push modules through the guiding and broadcasting server so that the guiding and broadcasting station and the WebRTC push modules are connected by P2P.

In a preferred embodiment, the positions of a plurality of the virtual cameras are preset.

In a preferred embodiment, the multicast station requests virtual camera pictures of a plurality of virtual cameras from the multicast client through the multicast server;

the camera service units are used for carrying out virtual camera picture rendering on the plurality of virtual cameras and sending the virtual camera pictures of the plurality of virtual cameras after rendering to the corresponding WebRTC plug flow modules;

the WebRTC streaming pushing modules are connected with the P2P established by the WebRTC streaming pushing modules through the guide table, and virtual camera pictures of the virtual cameras are sent to the guide table;

the guide station sends a virtual camera switching instruction to the guide client through the guide server;

the plurality of camera service units switch virtual camera pictures according to the virtual camera switching instruction sent by the broadcasting guide table and forward the virtual camera switching instruction to the logic server;

the logic server broadcasts a virtual camera switching instruction to a user client, and the user client locally renders a camera picture corresponding to the virtual camera according to the virtual camera switching instruction.

In a preferred embodiment, the camera business unit comprises an acquisition module and a processing module;

the acquisition module is used for acquiring a virtual picture of the virtual camera; the processing module is used for processing the virtual picture of the virtual camera.

In a preferred embodiment, the acquisition module comprises an audio acquisition module and a video acquisition module; the virtual pictures of the virtual camera comprise audio of the set virtual camera and video of the virtual camera;

the audio acquisition module is used for acquiring the audio of the virtual camera, and the video acquisition module is used for acquiring the video of the virtual camera;

the processing module comprises an audio rendering module, an audio encoding module, a video rendering module and a video encoding module;

the audio rendering module is used for rendering the audio, and the audio encoding module is used for encoding the rendered audio;

the video rendering module is used for rendering the video, and the video encoding module is used for encoding the rendered video.

In a preferred embodiment, the director stations include a primary director station and a plurality of secondary director stations;

the master broadcasting station and the plurality of slave broadcasting stations have different authorities of sending virtual camera switching instructions to the plurality of camera service units of the broadcasting client through the broadcasting guide server.

Another aspect of the present invention provides a virtual director method, including the following method steps:

s1, a multicast station establishes P2P connection with a plurality of multicast clients;

the guiding and broadcasting station acquires camera lists of a plurality of registered virtual cameras from a plurality of WebRTC streaming modules, and performs signaling exchange with the WebRTC streaming modules through the guiding and broadcasting server so that the guiding and broadcasting station and the WebRTC streaming modules establish P2P connection;

s2, pushing virtual camera pictures of a plurality of virtual cameras to the guide platform by the guide client;

s3, the guide and broadcast station sends a virtual camera switching instruction to the user client;

and S4, the user client locally renders a camera picture corresponding to the virtual camera according to the virtual camera switching instruction.

In a preferred embodiment, in step S2, the multicast station requests, from the multicast client, a virtual camera screen of a plurality of virtual cameras through the multicast server;

and the WebRTC streaming pushing modules are connected with the P2P established by the WebRTC streaming pushing modules through the guide table, and virtual camera pictures of the virtual cameras are sent to the guide table.

In a preferred embodiment, in step S3, the multicast table sends a virtual camera switching instruction to the multicast client through the multicast server;

and the logic server broadcasts a virtual camera switching instruction to the user client.

Compared with the prior art, the invention has the following beneficial effects:

the virtual guide system and the virtual guide method provided by the invention provide a complete solution for the characteristics of meta-space performance, and the virtual camera picture rendering and the virtual camera picture pushing of the virtual camera are respectively responsible for the virtual camera through the parallel work of a plurality of guide clients, so that the number of the virtual camera pictures of the guide is expanded to the greatest extent on the basis of reducing design and development work.

According to the virtual guide system and the virtual guide method, on the premise that the real-time performance meets the requirement, the hardware resources are fully utilized, a plurality of virtual camera pictures can be processed at the same time on one guide client, and the number of the virtual camera pictures for guide is expanded to the greatest extent on the basis of reducing design and development work.

According to the virtual guide system and the virtual guide method, virtual camera pictures of a plurality of virtual cameras are rendered through the guide client side at the same time, and the virtual camera pictures are pushed to the guide platform in a P2P mode. The guide broadcasting station previews all virtual camera pictures, selects a virtual camera picture suitable for real-time switching as a current live broadcast virtual camera picture, broadcasts the current live broadcast virtual camera picture to all user clients watching live broadcast through the logic server, and locally renders a finally presented camera picture according to the current live broadcast virtual camera picture by the user clients.

The virtual guiding system and the virtual guiding method provided by the invention realize the functions of switching guiding pictures in real time and synchronizing to the user client side in virtual performance, the guiding platform and the guiding client side are connected in a P2P mode, the guiding client side sends the virtual camera pictures of a plurality of virtual cameras to the guiding platform through the WebRTC protocol, guiding personnel do not need to download the guiding platform client side, the guiding platform is connected to the guiding server through a browser, the guiding server can be operated only through the browser, and the guiding team division cooperative work is supported.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the present invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.

Fig. 1 is a block diagram of a virtual multicast system according to the present invention.

Fig. 2 is a block diagram of the structure of the client for guiding the broadcasting of the present invention.

Fig. 3 is a timing diagram of a virtual multicast system according to the present invention.

Fig. 4 is a flow chart of a virtual director method of the present invention.

Detailed Description

To further clarify the above and other features and advantages of the present invention, a further description of the invention will be rendered by reference to the appended drawings. It should be understood that the specific embodiments presented herein are for purposes of explanation to those skilled in the art and are intended to be illustrative only and not limiting.

In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings are merely for convenience in describing the present invention and simplifying the description, and do not indicate or imply that the device or element being referred to must have a specific orientation, be configured and operated in a specific orientation, and therefore should not be construed as limiting the present invention.

Furthermore, the terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In the description of the present invention, the meaning of "plurality" means at least two, for example, two, three, etc., unless specifically defined otherwise.

Referring to fig. 1 to 3, according to an embodiment of the present invention, there is provided a virtual director system including: a plurality of multicast clients 100, a multicast server 200, a multicast station 400, a logical server 300, and a plurality of user clients 500.

In this embodiment, two multicast clients 100 are shown as an example, and in some embodiments, the multicast clients 100 are more, specifically configured according to actual needs. In this embodiment, three user clients 500 are shown as an example, and in some embodiments, the user clients 500 are more, specifically configured according to actual needs.

The multicast client 100 is communicatively connected to the multicast server 200, the multicast server 200 is communicatively connected to the multicast station 400, the multicast client 100 is communicatively connected to the logical server 300, and the logical server 300 is communicatively connected to the user client 500.

The unicast client 100 at least comprises a plurality of virtual cameras 103, a plurality of camera service units 101 and a plurality of WebRTC push modules 102, wherein one virtual camera 103 corresponds to one camera service unit 101, and one camera service unit 101 corresponds to one WebRTC push module 102.

In this embodiment, two virtual cameras 103, camera service units 101 and WebRTC plug modules 102 are exemplarily shown, and in some embodiments, the virtual cameras 103, camera service units 101 and WebRTC plug modules 102 are more, specifically configured according to actual needs.

As shown in fig. 1 and 3, the plurality of camera service units 101 are configured to read the position parameters of the plurality of virtual cameras 103, and register the virtual cameras with the plurality of WebRTC push modules 102.

According to an embodiment of the present invention, the positions of the plurality of virtual cameras 103 are preset. Specifically, a series of parameters such as the angle of view, position, moving route, and the like of the virtual camera 103 are set in advance. When the virtual camera switching instruction is transmitted from the director, the virtual camera switching instruction is issued by designating the preset camera position of the virtual camera (the director 400 transmits the virtual camera switching instruction as described below).

As shown in fig. 1 and 3, the WebRTC push modules 102 send registration information of the registered virtual cameras to the streaming server 200.

As shown in fig. 1 and fig. 3, the director 400 is configured to obtain a camera list of the registered plurality of virtual cameras 103 from the director server 200, and perform signaling exchange with the plurality of WebRTC push modules 102 through the director server 200, so that the director 400 establishes P2P connection with the plurality of WebRTC push modules 102. Specifically, the multicast table 400 establishes P2P (peer-to-peer) connections over UDP (User Datagram Protocol UDP protocol) with the WebRTC push modules 102.

As shown in fig. 1 and 3, in the bootstrapping process, the bootstrapping station 400 requests virtual camera pictures of a plurality of virtual cameras to the bootstrapping client 100 through the bootstrapping server 200 according to an embodiment of the present invention.

Specifically, the director 400 requests the director server 200 for the virtual camera pictures of the plurality of virtual cameras, the director server 200 requests the WebRTC streaming module 102 of the director client 100 for the virtual camera pictures of the plurality of virtual cameras, and the WebRTC streaming module 102 requests the camera service unit 101 of the director client 100 for the virtual camera pictures of the plurality of virtual cameras.

As shown in fig. 1 and 2, the camera business unit 101 includes an acquisition module 1011 and a processing module 1012. And the acquisition module is used for acquiring the virtual picture of the virtual camera. And the processing module is used for processing the virtual picture of the virtual camera.

The virtual pictures of the virtual camera include audio of the set virtual camera and video of the virtual camera. The acquisition module comprises an audio acquisition module and a video acquisition module; and the audio acquisition module is used for acquiring the audio of the virtual camera. And the video acquisition module is used for acquiring the video of the virtual camera.

The processing module comprises an audio rendering module, an audio encoding module, a video rendering module and a video encoding module. And the audio rendering module is used for rendering the audio. And the audio coding module is used for coding the rendered audio. The video rendering module is used for rendering the video, and the video encoding module is used for encoding the rendered video.

In the present embodiment, two virtual cameras, namely, the first virtual camera 103 'and the second virtual camera 103', are exemplified. The multicast station 400 requests virtual camera pictures (audio/video) of two virtual cameras (a first virtual camera 103' and a second virtual camera 103 ") to the multicast client 100 through the multicast server 200.

As shown in fig. 1 and 3, the plurality of camera service units 101 perform virtual camera frames (audio/video) rendering on the plurality of virtual cameras at the same time, encode the virtual camera frames of the plurality of virtual cameras after rendering, and send the virtual camera frames of the virtual cameras after encoding to the corresponding WebRTC push modules 102.

As shown in fig. 1 and 3, the WebRTC push modules 102 transmit the virtual camera pictures of the virtual cameras 103 to the multicast station 400 using WebRTC protocol (Web Real-time communication (Web Real-Time Communication)) through the P2P connection established by the multicast station 400 with the WebRTC push modules 102.

The WebRTC protocol is a real-time communication solution proposed by google, can realize lower delay, adapts to resolution according to network conditions, and can conveniently support multi-platform equipment to play videos. Because the streaming function has high requirements on real-time response, the streaming client 100 developed by the invention adopts WebRTC protocol in terms of video streaming.

In the present invention, one multicast client 100 may render and encode virtual camera frames of multiple virtual cameras 103 (as shown in fig. 2, in this embodiment, a first virtual camera 103' and a second virtual camera 103″ are exemplified), and send the virtual camera frames of the multiple virtual cameras 103 after rendering and encoding to the multicast station 400.

Conventional director systems typically have a separate device for each virtual camera view. The guide client 100 developed by the invention fully plays the computing power of the computer display card, simultaneously renders the virtual camera pictures of the plurality of virtual cameras 103 on a guide client 100 in a background mode based on the consideration of the requirement of high real-time performance, and encodes the virtual camera pictures of the plurality of virtual cameras 103 into a common video in an H264 format through hardware and sends the common video to the guide platform 400.

In the practical requirement, the number of virtual camera pictures guided can be very large, the hardware of one guiding platform of the traditional guiding system can not be enough for rendering, the guiding client 100 is developed, the guiding client 100 is supported to be started on different devices, the plurality of guiding clients 100 process the plurality of virtual cameras 103 simultaneously, and finally, the virtual camera pictures of the plurality of virtual cameras 103 are sent to the guiding platform 400, so that the number of the virtual camera pictures guided is expanded to the greatest extent on the basis of reducing design and development work.

As shown in fig. 1 and 3, the director 400 receives virtual camera pictures of the plurality of virtual cameras 103, and the virtual camera pictures of the plurality of virtual cameras 103 are presented at the director 400. And the guiding player operates to select the virtual camera picture of the virtual camera to be presented to the user client 500 for virtual camera picture switching.

Specifically, the director 400 designates a preset position of the virtual camera, issues a virtual camera switching instruction, and transmits the virtual camera switching instruction to the director client 100 through the director server 200.

The plurality of camera service units 101 switch the virtual camera screen according to the virtual camera switching instruction transmitted from the multicast table 400, and forward (transmit) the virtual camera switching instruction to the logical server 300.

The logic server 300 broadcasts a virtual camera switching instruction to the user client 500. The user client 500 locally renders a camera screen corresponding to the virtual camera according to the virtual camera switching instruction.

When the virtual camera screen of the virtual camera 103 is not transmitted by the multicast table 400, but a virtual camera switching instruction is transmitted, that is, the multicast table 400 designates a preset position (a series of parameters such as a viewing angle, a position, a moving route, etc. of the virtual camera 103) of the virtual camera 103 to transmit the virtual camera switching instruction, and the user client 500 receives the virtual camera switching instruction and locally renders the camera screen corresponding to the virtual camera 103.

For example, the director 400 designates a preset camera position (a series of parameters such as a viewing angle, a position, a moving route, etc. of the first virtual camera 103 ') of the first virtual camera 103' to send a virtual camera switching instruction, and the user client 500 receives the virtual camera switching instruction and locally renders a camera screen of the first virtual camera 103 '.

According to the invention, the guide broadcasting station locally renders the camera picture of the virtual camera 103 at the user client 500 by sending the virtual camera switching instruction, so that the live broadcast flow is effectively saved.

In some embodiments, the lead station 400 includes a primary lead station and a plurality of secondary lead stations. The master multicast station and the plurality of slave multicast stations have different rights to transmit a virtual camera switching instruction to the plurality of camera service units 101 of the multicast client 100 through the multicast server 200. In this embodiment, the exemplary director station 400 is the master director station, and sends a virtual camera switching instruction. When the director station 400 includes a secondary director station, the authority of the primary director station to transmit the virtual camera switching instruction is higher than the authority of the secondary director station to transmit the virtual camera switching instruction. For example, in some embodiments, only the master multicast is allowed to send virtual camera switch instructions.

The multicast station 400 establishes a communication connection with the multicast server 200 based on the browser, and the multicast station 400 does not need to download the multicast station client. The multicast station 400 previews the virtual camera screens of the plurality of virtual cameras 103 based on the browser, and transmits a virtual camera switching instruction to the multicast client 100 through the multicast server 200 based on the browser. That is, the director 400 transmits a virtual camera switching instruction to the director server 200 based on the browser, and the director server 200 transmits a virtual camera switching instruction to the director client 100.

The multicast station 400 establishes communication connection with the multicast server 200 based on the browser, and realizes the division of work and cooperation of a plurality of multicast stations 400 (a main multicast station and a plurality of auxiliary multicast stations). Different director stations 400 are responsible for monitoring the virtual camera frames of different virtual cameras 103, and maximally expanding the number of director virtual camera frames.

In a preferred embodiment, the logic server 300 is configured with a recording function for recording virtual camera switch instructions for subsequent playback.

The invention simultaneously renders the virtual camera pictures of the plurality of virtual cameras 103 through the multicast client 100 and pushes the virtual camera pictures to the multicast platform 400 in a P2P mode. The guide platform 400 previews all virtual camera pictures, selects a virtual camera picture suitable for real-time switching as a current live broadcast virtual camera picture, broadcasts the current live broadcast virtual camera picture to all user clients 500 watching live broadcast through the logic server 300, and the user clients 500 locally render finally presented camera pictures according to the current virtual camera picture, so that the number of the guide virtual camera pictures is expanded to the greatest extent on the basis of reducing design and development work, and live broadcast flow is effectively saved.

As shown in fig. 4, according to an embodiment of the present invention, there is provided a virtual director method, including the following method steps:

in step S1, the multicast table 400 establishes P2P connections with the plurality of multicast clients 100.

The unicast client 100 at least comprises a plurality of virtual cameras 103, a plurality of camera service units 101 and a plurality of WebRTC push modules 102, wherein one virtual camera 103 corresponds to one camera service unit 101, and one camera service unit 101 corresponds to one WebRTC push module 102. The positions of the plurality of virtual cameras 103 are set in advance.

The plurality of camera service units 101 read the position parameters of the plurality of virtual cameras 102, and register the virtual cameras with the plurality of WebRTC push modules 102.

The WebRTC push modules transmit registration information of the registered virtual cameras 103 to the streaming server 200.

The director 400 acquires the camera list of the registered virtual cameras from the director server 200, and performs signaling exchange with the WebRTC push modules 102 through the director server 200, so that the director 400 establishes P2P connection with the WebRTC push modules 102.

Step S2, the director client 100 pushes the virtual camera images of the plurality of virtual cameras to the director 400.

Specifically, the multicast station 400 requests the multicast client 100 for virtual camera screens of a plurality of virtual cameras through the multicast server 200.

The plurality of camera service units 101 perform virtual camera picture rendering on the plurality of virtual cameras 103 at the same time, encode virtual camera pictures of the plurality of virtual cameras after rendering, and send the virtual camera pictures of the virtual cameras after encoding to the corresponding WebRTC push modules 102.

The WebRTC push modules 102 transmit the virtual camera pictures of the virtual cameras 103 to the director station 400 through the P2P connection established by the director station 400 and the WebRTC push modules 102.

Step S3, the director 400 sends a virtual camera switching instruction to the user client 500.

Specifically, the multicast station 400 transmits a virtual camera switching instruction to the multicast client 100 through the multicast server 200.

The plurality of camera service units 101 switch the virtual camera screen according to the virtual camera switching instruction sent by the multicast table 400, and forward the virtual camera switching instruction to the logic server 300. The logic server 300 broadcasts a virtual camera switching instruction to the user client 500.

Step S4, the user client 500 locally renders a camera frame corresponding to the virtual camera according to the virtual camera switching instruction.

While embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and not to be construed as limiting the invention, and that variations, modifications, alternatives and variations may be made to the above embodiments by one of ordinary skill in the art within the scope of the invention.

Claims

1. A virtual director system, the virtual director system comprising: a plurality of guide clients, a guide server, a guide table, a logic server and a plurality of user clients;

the guiding and broadcasting station is used for acquiring camera lists of a plurality of registered virtual cameras from the guiding and broadcasting server, and carrying out signaling exchange with the WebRTC push modules through the guiding and broadcasting server so that the guiding and broadcasting station and the WebRTC push modules are connected in a P2P mode.

2. The virtual director system of claim 1, wherein the positions of a plurality of the virtual cameras are preset.

3. The virtual multicast system according to claim 1, wherein the multicast station requests virtual camera pictures of a plurality of virtual cameras from the multicast client via the multicast server;

4. The virtual director system of claim 3, wherein the camera business unit comprises an acquisition module and a processing module;

5. The virtual director system of claim 4, wherein the acquisition module comprises an audio acquisition module and a video acquisition module; the virtual pictures of the virtual camera comprise audio of the set virtual camera and video of the virtual camera;

6. The virtual multicast system of claim 1, wherein the multicast station comprises a primary multicast station and a plurality of secondary multicast stations;

7. The virtual guide method is characterized by comprising the following method steps:

the guiding and broadcasting station acquires camera lists of a plurality of registered virtual cameras from a guiding and broadcasting server, and performs signaling exchange with a plurality of WebRTC plug flow modules through the guiding and broadcasting server, so that the guiding and broadcasting station and the WebRTC plug flow modules are connected in a P2P mode;

8. The virtual multicast method according to claim 7, wherein in step S2, the multicast station requests virtual camera pictures of a plurality of virtual cameras from the multicast client through the multicast server;

9. The virtual multicast method according to claim 7, wherein in step S3, the multicast station sends a virtual camera switching instruction to the multicast client through the multicast server;

10. The virtual director method of claim 7, wherein the positions of a plurality of the virtual cameras are preset.