CN115209172B - XR-based remote interactive performance method - Google Patents
XR-based remote interactive performance method Download PDFInfo
- Publication number
- CN115209172B CN115209172B CN202210819362.8A CN202210819362A CN115209172B CN 115209172 B CN115209172 B CN 115209172B CN 202210819362 A CN202210819362 A CN 202210819362A CN 115209172 B CN115209172 B CN 115209172B
- Authority
- CN
- China
- Prior art keywords
- main
- conference
- place
- sub
- conference site
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/2187—Live feed
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
- H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
- H04N21/43076—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen of the same content streams on multiple devices, e.g. when family members are watching the same movie on different devices
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering
- H04N21/4312—Generation of visual interfaces for content selection or interaction; Content or additional data rendering involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/438—Interfacing the downstream path of the transmission network originating from a server, e.g. retrieving MPEG packets from an IP network
- H04N21/4383—Accessing a communication channel
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
- H04N21/44012—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
- H04N21/44016—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving splicing one content stream with another content stream, e.g. for substituting a video clip
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
- H04N21/643—Communication protocols
- H04N21/6437—Real-time Transport Protocol [RTP]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention discloses a remote interactive performance method based on XR, which belongs to the field of content production and comprises the following steps: a1: manufacturing a virtual background; a2: planning an area to be spliced; a3: rendering a main meeting place and a sub meeting place XR; a4: shooting a real person virtual scene by fusing the virtual reality of a main meeting place and the virtual reality of an L screen; a5: mixing the picture of the conference site camera with the sound of the conference site singer, and then carrying out coding transmission; a6: the main meeting place receives the streams transmitted by the branch meeting places, and the pictures are spliced and synthesized after decoding; a7: and rendering, outputting and playing the spliced video after VR implantation. According to the invention, the interactive, wheat connection and interactive cross-space realistic performance of the players in the main meeting place are realized, the lens switching during XR performance is supported, the immersive experience of the spectator is increased while the distance between the literature performance and the remote spectators is shortened, and the cool effect is presented by the digital creative content.
Description
Technical Field
The invention relates to the field of content production, in particular to an XR-based remote interactive performance method.
Background
The existing video real-time links such as variety programs, news links, video conferences, live broadcast links and the like. Many of them are in the form of windows, like video conferences, for performance activities, fusion between people and scenes is not considered, the feeling of reality is not strong, and the immersive experience is poor.
XR (Extended reality) is a term used to describe the environment or interaction behavior that combines virtual and real elements. Although XR is generally considered to include AR (augmented reality), VR (virtual reality), MR (mixed reality), it is considered that in the field of content production: by utilizing platform technologies such as a camera tracking system, a virtual studio software platform, a media server, a real-time rendering engine and the like, a performer is placed in the virtual world in real time without a virtual manufacturing mode of a green curtain and a post-manufacturing process.
In XR production, a high quality LED screen (typically presented as a ring or globe) will simultaneously display a 3D virtual environment preloaded by the graphics engine, with the LEDs simultaneously constituting the background for film and television shooting or live events. When an accurate camera tracking system is added, the camera begins to move around the seamlessly integrated real and virtual elements within the LED environment, creating a fused immersive illusion to the viewer looking through the camera's view. The immersion experience is strong, but at present, XR cannot realize remote collaborative production like Lian Mai, and XR is widely used for recording programs at present and is difficult to switch a plurality of shots.
Disclosure of Invention
The invention aims to overcome the defects of the prior art, provides an XR-based remote interactive performance method, can realize multi-place linked cloud performance, does not need to be like a traditional broadcast television window connection line, can realize cross-space realistic performance of limb interaction, wheat connection and interaction with audiences of a performer in a main meeting place, supports lens switching during XR performance, increases the immersive experience of viewing and playing while pulling the distance between the literature performance and the remote audiences, and presents cool effects with digital creative content.
The invention aims at realizing the following scheme:
after the XR system is respectively built in the main conference place and the sub conference place, the interactive performance flow of the main conference place and the sub conference place is executed as follows:
step A1: creating a virtual background for background pictures of performances of players in the main meeting place and the sub meeting place;
step A2: planning an area to be spliced, and splitting the virtual background into two parts, wherein one part is a picture part displayed by a main conference site, and the other part is a picture displayed by a sub conference site L screen;
step A3: rendering the main conference site and the sub conference site XR, respectively rendering and mapping the virtual background split in the step A2 on a back screen of the main conference site and an L screen of the sub conference site, and keeping the picture synchronization of the main conference site and the sub conference site through time code and frame alignment;
step A4: the main conference site camera is used for carrying out virtual-real fusion shooting of the real person virtual scene of the main conference site, and the sub conference site camera is responsible for carrying out virtual-real fusion shooting of the real person virtual scene of the L screen;
step A5: mixing the picture of the conference site camera with the sound of the conference site singer, and then carrying out coding transmission;
step A6: the main meeting place receives the streams transmitted by the branch meeting places, decodes, and the integrated machine performs splicing and synthesizing of pictures to form complete pictures, and the pictures are monitored by the integrated machine;
step A7: and rendering and outputting the spliced video after VR implantation, pushing the video to the cloud by the integrated machine in an RTC mode, playing the video by a conference place player, displaying pictures by a conference place large screen, and playing audio by a plurality of sound boxes.
Further, the method also comprises the following steps of carrying out the wheat connecting production process of the main meeting place and the branch meeting place:
step B1: singing by the singer of the main conference site and the sub conference site;
step B2: a5, synthesizing sound and picture of the branch meeting place, and splicing the part with pictures of the main branch meeting place;
step B3: the main meeting place receives the video and audio streams transmitted from the branch meeting place, decodes the video and audio streams, and is responsible for audio synchronous splicing, outputting complete songs and carrying out audio and video synchronous processing by the integrated machine;
step B4: the subsequent transmission and playback steps are the same as the first part.
Further, the method also comprises the following audience interaction flow of the main meeting place and the branch meeting place:
step C1: shooting the audience of the branch meeting place by using a camera, collecting video and audio, encoding and transmitting to a main meeting place;
step C2: decoding the stream transmitted by C1 by the main conference site, performing XR processing on the video picture, rendering and outputting the video picture to an L-shaped screen consisting of a front large screen and a ground screen, and forming a naked eye 3D audience effect;
step C3: the audio obtained by decoding is sent to a main conference console, and singers hear the sound of the audience side of the sub conference through an ear return system;
step C4: the camera lens is switched to face the front large screen, and the main conference place singer and the sub conference place audience on the front large screen are shot together to form the singing effect of the main conference place singer in the sub conference place audience;
step C5: the singer voice of the main conference place is collected through the microphone, and the picture and the voice shot by the camera are synchronized through the guide and broadcasting integrated machine.
Step C6: the sound is transmitted in an RTC mode, the branch conference place player decodes and plays, the large screen displays singing effect of a conference place singer in a branch conference place audience, and the branch conference place sound box plays main conference place sound, so that virtual picture interaction and sound bidirectional interaction are realized.
Further, in step A4, the method includes the sub-steps of: the camera position angle information of the shooting camera of the main conference place is used as the reference, and the camera position information of the sub conference place camera is consistent with the camera position information of the main conference place camera.
Further, in step A5, the method includes the sub-steps of: the camera shooting virtual scene performance of the branch conference site is kept consistent with the shooting machine position of the main conference site, if the switching between the shots is involved, the machine position of the main conference site is firstly tangential to only shoot the shots of actors of the main conference site, and at the moment, the shots of the branch conference site are kept consistent with the shots to be cut next to the main conference site.
Further, in step A6, the complete picture is a main meeting place actor and a sub-meeting place actor and a complete background.
Further, in step A7, the method includes the sub-steps of: and synthesizing the video of the main meeting place and the video transmitted by the branch meeting place to form the effect that the different-place actors perform in the same virtual scene.
Further, in step B1, the method includes the sub-steps of: a complete song can be divided into segments in advance, and singers in a main conference place sing the segments respectively; the singer in the main conference wears the ear back to monitor accompaniment and own sound.
Further, in step B3, the complete song is an accompaniment and main meeting place singer and a sub-meeting place singer.
Further, in step C3, the singer hears sounds on the audience side of the conference room including shouting, cheering, and applause through the otor system.
The beneficial effects of the invention include:
according to the embodiment of the invention, two sets of XR systems are built at the main conference place and the sub conference place, the depth fusion of different surface performers, live performers and performance scenes is realized by adopting a splicing mode, and different from the traditional connecting line of the open window, the realization that the performers at the main conference place can perform lifelike performances in a cross-space way through limb interaction, wheat connection and audience interaction is realized, and the lens switching during XR performance is supported.
According to the embodiment of the invention, the interaction method of the main conference place performer and the sub conference place audience is innovated, the sub conference place audience can feel that the main conference place performer enters the audience seat from the screen, the distance between the main conference place performer and the different-place audience is shortened, and the immersive experience of the sub conference place is enhanced.
The embodiment of the invention provides a remote interactive performance method based on XR, which can realize multi-place linked cloud performance, does not need to be like a traditional broadcast television window connection line, but can realize the cross-space lifelike performance of limb interaction, wheat connection and audience interaction of performers in a main conference room, supports lens switching during the XR performance, increases the immersive experience of viewing and playing while shortening the distance between the literature performance and the remote audience, and presents cool effects with digital creative content.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions of the prior art, the drawings which are used in the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the description below are only some embodiments of the invention, and that other drawings can be obtained according to these drawings without inventive faculty for a person skilled in the art.
FIG. 1 is a schematic illustration of an interactive performance of a main conference room according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of audience interaction at a main conference site in an embodiment of the present invention.
Detailed Description
All of the features disclosed in all of the embodiments of this specification, or all of the steps in any method or process disclosed implicitly, except for the mutually exclusive features and/or steps, may be combined and/or expanded and substituted in any way.
In a specific implementation manner, as one of the embodiments of the present invention, the interactive performance process includes a main conference site and a sub conference site: the interactive performance of the main meeting place in the scene is the interaction facing the performer, and comprises the fusion of pictures and the linking of sound. The main meeting place is provided with a back screen and a forward screen, wherein the back screen comprises two side screens, a back screen and a ground screen, and the forward screen comprises a front large screen; one LED large screen is arranged at the meeting place, and one L screen, one back screen and one ground screen are adopted for virtual scene performance at the meeting place. And splicing pictures of the main conference place and the sub conference place. The method specifically comprises the following steps:
step A1: and (3) making a virtual background for a background picture of the performance of the main meeting place performer.
Step A2: and planning an area to be spliced, and splitting the virtual background into two parts, wherein one part is a picture part displayed by a main conference site and the other part is a picture displayed by a sub conference site L screen. The L-shaped screen is utilized, a certain 3D effect is realized through XR, the integrated feeling of picture synthesis of a main conference site is enhanced, compared with a common single-sided screen, a common 2D picture is taken by a camera, and the problem of obvious depth inconsistency is solved when the picture is embedded into the background of the main conference site.
Step A3: and C, rendering the main and sub-conference sites XR, respectively rendering and mapping the virtual background split in the step A2 on a back screen of the main conference site and a sub-conference site L screen, and keeping the picture synchronization of the main and sub-conference sites through time code and frame alignment.
Step A4: the main conference site camera is used for carrying out virtual-real fusion shooting (real person and virtual scene), and the sub conference site camera is used for carrying out L-screen virtual-real fusion shooting (real person and virtual scene).
Step A5: and mixing the picture of the conference site camera with the sound of the conference site singer, and then carrying out coding transmission.
In the step, the camera shooting the virtual scene performance of the sub-conference site is kept consistent with the camera position of the main conference site so as to solve the problem that parallax can occur after splicing, if the switching between the lenses is involved, as the camera is kept synchronous for a certain time, considering the requirement in live broadcasting, the embodiment of the invention firstly selects to firstly tangentially shoot the camera position of the main conference site only with the lenses of the actors of the main conference site, and at the moment, the lenses of the sub-conference site are kept consistent with the next lenses to be cut of the main conference site.
Step A6: the main meeting place receives the stream from the branch meeting place, decodes, and the integrated machine performs picture splicing and synthesizing to form a complete picture (main meeting place actor + branch meeting place actor + complete background), and the picture is monitored by the integrated machine.
Step A7: and rendering and outputting the spliced video after VR implantation, pushing the video to the cloud by an integrated machine in an RTC mode, playing the video by a branch conference site player, displaying pictures by a large screen of the branch conference site, playing audio by a plurality of sound boxes, and combining the video of the main conference site with the video transmitted by the branch conference site to form the effect that different-place actors perform in the same virtual scene.
In a specific embodiment, as another embodiment of the present invention, the wheat connecting manufacturing process including the main meeting place and the branch meeting place specifically includes the following steps:
step B1: singing by the singer of the main conference site and the sub conference site; in this step, a complete song is divided into segments in advance, and singers in the main conference place sing the segments of the song respectively. Alternatively, the singer in the main conference wears the ear-returns, and can monitor accompaniment and own sound.
Step B2: and A5, synthesizing sound and picture of the branch conference, and splicing the sound and picture of the branch conference with pictures of the main conference.
Step B3: the main meeting place receives the video and audio streams transmitted by the sub meeting place, decodes, and is responsible for audio synchronous splicing by the all-in-one machine, outputs complete songs (accompaniment + main meeting place singer + sub meeting place singer) and performs audio and video synchronous processing;
step B4: the subsequent transmission and playback steps are the same as the first part.
Example 3
In a specific embodiment, as another embodiment of the present invention, the method includes a main conference site and a sub conference site audience interaction process, which specifically includes the following steps:
step C1: shooting the audience of the branch meeting place by using a camera, collecting video and audio, encoding and transmitting to a main meeting place;
step C2: decoding the stream transmitted by the S1 by the main conference site, performing XR processing on the video picture, rendering and outputting the video picture to an L-shaped screen consisting of a front large screen and a ground screen, and forming a naked eye 3D audience effect;
step C3: the audio obtained by decoding is sent to a main conference sound console, and singer A can hear the sound (shouting, cheering, applause, etc.) of the audience side of the branch conference through an otoback system;
step C4: the camera lens is switched to face the front large screen, and the main conference place singer and the sub conference place audience on the front large screen are shot together to form the singing effect of the main conference place singer in the sub conference place audience;
step C5: the singer voice of the main conference place is collected through the microphone, and the picture and the voice shot by the camera X are synchronized through the guide and broadcasting integrated machine.
Step C6: the sound box is used for playing the sound of the main conference site, so that virtual picture interaction and sound bidirectional interaction are realized.
Example 1
After the XR system is respectively built in the main conference place and the sub conference place, the interactive performance flow of the main conference place and the sub conference place is executed as follows:
step A1: creating a virtual background for background pictures of performances of players in the main meeting place and the sub meeting place;
step A2: planning an area to be spliced, and splitting the virtual background into two parts, wherein one part is a picture part displayed by a main conference site, and the other part is a picture displayed by a sub conference site L screen;
step A3: rendering the main conference site and the sub conference site XR, respectively rendering and mapping the virtual background split in the step A2 on a back screen of the main conference site and an L screen of the sub conference site, and keeping the picture synchronization of the main conference site and the sub conference site through time code and frame alignment;
step A4: the main conference site camera is used for carrying out virtual-real fusion shooting of the real person virtual scene of the main conference site, and the sub conference site camera is responsible for carrying out virtual-real fusion shooting of the real person virtual scene of the L screen;
step A5: mixing the picture of the conference site camera with the sound of the conference site singer, and then carrying out coding transmission;
step A6: the main meeting place receives the streams transmitted by the branch meeting places, decodes, and the integrated machine performs splicing and synthesizing of pictures to form complete pictures, and the pictures are monitored by the integrated machine;
step A7: and rendering and outputting the spliced video after VR implantation, pushing the video to the cloud by the integrated machine in an RTC mode, playing the video by a conference place player, displaying pictures by a conference place large screen, and playing audio by a plurality of sound boxes.
Example 2
On the basis of the embodiment 1, the method further comprises the following steps of carrying out the wheat connecting production flow of the main meeting place and the branch meeting place:
step B1: singing by the singer of the main conference site and the sub conference site;
step B2: a5, synthesizing sound and picture of the branch meeting place, and splicing the part with pictures of the main branch meeting place;
step B3: the main meeting place receives the video and audio streams transmitted from the branch meeting place, decodes the video and audio streams, and is responsible for audio synchronous splicing, outputting complete songs and carrying out audio and video synchronous processing by the integrated machine;
step B4: the subsequent transmission and playback steps are the same as the first part.
Example 3
Based on the embodiment 1, the method further comprises the following audience interaction flow of the main meeting place and the branch meeting place:
step C1: shooting the audience of the branch meeting place by using a camera, collecting video and audio, encoding and transmitting to a main meeting place;
step C2: decoding the stream transmitted by C1 by the main conference site, performing XR processing on the video picture, rendering and outputting the video picture to an L-shaped screen consisting of a front large screen and a ground screen, and forming a naked eye 3D audience effect;
step C3: the audio obtained by decoding is sent to a main conference console, and singers hear the sound of the audience side of the sub conference through an ear return system;
step C4: the camera lens is switched to face the front large screen, and the main conference place singer and the sub conference place audience on the front large screen are shot together to form the singing effect of the main conference place singer in the sub conference place audience;
step C5: the singer voice of the main conference place is collected through the microphone, and the picture and the voice shot by the camera are synchronized through the guide and broadcasting integrated machine.
Step C6: the sound is transmitted in an RTC mode, the branch conference place player decodes and plays, the large screen displays singing effect of a conference place singer in a branch conference place audience, and the branch conference place sound box plays main conference place sound, so that virtual picture interaction and sound bidirectional interaction are realized.
Example 4
On the basis of example 1, in step A4, the sub-steps are included: the camera position angle information of the shooting camera of the main conference place is used as the reference, and the camera position information of the sub conference place camera is consistent with the camera position information of the main conference place camera.
Example 5
On the basis of example 1, in step A5, the sub-steps are included: the camera shooting virtual scene performance of the branch conference site is kept consistent with the shooting machine position of the main conference site, if the switching between the shots is involved, the machine position of the main conference site is firstly tangential to only shoot the shots of actors of the main conference site, and at the moment, the shots of the branch conference site are kept consistent with the shots to be cut next to the main conference site.
Example 6
On the basis of embodiment 1, in step A6, the complete picture is a main conference actor and a sub conference actor and a complete background.
Example 7
On the basis of example 1, in step A7, the sub-steps are included: and synthesizing the video of the main meeting place and the video transmitted by the branch meeting place to form the effect that the different-place actors perform in the same virtual scene.
Example 8
On the basis of example 2, in step B1, the sub-steps are included: a complete song can be divided into segments in advance, and singers in a main conference place sing the segments respectively; the singer in the main conference wears the ear back to monitor accompaniment and own sound.
Example 9
On the basis of embodiment 2, in step B3, the complete song is an accompaniment and main meeting place singer and a sub-meeting place singer.
Example 10
On the basis of embodiment 2, in step C3, the singer hears the sound on the audience side of the conference room including the shout, cheering, and applause through the ear-return system.
The invention is not related in part to the same as or can be practiced with the prior art.
The foregoing technical solution is only one embodiment of the present invention, and various modifications and variations can be easily made by those skilled in the art based on the application methods and principles disclosed in the present invention, not limited to the methods described in the foregoing specific embodiments of the present invention, so that the foregoing description is only preferred and not in a limiting sense.
In addition to the foregoing examples, those skilled in the art will recognize from the foregoing disclosure that other embodiments can be made and in which various features of the embodiments can be interchanged or substituted, and that such modifications and changes can be made without departing from the spirit and scope of the invention as defined in the appended claims.
Claims (10)
1. The XR-based remote interactive performance method is characterized in that after an XR system is respectively built in a main conference place and a sub conference place, the following interactive performance flow of the main conference place and the sub conference place is executed:
step A1: creating a virtual background for background pictures of performances of players in the main meeting place and the sub meeting place;
step A2: planning an area to be spliced, and splitting the virtual background into two parts, wherein one part is a picture part displayed by a main conference site, and the other part is a picture displayed by a sub conference site L screen;
step A3: rendering the main conference site and the sub conference site XR, respectively rendering and mapping the virtual background split in the step A2 on a back screen of the main conference site and an L screen of the sub conference site, and keeping the picture synchronization of the main conference site and the sub conference site through time code and frame alignment;
step A4: the main conference site camera is used for carrying out virtual-real fusion shooting of the real person virtual scene of the main conference site, and the sub conference site camera is responsible for carrying out virtual-real fusion shooting of the real person virtual scene of the L screen;
step A5: mixing the picture of the conference site camera with the sound of the conference site singer, and then carrying out coding transmission;
step A6: the main meeting place receives the streams transmitted by the branch meeting places, decodes, and the integrated machine performs splicing and synthesizing of pictures to form complete pictures, and the pictures are monitored by the integrated machine;
step A7: and rendering and outputting the spliced video after VR implantation, pushing the video to the cloud by the integrated machine in an RTC mode, playing the video by a conference place player, displaying pictures by a conference place large screen, and playing audio by a plurality of sound boxes.
2. The XR-based remote interactive performance method of claim 1, further comprising performing a post-meeting production process for a main meeting place, a sub-meeting place, as follows:
step B1: singing by the singer of the main conference site and the sub conference site;
step B2: a5, synthesizing sound and picture of the branch meeting place, and splicing the part with pictures of the main branch meeting place;
step B3: the main meeting place receives the video and audio streams transmitted from the branch meeting place, decodes the video and audio streams, and is responsible for audio synchronous splicing, outputting complete songs and carrying out audio and video synchronous processing by the integrated machine;
step B4: the subsequent transmission and playback steps are the same as the first part.
3. The XR-based remote interactive performance method of claim 1, further comprising performing a spectator interaction process at the main venue and the sub venue as follows:
step C1: shooting the audience of the branch meeting place by using a camera, collecting video and audio, encoding and transmitting to a main meeting place;
step C2: decoding the stream transmitted by C1 by the main conference site, performing XR processing on the video picture, rendering and outputting the video picture to an L-shaped screen consisting of a front large screen and a ground screen, and forming a naked eye 3D audience effect;
step C3: the audio obtained by decoding is sent to a main conference console, and singers hear the sound of the audience side of the sub conference through an ear return system;
step C4: the camera lens is switched to face the front large screen, and the main conference place singer and the sub conference place audience on the front large screen are shot together to form the singing effect of the main conference place singer in the sub conference place audience;
step C5: collecting singer sounds of a main conference place through a microphone, and synchronizing pictures and sounds shot by a camera through a guide and broadcast integrated machine;
step C6: the sound is transmitted in an RTC mode, the branch conference place player decodes and plays, the large screen displays singing effect of a conference place singer in a branch conference place audience, and the branch conference place sound box plays main conference place sound, so that virtual picture interaction and sound bidirectional interaction are realized.
4. The XR-based remote interactive performance method of claim 1, comprising, in step A4, the sub-steps of: the camera position angle information of the shooting camera of the main conference place is used as the reference, and the camera position information of the sub conference place camera is consistent with the camera position information of the main conference place camera.
5. The XR-based remote interactive performance method of claim 1, comprising, in step A5, the sub-steps of: the camera shooting virtual scene performance of the branch conference site is kept consistent with the shooting machine position of the main conference site, if the switching between the shots is involved, the machine position of the main conference site is firstly tangential to only shoot the shots of actors of the main conference site, and at the moment, the shots of the branch conference site are kept consistent with the shots to be cut next to the main conference site.
6. The XR-based remote interactive performance method of claim 1, wherein in step A6, the complete picture is a main and a sub-venue actor and a complete background.
7. The XR-based remote interactive performance method of claim 1, comprising, in step A7, the sub-steps of: and synthesizing the video of the main meeting place and the video transmitted by the branch meeting place to form the effect that the different-place actors perform in the same virtual scene.
8. The XR-based remote interactive performance method of claim 2, comprising, in step B1, the sub-steps of: a complete song can be divided into segments in advance, and singers in a main conference place sing the segments respectively; the singer in the main conference wears the ear back to monitor accompaniment and own sound.
9. The XR-based remote interactive performance method of claim 2, wherein in step B3, the complete song is an accompaniment and main and sub-session singers.
10. The XR-based remote interactive performance method of claim 2, wherein in step C3, the singer hears the conference room audience side sounds through an ear-return system including shouting, cheering, and applause.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210819362.8A CN115209172B (en) | 2022-07-13 | 2022-07-13 | XR-based remote interactive performance method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210819362.8A CN115209172B (en) | 2022-07-13 | 2022-07-13 | XR-based remote interactive performance method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115209172A CN115209172A (en) | 2022-10-18 |
| CN115209172B true CN115209172B (en) | 2023-07-07 |
Family
ID=83579608
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210819362.8A Active CN115209172B (en) | 2022-07-13 | 2022-07-13 | XR-based remote interactive performance method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115209172B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115439587B (en) * | 2022-11-08 | 2023-02-14 | 成都索贝数码科技股份有限公司 | 2.5D rendering method based on object visual range |
| CN115657862B (en) * | 2022-12-27 | 2023-03-14 | 海马云（天津）信息技术有限公司 | Method, device, storage medium and equipment for automatically switching virtual KTV scene pictures |
| CN115802165B (en) * | 2023-02-10 | 2023-05-12 | 成都索贝数码科技股份有限公司 | Lens moving shooting method applied to live broadcast connection of different places and same scene |
| CN116931737A (en) * | 2023-08-03 | 2023-10-24 | 重庆康建光电科技有限公司 | System and method for realizing virtual reality interaction between person and scene |
| CN116781958B (en) * | 2023-08-18 | 2023-11-07 | 成都索贝数码科技股份有限公司 | XR-based multi-machine-position presentation system and method |
| CN117149016B (en) * | 2023-10-26 | 2024-01-30 | 锋尚文化集团股份有限公司 | Virtual object control method, device and system |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102170361A (en) * | 2011-03-16 | 2011-08-31 | 西安电子科技大学 | Virtual-reality-based network conference method |
| CN104702936A (en) * | 2015-03-31 | 2015-06-10 | 王子强 | Virtual reality interaction method based on glasses-free 3D display |
| CN106210703A (en) * | 2016-09-08 | 2016-12-07 | 北京美吉克科技发展有限公司 | The utilization of VR environment bust shot camera lens and display packing and system |
| CN106789991A (en) * | 2016-12-09 | 2017-05-31 | 福建星网视易信息系统有限公司 | A kind of multi-person interactive method and system based on virtual scene |
| CN111447460A (en) * | 2020-05-15 | 2020-07-24 | 杭州当虹科技股份有限公司 | Method for applying low-delay network to broadcasting station |
| CN112135158A (en) * | 2020-09-17 | 2020-12-25 | 重庆虚拟实境科技有限公司 | Live broadcasting method based on mixed reality and related equipment |
| CN112492231A (en) * | 2020-11-02 | 2021-03-12 | 重庆创通联智物联网有限公司 | Remote interaction method, device, electronic equipment and computer readable storage medium |
| US11100695B1 (en) * | 2020-03-13 | 2021-08-24 | Verizon Patent And Licensing Inc. | Methods and systems for creating an immersive character interaction experience |
| CN114401414A (en) * | 2021-12-27 | 2022-04-26 | 北京达佳互联信息技术有限公司 | Immersive live broadcast information display method and system and information push method |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160330408A1 (en) * | 2015-04-13 | 2016-11-10 | Filippo Costanzo | Method for progressive generation, storage and delivery of synthesized view transitions in multiple viewpoints interactive fruition environments |
| US10339715B2 (en) * | 2016-08-31 | 2019-07-02 | Factualvr, Inc. | Virtual reality system |
-
2022
- 2022-07-13 CN CN202210819362.8A patent/CN115209172B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102170361A (en) * | 2011-03-16 | 2011-08-31 | 西安电子科技大学 | Virtual-reality-based network conference method |
| CN104702936A (en) * | 2015-03-31 | 2015-06-10 | 王子强 | Virtual reality interaction method based on glasses-free 3D display |
| CN106210703A (en) * | 2016-09-08 | 2016-12-07 | 北京美吉克科技发展有限公司 | The utilization of VR environment bust shot camera lens and display packing and system |
| CN106789991A (en) * | 2016-12-09 | 2017-05-31 | 福建星网视易信息系统有限公司 | A kind of multi-person interactive method and system based on virtual scene |
| US11100695B1 (en) * | 2020-03-13 | 2021-08-24 | Verizon Patent And Licensing Inc. | Methods and systems for creating an immersive character interaction experience |
| CN111447460A (en) * | 2020-05-15 | 2020-07-24 | 杭州当虹科技股份有限公司 | Method for applying low-delay network to broadcasting station |
| CN112135158A (en) * | 2020-09-17 | 2020-12-25 | 重庆虚拟实境科技有限公司 | Live broadcasting method based on mixed reality and related equipment |
| CN112492231A (en) * | 2020-11-02 | 2021-03-12 | 重庆创通联智物联网有限公司 | Remote interaction method, device, electronic equipment and computer readable storage medium |
| CN114401414A (en) * | 2021-12-27 | 2022-04-26 | 北京达佳互联信息技术有限公司 | Immersive live broadcast information display method and system and information push method |
Non-Patent Citations (2)
| Title |
|---|
| Design of Broadcasting and Hosting Information Guiding Platform based on Virtual Environment Data Modeling with Mixed Reality System;Shanshan Yin;2022 6th International Conference on Trends in Electronics and Informatics (ICOEI);全文 * |
| XR智能演播室系统建设与应用解析;沈锦昌;广播电视信息(第4期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115209172A (en) | 2022-10-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115209172B (en) | XR-based remote interactive performance method | |
| CN106792246B (en) | Method and system for interaction of fusion type virtual scene | |
| US10531158B2 (en) | Multi-source video navigation | |
| US8289367B2 (en) | Conferencing and stage display of distributed conference participants | |
| CN210021183U (en) | Immersive interactive panoramic holographic theater and performance system | |
| US8291328B2 (en) | System and method for synchronizing a real-time performance with a virtual object | |
| Schreer et al. | Ultrahigh-resolution panoramic imaging for format-agnostic video production | |
| JP2010541415A (en) | Compositing multimedia event presentations | |
| US20110304735A1 (en) | Method for Producing a Live Interactive Visual Immersion Entertainment Show | |
| KR20180052496A (en) | Virtual performance-studio system | |
| KR20130106483A (en) | Physical picture machine | |
| JP2006041886A (en) | Information processor and method, recording medium, and program | |
| KR20180052494A (en) | Conference system for big lecture room | |
| US20090153550A1 (en) | Virtual object rendering system and method | |
| CN104202589A (en) | Multichannel three-dimensional film video synchronous playing method | |
| Williams et al. | A distributed theatre experiment with shakespeare | |
| KR20090000550A (en) | Methode of a cyber public performance on stage with 3d display | |
| Grau et al. | 3D-TV R&D activities in europe | |
| KR102273439B1 (en) | Multi-screen playing system and method of providing real-time relay service | |
| CN113259544A (en) | Remote interactive holographic demonstration system and method | |
| Koide et al. | Development of high-resolution virtual reality system by projecting to large cylindrical screen | |
| CN114079799A (en) | Music live broadcast system and method based on virtual reality | |
| De Wispelaere et al. | Multicam live production in a virtual environment | |
| Malsky | The Grandeur (s) of CinemaScope: Early Experiments in Cinematic Stereophony | |
| US20220343951A1 (en) | Method and apparatus for production of a real-time virtual concert or collaborative online event |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |