CN113115110A - Video synthesis method and device, storage medium and electronic equipment - Google Patents
Video synthesis method and device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN113115110A CN113115110A CN202110551884.XA CN202110551884A CN113115110A CN 113115110 A CN113115110 A CN 113115110A CN 202110551884 A CN202110551884 A CN 202110551884A CN 113115110 A CN113115110 A CN 113115110A
- Authority
- CN
- China
- Prior art keywords
- video
- real
- stop
- time
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000001308 synthesis method Methods 0.000 title abstract description 10
- 238000000034 method Methods 0.000 claims abstract description 48
- 230000015572 biosynthetic process Effects 0.000 claims abstract description 40
- 238000003786 synthesis reaction Methods 0.000 claims abstract description 39
- 230000002194 synthesizing effect Effects 0.000 claims abstract description 23
- 238000006748 scratching Methods 0.000 claims abstract description 6
- 230000002393 scratching effect Effects 0.000 claims abstract description 6
- 239000002131 composite material Substances 0.000 claims description 47
- 239000000284 extract Substances 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 2
- 230000000694 effects Effects 0.000 abstract description 12
- 230000000007 visual effect Effects 0.000 abstract description 6
- 230000008014 freezing Effects 0.000 abstract description 2
- 238000007710 freezing Methods 0.000 abstract description 2
- 239000000203 mixture Substances 0.000 description 35
- 238000012545 processing Methods 0.000 description 12
- 230000008569 process Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 230000009286 beneficial effect Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000012790 confirmation Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000001427 coherent effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/44016—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving splicing one content stream with another content stream, e.g. for substituting a video clip
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
- H04N21/440245—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/472—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
- H04N21/47205—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content for manipulating displayed content, e.g. interacting with MPEG-4 objects, editing locally
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Databases & Information Systems (AREA)
- Human Computer Interaction (AREA)
- Studio Devices (AREA)
- Processing Or Creating Images (AREA)
Abstract
The disclosure provides a video synthesis method, a video synthesis device, a storage medium and electronic equipment, and relates to the technical field of computers. The video synthesis method comprises the following steps: acquiring a real-time video, and scratching a target real-time video of a target object from the real-time video; synthesizing the target real-time video of the target object with the target virtual scene to obtain a real-time synthesized video; acquiring a stop motion image sequence shot by a plurality of entity cameras by referring to the real-time synthesized video, and matting an image sequence of a target object from the stop motion image sequence, wherein the stop motion image sequence is composed of images shot by the plurality of entity cameras; and synthesizing the image sequence of the target object with the target virtual scene to obtain a stop-motion synthesized video. The method and the device have the advantages that the real-time freezing effect of the target object in the virtual scene is realized, the problem of limited scene is avoided, and the more novel visual experience is brought to a user.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a video synthesis method, a video synthesis apparatus, a computer-readable storage medium, and an electronic device.
Background
With the continuous development of computer technology, the requirements for the effect presented by the video are higher and higher. To meet the novel visual experience, various video special effects approaches are coming up in succession, for example: bullet time, which is a simulated variable speed effect used in movies, advertisements, or games, can achieve a time-stop effect.
In the related art, a live-action shot picture is usually adopted, and the time freezing effect is achieved through later-stage software synthesis, and the live-action shot has high requirements on a scene, cannot overcome the limitation of the scene, and is difficult to meet various requirements of users.
It is to be noted that the information disclosed in the above background section is only for enhancement of understanding of the background of the present disclosure, and thus may include information that does not constitute prior art known to those of ordinary skill in the art.
Disclosure of Invention
The disclosure provides a video synthesis method, a video synthesis device, a computer readable storage medium and an electronic device, which aim to solve the problem of real-time freeze of a target object in a virtual scene, further avoid the problem of limited scene at least to a certain extent, and bring a more novel visual experience to a user.
Additional features and advantages of the disclosure will be set forth in the detailed description which follows, or in part will be obvious from the description, or may be learned by practice of the disclosure.
According to a first aspect of the present disclosure, there is provided a video composition method, including: acquiring a real-time video, and scratching a target real-time video of a target object from the real-time video; synthesizing the target real-time video of the target object with the target virtual scene to obtain a real-time synthesized video; acquiring a stop motion image sequence shot by a plurality of entity cameras by referring to the real-time synthesized video, and matting an image sequence of a target object from the stop motion image sequence, wherein the stop motion image sequence is composed of images shot by the plurality of entity cameras; and synthesizing the image sequence of the target object with the target virtual scene to obtain a stop-motion synthesized video.
In an exemplary embodiment of the present disclosure, the method further comprises: and sending the real-time composite video and the stop-motion composite video to a display device so as to display the real-time composite video and the stop-motion composite video.
In an exemplary embodiment of the present disclosure, the transmitting the real-time composite video and the stop-motion composite video to the display device includes: sending the real-time synthesized video to a first display device; and sending the stop-motion synthesized video to a second display device.
In an exemplary embodiment of the present disclosure, the method further comprises: and sending the light signal to a light controller, so that the light controller controls the light of the scene by adopting the light signal.
In an exemplary embodiment of the present disclosure, the light signal includes address information of light, and the light controller turns on the corresponding light according to the address information.
In an exemplary embodiment of the present disclosure, the method further comprises: and determining the sequence of the image arrangement in the stop-motion image sequence according to the position relation of the plurality of entity cameras.
In an exemplary embodiment of the present disclosure, the method further comprises: and outputting the stop-motion synthesized video as a video file.
In an exemplary embodiment of the present disclosure, the synthesizing the image sequence of the target object and the target virtual scene to obtain a stop-motion composite video includes: acquiring an image sequence of the target virtual scene; the image sequence of the target virtual scene is obtained by shooting the target virtual scene for multiple times by the virtual cameras according to the corresponding pose of each entity camera; and correspondingly synthesizing the image sequence of the target object and the images in the image sequence of the target virtual scene to obtain the stop-motion synthesized video.
According to a second aspect of the present disclosure, there is provided a video compositing apparatus comprising: the first acquisition module is used for acquiring a real-time video and scratching a target real-time video of a target object from the real-time video; the first synthesis module is used for synthesizing a target real-time video of the target object with a target virtual scene to obtain a real-time synthesized video; a second acquisition module, configured to acquire a sequence of stop motion images captured by the plurality of physical cameras with reference to the real-time composite video, and extract an image sequence of a target object from the sequence of stop motion images; and the second synthesis module is used for synthesizing the image sequence of the target object with the target virtual scene to obtain a stop-motion synthesis video.
In an exemplary embodiment of the disclosure, the first synthesis module is further configured to send a scene identifier of the target virtual scene to a second synthesis module, so that the second synthesis module determines the target virtual scene according to the scene identifier.
According to a third aspect of the present disclosure, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the above-described video composition method.
According to a fourth aspect of the present disclosure, there is provided an electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the above-described video compositing method via execution of the executable instructions.
The technical scheme of the disclosure has the following beneficial effects:
in the video synthesis method, a target real-time video of the target object is synthesized with a target virtual scene to obtain a real-time synthesized video, and an image sequence of the target object is synthesized with the target virtual scene to obtain a stop-motion synthesized video. On one hand, the pose information of the target object in the virtual scene can be previewed in real time, so that a user can determine a freeze frame picture required to be synthesized, a required freeze frame synthesized video is synthesized, and a more novel visual experience is brought to the user. On the other hand, the virtual scene is adopted for synthesis, which is beneficial to the selection and switching of scenes, and can avoid the limitation of scenes, so that a user can select a proper target virtual scene according to the requirement, and further meet the diversified requirements of the user.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure. It is apparent that the drawings in the following description are only some embodiments of the present disclosure, and that other drawings can be obtained from those drawings without inventive effort for a person skilled in the art.
FIG. 1 shows a system architecture diagram of an operating environment in the present exemplary embodiment;
fig. 2 shows a flowchart of a video composition method in the present exemplary embodiment;
FIG. 3 illustrates a flow diagram for composing a stop motion composite video in the present exemplary embodiment;
fig. 4 is a diagram showing an example of a video composition method in the present exemplary embodiment;
fig. 5 is a block diagram showing the configuration of a video compositing apparatus in the present exemplary embodiment;
fig. 6 shows an electronic device for implementing the above method in the present exemplary embodiment.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the subject matter of the present disclosure can be practiced without one or more of the specific details, or with other methods, components, devices, steps, and the like. In other instances, well-known technical solutions have not been shown or described in detail to avoid obscuring aspects of the present disclosure.
Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repetitive description will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.
Herein, "first", "second", etc. are labels for specific objects, and do not limit the number or order of the objects.
The live-action shooting methods used in the related art are difficult to satisfy various requirements of users, for example, scenes that the users want to synthesize cannot be realized through construction, the users want to change scenes, and the like, and a video synthesis method for acquiring a target object in a virtual scene is lacked to realize a time-stop special effect.
In view of one or more of the above problems, exemplary embodiments of the present disclosure provide a video composition method and a video composition apparatus, which may be applied to a special effect scene such as a movie, an advertisement, or a game.
First, the system architecture of the video composition method and the video composition apparatus operating environment will be described. Referring to fig. 1, the system architecture 100 may include: a video camera 110, a composition engine 120, a plurality of physical cameras 130. The camera 110 is configured to capture a target object, obtain a real-time video, and send the real-time video to the composition engine 120 for processing. The plurality of entity cameras 130 are configured to photograph the target object to obtain a stop-motion image sequence, and send the stop-motion image sequence to the composition engine 120 for processing, where the number of entity cameras in fig. 1 is merely illustrative, and any number of entity cameras may be set according to implementation requirements. The composition engine 120 is mainly used to perform matting and composition on the real-time video to obtain a real-time composite video, and perform matting and composition on the stop-motion image sequence to obtain a stop-motion composite video, which may be a device with data processing capability such as a desktop computer and a notebook computer.
The video synthesis method according to the present exemplary embodiment will be specifically described below with reference to fig. 2. It should be noted that the video composition method provided by the present exemplary embodiment may be executed by the composition engine 130.
Fig. 2 shows a schematic flow of the video synthesizing method in the present exemplary embodiment, including the following steps S210 to S240:
step S210, acquiring a real-time video, and extracting a target real-time video of a target object from the real-time video;
step S220, synthesizing a target real-time video of a target object with a target virtual scene to obtain a real-time synthesized video;
step S230, obtaining a stop motion image sequence shot by a plurality of entity cameras by referring to the real-time synthesized video, and matting the image sequence of the target object from the stop motion image sequence;
and step S240, synthesizing the image sequence of the target object with the target virtual scene to obtain a stop-motion synthesized video.
In the video synthesis method, a target real-time video of the target object is synthesized with a target virtual scene to obtain a real-time synthesized video, and an image sequence of the target object is synthesized with the target virtual scene to obtain a stop-motion synthesized video. On one hand, the pose information of the target object in the virtual scene can be previewed in real time, so that a user can determine a freeze frame picture required to be synthesized, a required freeze frame synthesized video is synthesized, and a more novel visual experience is brought to the user. On the other hand, the virtual scene is adopted for synthesis, which is beneficial to the selection and switching of scenes, and can avoid the limitation of scenes, so that a user can select a proper target virtual scene according to the requirement, and further meet the diversified requirements of the user.
Step S210, acquiring a real-time video, and extracting a target real-time video of a target object from the real-time video.
Real-time video refers to live video about a target object, which can be acquired in real-time during camera shooting. In addition, the camera can shoot the target background under the green curtain background without additional scene selection or set so as to carry out subsequent scratching operation. The target object may be a human, an animal, an object, or the like, may be a static object, or may be a dynamic object. The target real-time video of the target object refers to the video about the target object, which is extracted from the real-time video, and does not contain the background of the target object.
It should be noted that the Video signal captured by the camera may be accessed to the synthesis engine through an SDI (Serial Digital Interface) signal acquisition Card, so as to realize transmission of the real-time Video from the camera to the synthesis engine through the SDI signal acquisition Card.
Step S220, a target real-time video of the target object is synthesized with the target virtual scene to obtain a real-time synthesized video.
The target virtual scene refers to a three-dimensional virtual scene for synthesis, and a desired target virtual scene can be selected from a plurality of virtual scenes. The real-time synthesized video refers to a video obtained by synthesizing a target real-time video of a target object and a target virtual scene, and can acquire pose information of the target object in the target virtual scene in real time.
In an optional embodiment, a scene identifier is obtained; and determining a target virtual scene according to the scene identification.
A unique scene identifier may be set for each virtual scene, and the scene identifier is used to distinguish the virtual scenes. The user can control the synthesis engine to call the target virtual scene corresponding to the scene identification by selecting the scene identification.
It should be noted that, a user may directly select a target scene identifier at the composition engine, and the composition engine reads the scene identifier selected by the user. The user can also select a scene identifier through special control equipment, the control equipment sends the selected scene identifier to the synthesis engine, and the synthesis engine receives the scene identifier sent by the control equipment, wherein the control equipment can be intelligent mobile equipment such as a mobile phone and is used for realizing remote control of the target virtual scene.
Step S230, obtaining a sequence of stop motion images shot by the plurality of physical cameras with reference to the real-time composite video, and matting an image sequence of the target object from the sequence of stop motion images.
The stop motion image sequence is composed of a plurality of angle images of the target object at a certain moment, and the images are from a plurality of physical cameras. The image sequence of the target object refers to the image sequence about the target object, which is extracted from the stop motion image sequence, and does not contain the background of the target object. By watching the real-time composite video, the user can clearly know the position, approximate picture effect and other information of the target object in the virtual scene, so as to determine the stop-motion image sequence to be shot by the plurality of physical cameras.
In an alternative embodiment, the sequence of stop motion images may be generated by: and numbering each entity camera, sequentially placing each entity camera according to the number, naming the name of the shot image by the number, and forming a sequence diagram with an inherent format.
The image formats are unified through the numbering form in the process, so that the images shot by the entity camera can be automatically sequenced, and the situation that the images shot by the entity camera are disordered in arrangement and the finally formed effect of the stop-motion composite video is influenced is avoided.
In an alternative embodiment, the order in which the images in the sequence of stop motion images are arranged may be determined by: and determining the sequence of the image arrangement in the stop-motion image sequence according to the position relation of the plurality of entity cameras.
The positions of the plurality of entity cameras can be in a circular, semicircular, linear and other states, and the entity cameras can be placed according to the effect to be presented by a user. The positional relationship of the plurality of cameras may be, for example, an upper, lower, left, right, or the like. For example, if 4 physical cameras are placed in a linear state, which are the camera 1, the camera 2, the camera 3 and the camera 4, the positional relationship of the 4 cameras is as follows: the method comprises the following steps that a camera 1 is positioned on the left side of a camera 2, a camera 2 is positioned on the left side of a camera 3, a camera 3 is positioned on the left side of a camera 4, and images shot by the camera 1, the camera 2, the camera 3 and the camera 4 are sequentially arranged to obtain a stop-motion image sequence. The arrangement order of the images is determined according to the position relation, so that the stop-motion composite video synthesized by the image sequence is visually coherent and smooth.
And step S240, synthesizing the image sequence of the target object with the target virtual scene to obtain a stop-motion synthesized video.
The stop-motion composite video refers to a video obtained by compositing an image sequence of a target object and a target virtual scene, and can acquire a plurality of angle pictures of the target object in the virtual scene at a certain moment.
In an alternative embodiment, the step shown in fig. 3 may be used to synthesize the image sequence of the target object with the target virtual scene to obtain the stop-motion composite video, which specifically includes the following steps S310 to S320:
step S310, acquiring an image sequence of a target virtual scene; the image sequence of the target virtual scene is obtained by shooting the target virtual scene for multiple times by the virtual cameras according to the corresponding pose of each entity camera;
the pose corresponding to each camera refers to information such as the placing position and the shooting angle of the camera. The virtual camera can be arranged in the synthesis engine and used for acquiring an image sequence of a target virtual scene, the synthesis engine shoots the target virtual scene for multiple times according to the pose of each entity camera, the specific shooting sequence of the virtual camera can be determined by the position relation of a plurality of entity cameras, and can also be determined by the arrangement sequence of images in a stop-motion image sequence or an image sequence of the target virtual scene, so as to ensure that the acquired image sequence corresponds to the images in the sequence of the target object.
And step S320, correspondingly synthesizing the image sequence of the target object and the images in the image sequence of the target virtual scene to obtain a stop-motion synthesized video.
The process correspondingly synthesizes the image sequence with the images in the sequence of the target object to obtain a synthesized image sequence, and generates a stop-motion synthesized video from the synthesized image sequence.
The step shown in fig. 3 corresponds the position and pose of each virtual camera to obtain an image sequence of a three-dimensional target virtual scene, and synthesizes the image sequence of the target object and the image sequence of the three-dimensional target virtual scene to better integrate the target object into the three-dimensional target virtual scene, so that the finally generated freeze-frame synthesized video is more realistic from a visual perspective.
In an alternative embodiment, the real-time composite video and the stop motion composite video may be transmitted to a display device to display the real-time composite video and the stop motion composite video.
The display device may for example: television screens, projectors, computer screens, and the like, have display capabilities to facilitate previewing of the resultant video. In addition, the display device may also be the above-mentioned control device to enable real-time remote preview.
In an alternative embodiment, the display device may include a first display device and a second display device, and the sending of the real-time composite video and the stop-motion composite video to the display device may be implemented by: sending the real-time synthesized video to a first display device; the stop-motion synthesized video is sent to a second display device.
The process displays the real-time synthesized video and the stop-motion synthesized video on different display devices, is beneficial to the cooperation of multiple persons to synthesize the video, and different persons monitor different display devices to ensure the synthesized video effect, and can be applied to scenes where multiple persons collaboratively make the video.
In an alternative embodiment, the real-time composite video and the stop-motion composite video may also be transmitted to the same display device.
The real-time composite video and the stop-motion composite video are displayed on the same display device in a split screen mode, and the method can be applied to scenes of personal video production.
When the stop-motion composite video is sent to the display device, the stop-motion composite video may be sent to the display device in a form of a video stream for previewing.
In an alternative embodiment, the stop-motion composite video may also be output as a video file.
And outputting the stop-motion composite video into a file form so as to be convenient for sharing the stop-motion composite video with other users.
In an optional implementation manner, a light signal configured for the target virtual scene in advance may be acquired, and the light signal is sent to the light controller, so that the light controller controls the light on the spot by using the light signal.
The light signal refers to the light information of the virtual scene, different light information can be set for different virtual scenes, and when the virtual scene of the target scene is switched, the light information can be correspondingly changed, so that the shot target object can be more truly integrated into the virtual scene after being synthesized, and the synthesized video is more vivid.
In an optional implementation manner, a protocol used by the light signal may be converted into a preset protocol, and the protocol used by the light signal is unified, so as to prevent the light signal sent by the synthesis engine from being unable to communicate with the light on the spot, and further unable to control the light on the spot.
For example, when the protocol used by the light signal sent by the synthesis engine is an Art-net protocol, which cannot directly communicate with the thread light, the Art-net protocol of the light signal needs to be converted into a DMX512 protocol before being sent to the light console, and the light console sends specific control instruction information to the light on site according to the sent light signal to control the light on site.
It should be noted that the Art-net protocol is an ethernet protocol based on TCP/IP protocol stack, and aims to allow large amounts of DMX512 data to be transmitted over a wide area using standard network technologies. The DMX512 protocol is a data dimming protocol, and provides a protocol standard for communication between a light controller and a luminaire device.
In an alternative embodiment, the light signal includes address information of the light, and the light controller turns on the corresponding light according to the address information.
The light signal may include address information of the light, which refers to an IP (Internet Protocol) address of the light, and each light has a fixed and unique IP address. The lamp light to be controlled can be determined by the IP address of the lamp light contained in the lamp light signal.
Fig. 4 provides an example diagram of a video composition method, which integrates devices involved in the video composition process, and is completed by the cooperative operation of 3 composition engines, so as to relieve the processing pressure of a single composition engine.
The control device 410 controls the first synthesis engine 420, and sends control operation information to the first synthesis engine 420 to control the first synthesis engine 420 to select a target virtual scene, where the control operation information may include a scene identifier corresponding to the target virtual scene. The camera 430 transmits the photographed real-time video to the first composition engine 420. The video of the target object is extracted from the real-time video by the first composition engine 420, the video of the target object is composed with the target virtual scene, and the obtained real-time composite video is sent to the first display device 440 or the control device 410 for preview. In addition, the first composition engine 420 sends scene confirmation data to the second composition engine 450 and the third composition engine 460, and synchronizes the target virtual scene on the second composition engine 450 and the third composition engine 460, where the scene confirmation data includes a scene identifier corresponding to the target virtual scene. The plurality of physical cameras 470 generate a stop motion image sequence from the captured image sequence, and transmit the stop motion image sequence to the second composition engine 450. The second composition engine 450 extracts the image sequence of the target object from the stop motion image sequence, composes the image sequence of the target object with the target virtual scene, and sends the obtained stop motion composed video to the second display device 480 for previewing. The third composition engine 460 obtains a sequence of stop-motion images of the plurality of physical cameras 470, extracts a sequence of images of the target object from the sequence of stop-motion images, composes the sequence of images of the target object with the target virtual scene, and generates a video file from the resulting stop-motion composed video. The spot lights 490 are controlled by the light signals sent by the first composition engine 420.
Exemplary embodiments of the present disclosure also provide a video compositing apparatus. As shown in fig. 5, the video compositing apparatus 500 may include:
a first obtaining module 510, configured to obtain a real-time video, and extract a target real-time video of a target object from the real-time video;
a first synthesis module 520, configured to synthesize a target real-time video of the target object with a target virtual scene to obtain a real-time synthesized video;
a second obtaining module 530, configured to obtain a sequence of stop motion images captured by the plurality of physical cameras with reference to the real-time composite video, and extract an image sequence of the target object from the sequence of stop motion images;
a second synthesizing module 540, configured to synthesize the image sequence of the target object and the target virtual scene to obtain a stop-motion synthesized video.
In an alternative embodiment, the video compositing apparatus 500 further comprises a display module: and the video processing module is used for sending the real-time composite video and the stop-motion composite video to the display equipment so as to display the real-time composite video and the stop-motion composite video.
In an alternative embodiment, the display device comprises a first display device and a second display device, and the display module may be configured to: sending the real-time synthesized video to a first display device; the stop-motion synthesized video is sent to a second display device.
In an alternative embodiment, the video compositing apparatus 500 further comprises a light control module: the light signal is sent to the light controller, so that the light controller adopts the light signal to control the light on site.
In an optional embodiment, the light signal in the light control module includes address information of the light, and is used for enabling the light controller to turn on the corresponding light according to the address information.
In an optional implementation, the second obtaining module 530 is further configured to: and determining the sequence of the image arrangement in the stop-motion image sequence according to the position relation of the plurality of entity cameras.
In an alternative embodiment, the video synthesizing apparatus 500 further includes an output module: for outputting the stop-motion composite video as a video file.
In an alternative embodiment, the second synthesis module 540 is further configured to: acquiring an image sequence of a target virtual scene; the image sequence of the target virtual scene is obtained by shooting the target virtual scene for multiple times by the virtual cameras according to the corresponding pose of each entity camera; and correspondingly synthesizing the image sequence of the target object and the images in the image sequence of the target virtual scene to obtain the stop-motion synthesized video.
In an alternative embodiment, the first synthesis module 520 is further configured to: and sending the scene identifier of the target virtual scene to the second synthesis module, so that the second synthesis module determines the target virtual scene according to the scene identifier.
It should be noted that the scene identifier is used to synchronize the target virtual scenes used in the first synthesis module and the second synthesis module. When the first synthesis module and the second synthesis module are executed on different devices, the virtual scene information adopted by the two modules can be ensured to be the same.
The specific details of each part in the video synthesizing apparatus 500 are described in detail in the method part embodiment, and details that are not disclosed may refer to the method part embodiment, and thus are not described again.
Exemplary embodiments of the present disclosure also provide a computer-readable storage medium having stored thereon a program product capable of implementing the above-described video composition method of the present specification. In some possible embodiments, various aspects of the disclosure may also be implemented in the form of a program product comprising program code for causing an electronic device to perform the steps according to various exemplary embodiments of the disclosure described in the above-mentioned "exemplary methods" section of this specification, when the program product is run on the electronic device. The program product may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on an electronic device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
A computer readable signal medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
Exemplary embodiments of the present disclosure also provide an electronic device capable of implementing the above video composition method. An electronic device 600 according to this exemplary embodiment of the present disclosure is described below with reference to fig. 6. The electronic device 600 shown in fig. 6 is only an example and should not bring any limitations to the function and scope of use of the embodiments of the present disclosure.
As shown in fig. 6, electronic device 600 may take the form of a general purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one memory unit 620, a bus 630 that couples various system components including the memory unit 620 and the processing unit 610, and a display unit 640.
The storage unit 620 stores program code that may be executed by the processing unit 610, such that the processing unit 610 performs the steps according to various exemplary embodiments of the present disclosure described in the above-mentioned "exemplary methods" section of this specification. For example, processing unit 610 may perform any one or more of the method steps of fig. 2-3.
The storage unit 620 may include readable media in the form of volatile memory units, such as a random access memory unit (RAM)621 and/or a cache memory unit 622, and may further include a read only memory unit (ROM) 623.
The storage unit 620 may also include a program/utility 624 having a set (at least one) of program modules 625, such program modules 625 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 600, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 600 to communicate with one or more other computing devices. Such communication may occur via an input/output (I/O) interface 650. Also, the electronic device 600 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via the network adapter 660. As shown, the network adapter 660 communicates with the other modules of the electronic device 600 over the bus 630. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the exemplary embodiments of the present disclosure.
Furthermore, the above-described figures are merely schematic illustrations of processes included in methods according to exemplary embodiments of the present disclosure, and are not intended to be limiting. It will be readily understood that the processes shown in the above figures are not intended to indicate or limit the chronological order of the processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, e.g., in multiple modules.
It should be noted that although in the above detailed description several modules or units of the device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functions of two or more modules or units described above may be embodied in one module or unit, according to exemplary embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.
As will be appreciated by one skilled in the art, aspects of the present disclosure may be embodied as a system, method or program product. Accordingly, various aspects of the present disclosure may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system. Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is to be limited only by the following claims.
Claims (12)
1. A method for video compositing, comprising:
acquiring a real-time video, and scratching a target real-time video of a target object from the real-time video;
synthesizing the target real-time video of the target object with the target virtual scene to obtain a real-time synthesized video;
acquiring a stop motion image sequence shot by a plurality of entity cameras by referring to the real-time synthesized video, and matting an image sequence of a target object from the stop motion image sequence;
and synthesizing the image sequence of the target object with the target virtual scene to obtain a stop-motion synthesized video.
2. The method of claim 1, further comprising:
and sending the real-time composite video and the stop-motion composite video to a display device so as to display the real-time composite video and the stop-motion composite video.
3. The method of claim 2, wherein the display device comprises a first display device and a second display device, and wherein sending the live composite video and the stop motion composite video to the display devices comprises:
sending the real-time synthesized video to a first display device;
and sending the stop-motion synthesized video to a second display device.
4. The method of claim 1, further comprising:
and sending the light signal to a light controller, so that the light controller controls the light of the scene by adopting the light signal.
5. The method of claim 4, wherein the light signal comprises address information of the light, and the light signal is used for enabling the light controller to turn on the corresponding light according to the address information.
6. The method of claim 1, further comprising:
and determining the sequence of the image arrangement in the stop-motion image sequence according to the position relation of the plurality of entity cameras.
7. The method of claim 1, further comprising:
and outputting the stop-motion synthesized video as a video file.
8. The method of claim 1, wherein the compositing the sequence of images of the target object with the target virtual scene to obtain a stop-motion composite video comprises:
acquiring an image sequence of the target virtual scene; the image sequence of the target virtual scene is obtained by shooting the target virtual scene for multiple times by the virtual cameras according to the corresponding pose of each entity camera;
and correspondingly synthesizing the image sequence of the target object and the images in the image sequence of the target virtual scene to obtain the stop-motion synthesized video.
9. A video compositing apparatus, comprising:
the first acquisition module is used for acquiring a real-time video and scratching a target real-time video of a target object from the real-time video;
the first synthesis module is used for synthesizing a target real-time video of the target object with a target virtual scene to obtain a real-time synthesized video;
a second acquisition module, configured to acquire a sequence of stop motion images captured by the plurality of physical cameras with reference to the real-time composite video, and extract an image sequence of a target object from the sequence of stop motion images;
and the second synthesis module is used for synthesizing the image sequence of the target object with the target virtual scene to obtain a stop-motion synthesis video.
10. The apparatus of claim 9, wherein the first synthesis module is further configured to send a scene identifier of the target virtual scene to a second synthesis module, so that the second synthesis module determines the target virtual scene according to the scene identifier.
11. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method of any one of claims 1 to 8.
12. An electronic device, comprising:
a processor; and
a memory for storing executable instructions of the processor;
wherein the processor is configured to perform the method of any of claims 1 to 8 via execution of the executable instructions.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110551884.XA CN113115110B (en) | 2021-05-20 | 2021-05-20 | Video synthesis method and device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110551884.XA CN113115110B (en) | 2021-05-20 | 2021-05-20 | Video synthesis method and device, storage medium and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113115110A true CN113115110A (en) | 2021-07-13 |
| CN113115110B CN113115110B (en) | 2022-04-08 |
Family
ID=76723105
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110551884.XA Active CN113115110B (en) | 2021-05-20 | 2021-05-20 | Video synthesis method and device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113115110B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114302128A (en) * | 2021-12-31 | 2022-04-08 | 视伴科技(北京)有限公司 | Video generation method and device, electronic equipment and storage medium |
| CN114845136A (en) * | 2022-06-28 | 2022-08-02 | 北京新唐思创教育科技有限公司 | Video synthesis method, device, equipment and storage medium |
| WO2023131327A1 (en) * | 2022-01-10 | 2023-07-13 | 华为技术有限公司 | Video synthesis method, apparatus and system |
| CN116708899A (en) * | 2022-06-30 | 2023-09-05 | 北京生数科技有限公司 | Video processing method, device and storage medium applied to virtual image synthesis |
| WO2023207354A1 (en) * | 2022-04-29 | 2023-11-02 | 北京字跳网络技术有限公司 | Special effect video determination method and apparatus, electronic device, and storage medium |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020118969A1 (en) * | 2001-02-12 | 2002-08-29 | Takeo Kanade | System and method for stabilizing rotational images |
| US20050024487A1 (en) * | 2003-07-31 | 2005-02-03 | William Chen | Video codec system with real-time complexity adaptation and region-of-interest coding |
| US7366742B1 (en) * | 2004-09-10 | 2008-04-29 | Symantec Operating Corporation | System and method for distributed discovery and management of frozen images in a storage environment |
| CN103814382A (en) * | 2012-09-14 | 2014-05-21 | 华为技术有限公司 | Augmented reality processing method and device of mobile terminal |
| CN106101597A (en) * | 2016-06-30 | 2016-11-09 | 维沃移动通信有限公司 | The image pickup method of a kind of video that fixes and mobile terminal |
| CN110225241A (en) * | 2019-04-29 | 2019-09-10 | 努比亚技术有限公司 | A kind of video capture control method, terminal and computer readable storage medium |
| CN110942018A (en) * | 2019-11-25 | 2020-03-31 | 北京华严互娱科技有限公司 | Real-time multi-degree-of-freedom dynamic visual background wall shooting method and system |
| CN111479036A (en) * | 2019-01-24 | 2020-07-31 | 上海赢赞数字科技有限公司 | Bullet time special effect shooting system and method based on camera |
| CN112562056A (en) * | 2020-12-03 | 2021-03-26 | 广州博冠信息科技有限公司 | Control method, device, medium and equipment for virtual light in virtual studio |
-
2021
- 2021-05-20 CN CN202110551884.XA patent/CN113115110B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020118969A1 (en) * | 2001-02-12 | 2002-08-29 | Takeo Kanade | System and method for stabilizing rotational images |
| US20050024487A1 (en) * | 2003-07-31 | 2005-02-03 | William Chen | Video codec system with real-time complexity adaptation and region-of-interest coding |
| US7366742B1 (en) * | 2004-09-10 | 2008-04-29 | Symantec Operating Corporation | System and method for distributed discovery and management of frozen images in a storage environment |
| CN103814382A (en) * | 2012-09-14 | 2014-05-21 | 华为技术有限公司 | Augmented reality processing method and device of mobile terminal |
| CN106101597A (en) * | 2016-06-30 | 2016-11-09 | 维沃移动通信有限公司 | The image pickup method of a kind of video that fixes and mobile terminal |
| CN111479036A (en) * | 2019-01-24 | 2020-07-31 | 上海赢赞数字科技有限公司 | Bullet time special effect shooting system and method based on camera |
| CN110225241A (en) * | 2019-04-29 | 2019-09-10 | 努比亚技术有限公司 | A kind of video capture control method, terminal and computer readable storage medium |
| CN110942018A (en) * | 2019-11-25 | 2020-03-31 | 北京华严互娱科技有限公司 | Real-time multi-degree-of-freedom dynamic visual background wall shooting method and system |
| CN112562056A (en) * | 2020-12-03 | 2021-03-26 | 广州博冠信息科技有限公司 | Control method, device, medium and equipment for virtual light in virtual studio |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114302128A (en) * | 2021-12-31 | 2022-04-08 | 视伴科技(北京)有限公司 | Video generation method and device, electronic equipment and storage medium |
| WO2023131327A1 (en) * | 2022-01-10 | 2023-07-13 | 华为技术有限公司 | Video synthesis method, apparatus and system |
| WO2023207354A1 (en) * | 2022-04-29 | 2023-11-02 | 北京字跳网络技术有限公司 | Special effect video determination method and apparatus, electronic device, and storage medium |
| CN114845136A (en) * | 2022-06-28 | 2022-08-02 | 北京新唐思创教育科技有限公司 | Video synthesis method, device, equipment and storage medium |
| CN114845136B (en) * | 2022-06-28 | 2022-09-16 | 北京新唐思创教育科技有限公司 | Video synthesis method, device, equipment and storage medium |
| WO2024001661A1 (en) * | 2022-06-28 | 2024-01-04 | 北京新唐思创教育科技有限公司 | Video synthesis method and apparatus, device, and storage medium |
| CN116708899A (en) * | 2022-06-30 | 2023-09-05 | 北京生数科技有限公司 | Video processing method, device and storage medium applied to virtual image synthesis |
| CN116708899B (en) * | 2022-06-30 | 2024-01-23 | 北京生数科技有限公司 | Video processing method, device and storage medium applied to virtual image synthesis |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113115110B (en) | 2022-04-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113115110B (en) | Video synthesis method and device, storage medium and electronic equipment | |
| CN111698390B (en) | Virtual camera control method and device, and virtual studio implementation method and system | |
| CN107040794A (en) | Video broadcasting method, server, virtual reality device and panoramic virtual reality play system | |
| CN112073765A (en) | Display device | |
| CN112073798B (en) | Data transmission method and equipment | |
| CN113473207B (en) | Live broadcast method and device, storage medium and electronic equipment | |
| CN113240700B (en) | Image processing method and device, computer readable storage medium and electronic equipment | |
| KR100901111B1 (en) | Live-Image Providing System Using Contents of 3D Virtual Space | |
| CN113395540A (en) | Virtual broadcasting system, virtual broadcasting implementation method, device and equipment, and medium | |
| CN105847672A (en) | Virtual reality helmet snapshotting method and system | |
| CN110730340B (en) | Virtual audience display method, system and storage medium based on lens transformation | |
| CN112543344A (en) | Live broadcast control method and device, computer readable medium and electronic equipment | |
| WO2009078909A1 (en) | Virtual object rendering system and method | |
| CN112153472A (en) | Method and device for generating special picture effect, storage medium and electronic equipment | |
| CN116320363B (en) | Multi-angle virtual reality shooting method and system | |
| WO2023236656A1 (en) | Method and apparatus for rendering interactive picture, and device, storage medium and program product | |
| CN112019921A (en) | Body motion data processing method applied to virtual studio | |
| CN113315885B (en) | Holographic studio and system for remote interaction | |
| CN116112617A (en) | Method and device for processing performance picture, electronic equipment and storage medium | |
| CN114286077B (en) | Virtual reality device and VR scene image display method | |
| CN112887620A (en) | Video shooting method and device and electronic equipment | |
| CN112019922A (en) | Facial expression data processing method applied to virtual studio | |
| CN112887653A (en) | Information processing method and information processing device | |
| US11792380B2 (en) | Video transmission method, video processing device, and video generating system for virtual reality | |
| CN114461165B (en) | Virtual-real camera picture synchronization method, device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20230117 Address after: 1/F, Block C, Building 7, West Zone, Zhongguancun Software Park, Yard 10, Northwest Wangdong Road, Haidian District, Beijing, 100193 Patentee after: NETEASE INFORMATION TECHNOLOGY(BEIJING) CO.,LTD. Address before: Room 801901, building 5, No.16 Keyun Road, Tianhe District, Guangzhou City, Guangdong Province 510000 Patentee before: GUANGZHOU BOGUAN INFORMATION SCIENCE & TECHNOLOGY Co.,Ltd. |