CN112087663B - Method for generating dance video with adaptive light and shade environment by mobile terminal - Google Patents
Method for generating dance video with adaptive light and shade environment by mobile terminal Download PDFInfo
- Publication number
- CN112087663B CN112087663B CN202010946914.2A CN202010946914A CN112087663B CN 112087663 B CN112087663 B CN 112087663B CN 202010946914 A CN202010946914 A CN 202010946914A CN 112087663 B CN112087663 B CN 112087663B
- Authority
- CN
- China
- Prior art keywords
- user
- video
- dance
- background
- template
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
- H04N21/44016—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving splicing one content stream with another content stream, e.g. for substituting a video clip
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/472—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
- H04N21/47205—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content for manipulating displayed content, e.g. interacting with MPEG-4 objects, editing locally
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
Abstract
The invention relates to a method for generating a dance video with a light and shade environment adaptive function by a mobile terminal, which comprises the following steps: determining information of a user video light source according to a user video under a pure color background; determining configuration information of a template light source according to a dance team template selected by a user; modifying the information of the user video light source or the configuration information of the template light source; and combining the dance team template with the background-free user dance video. This application is in order to be more true with a plurality of no background user dance videos and background template synthesis, through configuration light source etc. in the dance video, increases the function of light and shade simulation adaptation dance video, increases the third dimension of video.
Description
Technical Field
The invention relates to the technical field of video synthesis, in particular to a method for generating a dance video with a light and shade environment adapted by a mobile terminal.
Background
Currently, various types of small videos are popular with users, wherein the types of videos include various types such as music, science and technology, dancing and the like, and audiences and crowds are wide. With the rapid development of small videos, various video editing and processing functions become more and more abundant.
With the improvement of the processing capacity of the mobile terminal, the popularization and development of small videos are promoted by video synthesis on the mobile terminal. During the dance video production process, the user can add and modify some elements which are favorite by the user at will, such as selecting background types which are favorite by the user. Because the synthesized environment lacks factors such as light rays and the like, the synthesized trace of the dance video is serious, and the rendering effect is not real.
Disclosure of Invention
Aiming at the technical problems in the prior art, the invention provides a method for generating a dance video with a light and shade environment adapted by a mobile terminal, which comprises the following steps: determining information of a user video light source according to a user video under a pure color background; determining configuration information of a template light source according to a dance team template selected by a user; modifying the information of the user video light source or the configuration information of the template light source; and combining the dance team template with the background-free user dance video.
The method as described above, further comprising: collecting pixel points of a plurality of pure color backgrounds in a user video under the pure color background; analyzing pixel points of a plurality of pure color backgrounds by utilizing a neural network model in deep learning to determine a plurality of light paths; and determining information of the user video light source based on the determined plurality of light paths.
The method as described above, wherein the configuration information of the template light source is kept consistent with the information of the user video light source; wherein the configuration information comprises: one or more of location, color, brightness, and dynamic effects.
The method as described above, wherein the shading of the user's body in the background-free user dance video is modified in response to a change in the position of the template light source.
The method as above, wherein the configuration information of the template light sources is modified based on an accompaniment of the dance video.
The method as described above, wherein the configuration information of the template light source is modified based on the mobile-end system time.
The method as described above, further comprising: and simulating user shadows in the multiple background-free user dance videos in a combined mode of the configuration information of the template light source and the background-free user dance videos.
The method as described above, further comprising: generating shadow layers by user shadows in a plurality of background-free user dance videos; and superposing and synthesizing the shadow image layer with a plurality of background-free user dance videos and/or background templates.
The method as described above, wherein the overlay blending process generates the color of the overlaid portion by a plurality of template light sources and/or a plurality of user shadows generated by the backless user dance video.
The method as described above, wherein the color of the light emitted by the template light source and the color of the light irradiated on the background-free user dance video part are subjected to a mixed rendering process by using a pixel mixing mode.
The method as described above, wherein the light emitted by the template light source is directional light.
According to another aspect of the present invention, there is provided a mobile terminal, including: a camera module configured to acquire a user video in a solid background; an interaction module configured to select a dance template in response to a user operation; a video processing module configured to determine information of a user video light source from a user video in a solid background; determining configuration information of a template light source according to a dance team template selected by a user; modifying the information of the user video light source or the configuration information of the template light source; and combining the dance team template with the background-free user dance video.
The mobile terminal as described above, wherein the video processing module includes: a template providing unit configured to provide a plurality of background templates and/or dance templates; a video matting unit configured to acquire a background-free user dance video based on a user video in a solid background from the camera module; and a video synthesis unit configured to obtain a synthesized dance video using the background template, the dance team template, and the background-free user dance video.
As described above, at the mobile terminal, the video processing module is configured to collect pixel points of a plurality of solid backgrounds in the user video under the solid background; analyzing pixel points of a plurality of pure color backgrounds by utilizing a neural network model in deep learning to determine a plurality of light paths; and determining information of the user video light source based on the determined plurality of light paths.
The mobile terminal as described above, wherein the shading of the user's body in the background-free user dance video is modified in response to a change in the position of the template light source.
The mobile terminal modifies the configuration information of the template light source based on the accompaniment of the dance video.
The mobile terminal modifies the configuration information of the template light source based on the system time of the mobile terminal.
As described above, the video processing module is configured to simulate the user shadows in the multiple background-free user dance videos by using the configuration information of the template light source and the combined manner of the background-free user dance videos.
The mobile terminal as described above, further comprising: generating shadow layers by user shadows in a plurality of background-free user dance videos; and superposing and synthesizing the shadow image layer with a plurality of background-free user dance videos and/or background templates.
The mobile terminal as described above, wherein the color of the superimposed portion is generated by superimposing and mixing the user shadows generated by the plurality of template light sources and/or the plurality of background-free user dance videos.
This application is in order to be more true with a plurality of no background user dance videos and background template synthesis, through configuration light source etc. in the dance video, increases the function of light and shade simulation adaptation dance video, increases the third dimension of video.
Drawings
Preferred embodiments of the present invention will now be described in further detail with reference to the accompanying drawings, in which:
FIG. 1 is a schematic diagram of a structure of a system for generating a dance combination dance video according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a mobile terminal according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a server according to one embodiment of the invention;
FIG. 4 is a flowchart of a method for a mobile terminal to obtain a customized background real-time dance video according to an embodiment of the present invention;
FIGS. 5A-E are graphical user interfaces for a mobile terminal to generate dance combination dance videos;
FIG. 6 is a flow chart of a method for a mobile terminal to generate a dance combination dance video according to an embodiment of the present invention;
FIG. 7 is a schematic diagram of a dance formation according to an embodiment of the present invention;
FIG. 8 is a schematic view of a scene of a composite dance video according to one embodiment of the present invention; and
fig. 9 is a method for generating a dance video with adapted light and dark environments by a mobile terminal according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In the following detailed description, reference is made to the accompanying drawings that form a part hereof and in which is shown by way of illustration specific embodiments of the application. In the drawings, like numerals describe substantially similar components throughout the different views. Various specific embodiments of the present application are described in sufficient detail below to enable those skilled in the art to practice the teachings of the present application. It is to be understood that other embodiments may be utilized and structural, logical or electrical changes may be made to the embodiments of the present application.
FIG. 1 is a schematic diagram of a structure of a system for generating a dance combination dance video according to an embodiment of the present invention. As shown, the system includes one or more mobile terminals 10 and a server 20 communicatively coupled via a wireless network. The mobile terminal 10 includes a mobile phone, a tablet computer, a personal entertainment device or a combination thereof, and is used for recording a dance video and changing the background of the video into a background template in real time based on a specific application. After the recording is finished, the user can store the videos to the personal account of the server by using the wireless network, and can also share the dance videos to be displayed to more people for watching.
Fig. 2 is a schematic structural diagram of a mobile terminal according to an embodiment of the present invention. As shown in the figure, the mobile terminal 10 includes a camera module 101, a video processing module 102, a communication module 103, and an interaction module 104, where the video processing module 102 is electrically connected to the camera module 101, the communication module 103, and the interaction module 104, respectively, and the camera module 101 is electrically connected to the interaction module 104.
The camera module 101 is one or more cameras, and is configured to obtain a user video in a solid background. The camera can be a front camera at the mobile end or a rear camera. The camera comprises the functions of automatic focusing and manual focusing, and the requirements of different users are met.
The video processing module 102 includes a template providing unit 1021, a video matting unit 1022, a property editing unit 1023 and a video synthesizing unit 1024, wherein the template providing unit 1021 is electrically connected with the interaction module 104 and the property editing module 1023 respectively; the video matting unit 1022 is electrically connected to the image capturing module 101 and the attribute editing unit 1023, respectively; the attribute editing unit 1023 is electrically connected to the interaction module 104 and the video composition unit 1024, respectively, and the video composition unit 1024 is electrically connected to the communication module 103.
The template providing unit 1021 is used for providing a plurality of background templates and/or dance templates, wherein the background template may be a video or a picture. The background template types include, but are not limited to, beauty, interior, stage, holiday, atmosphere, and green screen, among others. The background of the green curtain type is used for recording the customized background template during video in real time, can adapt to the whole screen, is synthesized with the scratched-out dance video of the user without the background, and cannot see the real recording environment completely. Dynamic elements such as birds, waterfalls, rowing and the like can be added in the green curtain type background template, and video content is enriched.
The video matting unit 1022 receives the image from the camera module 101, and acquires the dance video of the background-free user by using the color median of the solid background. In some embodiments, solid background colors include, but are not limited to, green, blue, and red, and those skilled in the art will appreciate that solid backgrounds that can be used to obtain background-free user dance videos may be used in the aspects of the present application.
The attribute editing unit 1023 receives instructions from the interaction module 104 to adjust the video attributes. Such as adjusting parameters like hue, contrast and aspect ratio of the background-free user dance video and/or the background template. In response to the user generating an instruction on the interaction module 104, the user in the video is beautified, or the displayed figure is adjusted, or the video tone is adjusted, etc.
In some embodiments, the property editing unit 1023 further includes functions to adjust the size, position, distance, and combination dancing of the background-free user dance video. When a user records a video, the angle, the recording position and the like of the dance video of the mobile terminal and the background-free user cannot present the best shooting effect, so that the size, the position and the distance of the obtained dance video of the background-free user in the background template can be adjusted, and the dance video is more natural to be fused with the background template. The size of the dance video of the background-free user is adjusted; the distance is the distance effect of the simulation adjustment camera; the position is the position of the dance video of the background-free user in the background template. The dance video is shot to the user for the combination dance team, will not have a plurality of images of background user dance video duplicate, carries out the video of making up many people dance simultaneously according to the station position of dance team, and the action in the dance is neat and uniform, has improved the observability of dance video. Wherein, the dance change mode includes 1 people dance mode, 2 people dance mode, parallel 3 people dance mode, triangle-shaped 3 people dance mode etc..
And the video synthesis unit 1024 is used for obtaining a synthesized dance video from the background template, the dance team template and the background-free user dance video in real time. The video synthesizing unit 1024 is electrically connected to the interactive module 104, and sends the synthesized video to the interactive module 104, so that the user can view the synthesized video on the interactive module 104 in real time. The video synthesizing unit 1024 may further send the synthesized video to the server 20 through the communication module 103, and share the video produced by more people.
The communication module 103 can include one or more wired or wireless communication interfaces. Such as a communications interface network interface card, wireless modem, or wired modem. In one application, the communication module 103 can be a WiFi modem. In other applications, the communication module 103 can be a 3G modem, a 4G modem, an LTE modem, a bluetooth component, a radio frequency receiver, an antenna, or a combination thereof. The communication module 103 is capable of wireless communication with the server 20, and the mobile device 10 is capable of transmitting information or receiving packets or messages using the communication module 103.
The interaction module 104 is used for presenting a Graphical User Interface (GUI) for interaction with a user, and may also present functional content to the user, such as functional buttons or text boxes or links for displaying background templates and editing video attributes. The user may select the background template by a gesture, such as a swipe, click, etc., on the interactive module 104. The interaction module 104 may be a touch screen display of a Liquid Crystal Display (LCD), and may be a touch screen display of an Organic Light Emitting Diode (OLED).
Fig. 3 is a schematic structural diagram of a server according to an embodiment of the present invention. As shown, the server includes one or more processing modules 301, a communication module 302, and a storage module 303.
The processing module 301 can include one or more Central Processing Units (CPUs), Graphics Processing Units (GPUs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), or a combination thereof. The processing module 301 may be capable of executing software or computer readable instructions stored in the storage module 303 to perform the methods or operations described herein. The processing module 301 can be implemented in a number of different ways. For example, the processing module 301 can include one or more embedded processors, processor cores, microprocessors, logic circuits, hardware Finite State Machines (FSMs), Digital Signal Processors (DSPs), or a combination thereof.
The communication module 302 can include one or more wired or wireless communication interfaces. Such as a communications interface network interface card, wireless modem, or wired modem. In one application, the communication module 302 can be a WiFi modem. In other applications, the communication module 302 can be a 3G modem, a 4G modem, an LTE modem, a bluetooth component, a radio frequency receiver, an antenna, or a combination thereof.
The storage module 303 can store software, data, logs, or a combination thereof. The storage module 303 can be an internal memory or an external memory. For example, the memory can be volatile memory or non-volatile memory, such as non-volatile random access memory (NVRAM), flash memory, disk storage, or volatile memory such as Static Random Access Memory (SRAM).
FIG. 4 is a flowchart of a method for a mobile terminal to obtain a customized background real-time dance video according to an embodiment of the present invention; fig. 5A-E are graphical user interfaces for a mobile terminal to obtain a customized background real-time dance video. At step 410, a video of the user is obtained in a solid background. Referring to fig. 5A, in response to a user selecting an accompaniment, a camera of the mobile device acquires a user video in a solid background. In some embodiments, the user may adjust the position of the mobile device, the shooting distance, etc., based on the user video displayed on the interaction means 104.
At step 420, a background template is selected in response to a user operating on the interactive module. Referring to FIG. 5B, the user selects the "background" button at the interactive module and a list of background templates appears. The background template that can be customized is in the type of "green screen". Not all background templates may be customized background templates, which need to meet certain requirements, such as not being transparent background, and needing to completely cover the interactive module. In response to a user selecting any of the background templates in the "green screen" background template list, the selected background template is displayed on the interactive module.
In step 430, the dance video of the user without the background is obtained by using the color median of the solid background. In some embodiments, the specific steps of obtaining the dance video of the background-free user are as follows:
1. sampling color values of a plurality of positions of a pure color background to obtain color median values of the color values of the plurality of positions;
2. expanding the color value range by taking the color median value as a center; and
3. the background image within the color value range is removed.
In some embodiments, the color values are defined using an HSV color model. The color median value may be a median value obtained by sorting the color values of the plurality of positions according to size and taking the plurality of color values according to the sorting. When a user records a video, sometimes, the color value of the pure background of the recorded video is inconsistent due to the fact that the light of the recording environment is not uniformly irradiated on the pure background, and the matting effect is influenced. Therefore, a plurality of color values of different pure color background positions are obtained, the median value is obtained, and a certain range is expanded, so that the dance video quality of the user without the background is improved. Where the extended range is the range of numerical experiences for a certain color (green/blue/red, etc.) in the HSV color space.
In some embodiments, nine-grid convolution is performed on the pure-color background image to determine boundary pixels and the transparency values of the boundary pixels are adjusted according to the convolution values, so that the jaggy of the cutout is effectively reduced. According to the method and the device, the convolutional neural network is utilized to judge the boundary of the video pixels, and different pixels are set to different transparent values through convolution values, so that the obtained target user image is more complete and more natural.
At step 440, the video properties are edited in response to a user operating on the interactive module. Referring to fig. 5B, sometimes, the mobile terminal automatically acquires the dance video of the background-free user with low quality, and a partial green background still exists. At this time, the user can click the "green screen" function key of the screen, enter fig. 5C, and adjust the slider of the "color key", and as the slider moves, the green part in the screen increases or decreases, and the user stops sliding when the green part completely disappears or is at least, and at this time, the best dance video of the user without the background is presented. In some embodiments, the mobile end presents the color median value to the user; and adjusting the removed background color range in real time in response to the user manually adjusting the color value. If the slide block on the color key stays at the color median, the user only needs to finely adjust the color value, the best dance video without the background can be presented, and the adjustment range of the user is reduced. Referring to fig. 5D, sliding sliders of different functions may adjust one or more of the size, position, cropping width, and distance of the background-free user dance video to fit the background template.
In some embodiments, editing the video properties further comprises one or more of beautifying, slimming, and filter manipulation of the user in the composite dance video. The user can perform personalized adjustment on the user image in the video according to the preference of the user, so that the user in the video is more beautiful.
In step 450, a composite dance video is obtained in real time based on the background template and the background-free user dance video. After the user finishes editing the video attribute, responding to the click of a 'shooting' function key by the user, the mobile terminal starts to record the video, synthesizes the background template and the dance video of the user without the background in real time into the video, and stores the synthesized video in the mobile terminal after the user clicks 'end'.
In some embodiments, before recording a video, the mobile terminal may automatically or manually adjust focusing and exposure parameters by a user, so that the camera achieves an optimal shooting effect; when recording a video, the mobile terminal locks focusing and exposure parameters, and a high-quality background-free user dance video is obtained in the recording process. If the camera opens the automatic focusing or automatic exposure function in the recording process, the color value of the pure-color background acquired by the mobile terminal may change, so that the acquired dance video of the background-free user contains the pure-color background, and the problem of the reduction of the matting quality is caused.
Referring to FIG. 5E, in response to a user clicking on the "split screen" function key on the left side of the screen, the composite dance video may be edited as a multi-person dance team. In some embodiments, a plurality of dance templates in the split screen function can be selected by the user, and the formation position of the dance templates is fixed. In other embodiments, the user can customize the positions of dance videos of the background-free users, and the user can freely select the number and the positions of the dance videos of the background-free users to design dance shapes. Wherein in the custom dancing team, the number of dance videos of the background-free users is at most 8.
In order to solve the problem that dance videos of background-free users are stacked unnaturally in a composite dance video, traditional dance videos are calculated by using a pixel mixed mode on different layers of the dance videos of the background-free users, and the visual effect of real overlapping is achieved. However, in the processing, each pixel of the adjacent layer needs to be computationally rendered, thereby causing a problem of a large computational load. The method is difficult to popularize at a mobile terminal with limited calculation amount, so that the method can be used for performing mixed optimization on the background-free user dance videos of different levels at the mobile terminal and has the advantages of small calculation amount, high response speed and the like.
Fig. 6 is a flowchart of a method for generating a dance combination dance video by a mobile terminal according to an embodiment of the present invention. At step 610, a background-free user dance video is acquired in a solid color background. The method for acquiring the dance video of the background-free user is the same as the step for acquiring the dance of the background-free user in the method shown in fig. 4, and is not described herein again.
In step 620, according to the dance team template selected by the user, a combination mode of the dance videos of the users without the background is determined. Before the video recording is started, the user can operate on an interactive module of the mobile terminal to select the dance team template. In some embodiments, a dance template list is displayed on the interactive module, and after a user selects a dance template, the interactive module displays dance formation in real time; in other embodiments, the user clicks a function button on the interactive module, and the dance team formation is changed on the interactive module in real time. Wherein the dance template comprises a plurality of dance formations, different dance formations and different corresponding levels.
Wherein, dance team formation is the position of dancer, and different dance team formations correspondingly will not have different combination modes of background user dance video show to the system. The dance team formation can three-dimensionally divide the space in the video into front, back, left, right and other directions. The dance team formation can be combined into a triangle, a pyramid and the like, and the visual effect of the video is improved.
FIG. 7 is a schematic diagram of a dance formation according to an embodiment of the present invention. Generally, the more people dancing in a dance, the more levels. Referring to fig. 7, the front 1 and rear 2 dance teams are taken as an example, the front 1 dancer is level 0, and the rear parallel 2 dancers are level 1, wherein the display scale of the level 0 is larger than the level 1. The dancing video of the user without the background protruding forward can be more clearly appreciated by the user watching the video, and watching satisfaction is improved. The dancers in the level 0 and the level 1 are copied by the background-free user dance videos and distributed at different positions, so that the actions of the dancers at different levels are completely consistent. It will be appreciated by those skilled in the art that the levels in the dance template are not limited to 2 levels, but may be 3 or more levels.
At step 630, one or more overlapping portions of the combined background-free user dance video are determined. In some embodiments, the priority of each layer formed by the dance video of the background-free user is determined according to the dance formation in the dance template, wherein the layer of the dance video of the background-free user positioned behind the background-free user is drawn preferentially. And combining the dance videos into a dance team formation, and drawing the dance videos of the background-free users layer by layer. In some embodiments, the backless user dance video may be drawn sequentially according to a back-to-front order. The overlapped part of the background-free user dance video is the overlapped part of the background-free user dance video in different levels before and after, such as the overlapped part 710 and the overlapped part 720 in fig. 7. Furthermore, one or more overlapping part boundaries can be determined, so that the purpose of reducing the calculation amount is achieved, and the rendering speed is improved.
In some embodiments, the boundaries of one or more overlapping portions are determined using pixel coordinate values of a background-free user dance video. For example, the level pixels of level 0 and level 1 are respectively determined with respect to the interactive module as coordinate values, and the same coordinate represents the same position on the interactive module under the same coordinate system. Therefore, by comparing the coordinate values of the user images in the plurality of background-free user dance videos, the boundary of the overlapping portion can be determined.
In other embodiments, the boundaries of one or more overlapping portions are determined using pixel color values between background-free user dance videos. Referring to fig. 7, the background-free user dance video in level 0 is transparent in pixels except for the portrait portion. Therefore, by acquiring video pixels of a level 0 adjacent hierarchy, it is possible to quickly acquire the boundary of the overlapping portion of the background-free user dance video. Those skilled in the art will appreciate that any technique that can determine the overlapping portion of the dance video can be applied to the aspects of the present application.
At step 640, rendering transition processing is performed on the one or more overlapping portions. Wherein the step of performing rendering transition processing on the one or more overlapping portions comprises:
determining a boundary extension range based on the one or more overlapping portions;
the boundary extension range is 2-8 pixels inward from one or more overlapping part boundaries; and
and performing rendering transition processing on the pixels in the extended range by using the pixel blending mode.
Wherein the size of the boundary extension range is related to the display scale of the dance video of the user without the background. As can be seen from the above, the display scale of the dance video of the background-free user arranged at the front level is larger than that of the dance video arranged at the back level. Therefore, the boundary extension range is determined according to the display scale of the dance video of the background-free user, the rendering transition processing range can be detailed, and the quality of the synthesized dance video is improved.
In some embodiments, the rendering transition process is a color mixing special effects process. For example, the pixels in the extended range are rendered by openGL ES, and further processed by glBlendFunc (GL _ ONE, GL _ ONE _ MINUS _ SRC _ ALPHA) function. In the extended range, the colors in the dance videos of different levels are respectively taken out and multiplied by a coefficient, and then a new color is obtained by mixing, so that the stacking relation among the dance videos of different levels is more natural and real. The algorithm in the glBlendFunc function may be adjusted according to actual requirements, and is not limited herein.
In other embodiments, the rendering transition process is a gaussian weighted averaging process. Gaussian weighted averaging is to use the gray values of a plurality of pixels in a pixel neighborhood to carry out weighted average processing and then replace the gray value of the original pixel, so that the problems of saw teeth and unnaturalness of a plurality of video laminated parts are eliminated, and the laminated parts are smooth, natural and real. It should be understood by those skilled in the art that the above-mentioned technical means for processing the overlapped parts of the multiple videos is only an illustrative solution in the present application, and any solution capable of solving the above-mentioned technical problem can be applied in the present application without limitation.
FIG. 8 is a schematic view of a scene of a composite dance video according to one embodiment of the present invention. As shown, the dance video includes a background template 810 and one or more background-free user dance videos 820, where the background template 810 includes a dance area 811 of the user. The background template 810 is an outdoor landscape template, the light source is set as the sun 813 correspondingly, the requirement of the outdoor landscape template is met, and natural scenes such as a white cloud 812, a river 814 and a tree 815 are added, so that the content of the background template is richer. The background templates are divided into various types, and can be divided into indoor scenery and outdoor scenery according to scenes; the method can be divided into a dynamic background and a static background according to the dynamic and static types, wherein elements in the dynamic background include but are not limited to rivers, birds and waterfalls; the background can be divided into landscape background, florist background, human landscape background, etc. according to the environment. In addition, the user can select the dance template to determine the positions of a plurality of dance users in the dance area. This application for it is more true with the synthesis of a plurality of no background user dance videos and background template, increase the function of light and shade simulation adaptation dance video, through dispose elements such as light source in the dance video, increase the third dimension of video. The specific method comprises the following steps:
fig. 9 is a method for generating a dance video with adapted light and dark environments by a mobile terminal according to an embodiment of the present invention. At step 910, information of the user video light source is determined according to the user video in the solid background. The method comprises the following specific steps: collecting pixel points of a plurality of pure color backgrounds in a user video under the pure color background; analyzing pixel points of a plurality of pure color backgrounds by utilizing a neural network model in deep learning to determine a plurality of light paths; information of the user video light source is determined based on the determined plurality of light paths. Wherein the template light source information includes: one or more of brightness, position, number color, and dynamic effect of the light source. For example, the user station dances to record a video under a green screen background, and a bulb is arranged at the upper left of the user to provide illumination for the user to record the video; according to the principle that light propagates along a straight line, light rays striking a user body are required to be downward in an inclined mode, and when the light rays strike the user body, the color and the brightness of pixels can change, so that a neural network model in deep learning is utilized to analyze pixel points of a plurality of pure-color backgrounds to determine the irradiation directions of a plurality of light paths; the position of the user video light source can be determined according to the illumination directions of the plurality of light paths. It should be understood by those skilled in the art that the present application only needs to determine the approximate information of the user video light source, and does not need to have particularly accurate light source information, and the technical solutions that can achieve the above-mentioned objectives can be applied to the present application, and are not limited herein.
At step 920, configuration information of the template light source is determined according to the dance template selected by the user. Wherein, different dance templates are provided with different template light sources. As described above, in the outdoor scene template, the template light source may be the sun; in an indoor background template, the template light source may be a light bulb or the like. And the information of configuring the color, lighting and the like of the template light source can be different according to the characteristics of different background templates.
At step 930, information of the user video light source or configuration information of the template light source is modified. In some embodiments, the configuration information of the template light source is kept consistent with the information of the user video light source. In order to ensure that the configuration information of the template light source is consistent with the information of the user video light source, the information of the user video light source can be modified, and the configuration information of the template light source can also be modified, wherein the brightness change of the user body in the dance video of the background-free user is modified in response to the position change of the template light source. Preferably, the light source configuration information with a small calculation amount is modified. The dance video synthesis method and the dance video synthesis device keep the positions, the colors, the brightness and the like of the template light source and the user video light source consistent, so that the dance video after synthesis is more harmonious and natural.
In some embodiments, the configuration information of the template light source is modified based on the mobile-end system time. The types of the template light source include a natural light source and an artificial light source. For example, in response to the user recording a dance video by using the mobile terminal, when the system time of the mobile terminal is 12 pm, the sun is set at the aerial position, and the brightness is adjusted to the highest value; when the system time is 8 o' clock at night, the sun can be replaced by a street lamp as a light source, and the color tone of the light source is adjusted to be warm. In the scheme of this application, the light that the template light source sent all is direction light, builds a light and shade's lines in synthetic dance video through the light that direction light sent, highlights the authenticity of scene.
In some embodiments, the configuration information of the template light sources is modified based on the accompaniment of the dance video. Music and lighting enhance the dance atmosphere and mobilize the mood of the user. When a user records a dance video, the system can adjust the brightness, the position, the color, the dynamic effect and other information of the template light source according to the playing time of the accompanying music. For example, when the user plays the dance accompaniment to the climax part, the template light source can flash and change the color of the light source at a certain frequency, thereby improving the video interest.
At step 940, the dance template and the background-free user dance video are combined. And combining the dance team template and the dance video of the background-free user in response to the adjusted information of the user video light source or the configuration information of the template light source, wherein in the combined dance video, light and dark line changes exist according to the configuration information of the template light source. Wherein, in a plurality of template light sources and/or a plurality of background-free user dance video environments, when the directional light is projected on the user, the brightness of the irradiated part on the user is increased. When a plurality of template light sources are arranged in the background template, light rays can be superposed on the dance video of the user without the background, and therefore the brightness of the superposed part is higher than that of the non-superposed part. In some embodiments, the color of the light emitted by the template light source and the color of the light irradiated on the background-free user dance video part are subjected to mixed rendering processing by using a pixel mixing mode.
In addition, the light source produces a correspondingly shaped shadow on the ground, according to the profile of the light blockage. For example, the light source emits directional light to illuminate the user, and the audience can see bright parts and shadow parts in the environment, so that the whole environment is more three-dimensional and real. Wherein, the number of the light sources is increased by one, and the shadow generation is doubled; multiple background-free user dance videos may produce multiple shadows.
In some embodiments, user shadows in multiple background-free user dance videos are simulated using configuration information of the template light sources and/or a combination of the background-free user dance videos; generating shadow layers by user shadows in a plurality of background-free user dance videos; and superposing and synthesizing the shadow image layer with the plurality of the dance videos of the background-free users and the background template. Wherein the color of the superimposed portion is generated by superimposing and mixing user shadows generated by the plurality of template light sources and/or the plurality of background-free user dance videos. The application carries out the stack to the stack of a plurality of light and the stack of a plurality of shadow parts and mixes the processing for whole environment light and shade is obvious, builds a very real dance environment, improves user's use and experiences.
The above embodiments are provided only for illustrating the present invention and not for limiting the present invention, and those skilled in the art can make various changes and modifications without departing from the scope of the present invention, and therefore, all equivalent technical solutions should fall within the scope of the present invention.
Claims (18)
1. A method for generating a dance video with a light and shade environment adapted by a mobile terminal comprises the following steps:
determining information of a user video light source according to a user video under a pure color background;
determining configuration information of a template light source according to a dance team template selected by a user;
modifying the information of the user video light source or the configuration information of the template light source; and
combining the dance team template with the dance video of the background-free user;
wherein determining information of the user video light source comprises: collecting pixel points of a plurality of pure color backgrounds in a user video under the pure color background;
analyzing pixel points of a plurality of pure color backgrounds by utilizing a neural network model in deep learning to determine a plurality of light paths; and
information of the user video light source is determined based on the determined plurality of light paths.
2. The method of claim 1, wherein the configuration information of the template light source is kept consistent with the information of the user video light source; wherein the configuration information comprises: one or more of location, color, brightness, and dynamic effects.
3. The method of claim 2, wherein the shading of the user's body in the background-free user dance video is modified in response to a change in the position of the template light source.
4. The method of claim 1, wherein configuration information of the template light sources is modified based on an accompaniment of the dance video.
5. The method of claim 1, wherein the configuration information of the template light source is modified based on a mobile-end system time.
6. The method of claim 1, further comprising: and simulating user shadows in the multiple background-free user dance videos in a combined mode of the configuration information of the template light source and the background-free user dance videos.
7. The method of claim 6, further comprising: generating shadow layers by user shadows in a plurality of background-free user dance videos; and superposing and synthesizing the shadow image layer with a plurality of background-free user dance videos and/or background templates.
8. The method of claim 7, wherein the overlay blending processes user shadows generated by the plurality of template light sources and/or the plurality of background-free user dance videos to produce the color of the overlaid portion.
9. The method of claim 1, wherein the color of the light emitted by the template light source is mixed with the color of the light shining on the non-background user dance video portion using a pixel mixing mode for rendering.
10. The method of claim 1, wherein the light emitted by the template light source is directional light.
11. A mobile terminal, comprising:
a camera module configured to acquire a user video in a solid background;
an interaction module configured to select a dance template in response to a user operation;
a video processing module configured to determine information of a user video light source from a user video in a solid background; determining configuration information of a template light source according to a dance team template selected by a user; modifying the information of the user video light source or the configuration information of the template light source; combining the dance team template with the dance video of the background-free user;
wherein determining information of the user video light source comprises: the video processing module is configured to collect pixel points of a plurality of pure color backgrounds in a user video under the pure color background; analyzing pixel points of a plurality of pure color backgrounds by utilizing a neural network model in deep learning to determine a plurality of light paths; and determining information of the user video light source based on the determined plurality of light paths.
12. The mobile terminal according to claim 11, wherein the video processing module comprises:
a template providing unit configured to provide a plurality of background templates and/or dance templates;
a video matting unit configured to acquire a background-free user dance video based on a user video in a solid background from the camera module; and
a video composition unit configured to obtain a composite dance video using the background template, the dance team template, and the background-free user dance video.
13. The mobile terminal of claim 12, wherein the shading of the user's body in the background-free user dance video is modified in response to a change in the position of the template light source.
14. The mobile terminal of claim 11, wherein configuration information of the template light source is modified based on an accompaniment of the dance video.
15. The mobile terminal of claim 11, wherein the configuration information of the template light source is modified based on a system time of the mobile terminal.
16. The mobile terminal of claim 11, wherein the video processing module is configured to simulate user shadows in the plurality of non-background user dance videos using a combination of information from the template light source and the non-background user dance videos.
17. The mobile terminal of claim 16, further comprising: generating shadow layers by user shadows in a plurality of background-free user dance videos; and superposing and synthesizing the shadow image layer with a plurality of background-free user dance videos and/or background templates.
18. The mobile terminal of claim 17, wherein the overlay blending process generates a color of the overlay portion by processing a plurality of template light sources and/or a plurality of user shadows generated by the background-free user dance video.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010946914.2A CN112087663B (en) | 2020-09-10 | 2020-09-10 | Method for generating dance video with adaptive light and shade environment by mobile terminal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010946914.2A CN112087663B (en) | 2020-09-10 | 2020-09-10 | Method for generating dance video with adaptive light and shade environment by mobile terminal |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112087663A CN112087663A (en) | 2020-12-15 |
| CN112087663B true CN112087663B (en) | 2021-09-28 |
Family
ID=73732243
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010946914.2A Active CN112087663B (en) | 2020-09-10 | 2020-09-10 | Method for generating dance video with adaptive light and shade environment by mobile terminal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112087663B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5747806A (en) * | 1996-02-02 | 1998-05-05 | Instrumentation Metrics, Inc | Method and apparatus for multi-spectral analysis in noninvasive nir spectroscopy |
| CN105657494A (en) * | 2015-12-31 | 2016-06-08 | 北京小鸟看看科技有限公司 | Virtual cinema and implementation method thereof |
| CN110147231A (en) * | 2019-05-23 | 2019-08-20 | 腾讯科技（深圳）有限公司 | Combine special efficacy generation method, device and storage medium |
| CN110266968A (en) * | 2019-05-17 | 2019-09-20 | 北京小糖科技有限责任公司 | A kind of production method and device of video of dancing together |
| CN110290425A (en) * | 2019-07-29 | 2019-09-27 | 腾讯科技（深圳）有限公司 | A kind of method for processing video frequency, device and storage medium |
-
2020
- 2020-09-10 CN CN202010946914.2A patent/CN112087663B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5747806A (en) * | 1996-02-02 | 1998-05-05 | Instrumentation Metrics, Inc | Method and apparatus for multi-spectral analysis in noninvasive nir spectroscopy |
| CN105657494A (en) * | 2015-12-31 | 2016-06-08 | 北京小鸟看看科技有限公司 | Virtual cinema and implementation method thereof |
| CN110266968A (en) * | 2019-05-17 | 2019-09-20 | 北京小糖科技有限责任公司 | A kind of production method and device of video of dancing together |
| CN110147231A (en) * | 2019-05-23 | 2019-08-20 | 腾讯科技（深圳）有限公司 | Combine special efficacy generation method, device and storage medium |
| CN110290425A (en) * | 2019-07-29 | 2019-09-27 | 腾讯科技（深圳）有限公司 | A kind of method for processing video frequency, device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112087663A (en) | 2020-12-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Birn | Digital lighting & rendering | |
| CN111968216B (en) | Volume cloud shadow rendering method and device, electronic equipment and storage medium | |
| US20070146360A1 (en) | System And Method For Generating 3D Scenes | |
| JP4131929B2 (en) | Object image composition method | |
| CN109960872B (en) | AR virtual soft package matching management system and working method thereof | |
| US9183654B2 (en) | Live editing and integrated control of image-based lighting of 3D models | |
| US11663467B2 (en) | Methods and systems for geometry-aware image contrast adjustments via image-based ambient occlusion estimation | |
| CN113436343A (en) | Picture generation method and device for virtual studio, medium and electronic equipment | |
| CN109255841A (en) | AR image presentation method, device, terminal and storage medium | |
| CN112087664A (en) | Method for obtaining customized background real-time dance video by mobile terminal | |
| CN110090437A (en) | Video acquiring method, device, electronic equipment and storage medium | |
| US20220343590A1 (en) | System and techniques for lighting adjustment for an immersive content production system | |
| CN112087662B (en) | Method for generating dance combination dance video by mobile terminal and mobile terminal | |
| CN114820292A (en) | Image synthesis method, device, equipment and storage medium | |
| Mirbauer et al. | SkyGAN: Towards Realistic Cloud Imagery for Image Based Lighting. | |
| CN112087663B (en) | Method for generating dance video with adaptive light and shade environment by mobile terminal | |
| US11887251B2 (en) | System and techniques for patch color correction for an immersive content production system | |
| CN110136239B (en) | Method for enhancing illumination and reflection reality degree of virtual reality scene | |
| CN114902277A (en) | System and method for processing shadows on portrait image frames | |
| US11762481B2 (en) | Light capture device | |
| AU2022202424B2 (en) | Color and lighting adjustment for immersive content production system | |
| US20220343562A1 (en) | Color and lighting adjustment for immersive content production system | |
| CN115546041B (en) | Training method of light supplementing model, image processing method and related equipment thereof | |
| Kloskowski | Photoshop compositing secrets: Unlocking the key to perfect selections and amazing photoshop effects for totally realistic composites | |
| CN103354597B (en) | Image-reproducing means and imaging reconstruction method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |