CN116055800A - Method for mobile terminal to obtain customized background real-time dance video - Google Patents
Method for mobile terminal to obtain customized background real-time dance video Download PDFInfo
- Publication number
- CN116055800A CN116055800A CN202211667026.2A CN202211667026A CN116055800A CN 116055800 A CN116055800 A CN 116055800A CN 202211667026 A CN202211667026 A CN 202211667026A CN 116055800 A CN116055800 A CN 116055800A
- Authority
- CN
- China
- Prior art keywords
- background
- video
- user
- color
- dance
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
- H04N21/44016—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving splicing one content stream with another content stream, e.g. for substituting a video clip
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/472—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
- H04N21/47205—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content for manipulating displayed content, e.g. interacting with MPEG-4 objects, editing locally
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
Abstract
The invention relates to a method for a mobile terminal to obtain a customized background real-time dance video, which comprises the following steps: acquiring a user video under a solid background; responding to interactive operation of a user, and selecting a background template; acquiring a background-free user dance video by using the color median of the solid background; acquiring a synthesized dance video in real time based on a background template and a non-background user dance video; wherein the obtaining the background-free user dance video by utilizing the color median of the solid background comprises: sampling color values of a plurality of positions of the solid color background to obtain color median values of the color values of the plurality of positions; expanding a color value range with the color median as a center; removing the background image in the color value range; displaying the color median to a user; the removed background color range is adjusted in real-time in response to the user manually adjusting the color value range. According to the method and the device, when the mobile terminal records the video, the customized background template is synthesized in the recorded video in real time, so that the threshold for replacing the background is reduced, and the playability of the dance video is improved.
Description
Information about the divisional application
The scheme is a divisional application. The parent case of the division case is an invention patent application with the application date of 2020, 9, 10, the application number of 202010947691.1 and the invention name of 'a method for obtaining customized background real-time dance video by a mobile terminal'.
Technical Field
The invention relates to the technical field of video synthesis, in particular to a method for obtaining a customized background real-time dance video by a mobile terminal.
Background
Currently, various types of small videos are widely liked by users, wherein the types of videos comprise various types of music, science and technology, dance and the like, and audience groups are wide. With the rapid development of small videos, various video editing and processing functions are becoming more and more rich.
However, the processing and synthesis of video is currently severely dependent on the computer for its processing power. Especially dance-like small videos, the recorded environment directly leads to the quality of the recorded video. Therefore, a technology for customizing dance video background in real time at a mobile terminal is needed to solve the above-mentioned problems.
Disclosure of Invention
Aiming at the technical problems in the prior art, the invention provides a method for obtaining a customized background real-time dance video by a mobile terminal, which comprises the following steps: acquiring a user video under a solid background; responding to interactive operation of a user, and selecting a background template; acquiring a background-free user dance video by using the color median of the solid background; acquiring a synthesized dance video in real time based on a background template and a non-background user dance video; wherein the obtaining the background-free user dance video by utilizing the color median of the solid background comprises: sampling color values of a plurality of positions of the solid color background to obtain color median values of the color values of the plurality of positions; expanding a color value range with the color median as a center; removing the background image in the color value range; displaying the color median to a user; and adjusting the removed background color range in real time in response to the user manually adjusting the color value range.
The method as described above, further comprising: nine-lattice convolution is carried out on the solid background image to determine boundary pixels, and the transparent values of the boundary pixels are adjusted according to the convolution values.
The method as described above, wherein the mobile terminal automatically or manually adjusts the focus and/or exposure parameters before recording the video; when recording video, focus and/or exposure parameters are locked.
The method as described above, further comprising: one or more attributes of the composite dance video are edited in response to user interaction prior to recording the video.
The method as described above, further comprising: editing the synthesized dance video into a multi-person dance team.
The method as described above, further comprising: wherein one or more of beautifying, slimming, and filter operations are performed on a user in the composite dance video.
The method as described above, further comprising: one or more of a size, a position, a clipping width, and a near-far operation of the background-free user dance video are adjusted.
A method as above wherein the color values are defined using an HSV color model.
The method as above wherein the colors of the solid background include green, blue and red.
According to another aspect of the present application, a mobile terminal is provided, including: the camera module is configured to acquire user videos in a solid-color background; an interaction module configured to select a background template in response to a user operation; the video processing module is configured to acquire a background-free user dance video by utilizing the color median of the solid background; the method comprises the steps of obtaining a synthesized dance video in real time based on a background template and a non-background user dance video; the video processing module comprises a video matting unit which is configured to acquire a background-free user dance video based on the user video under the solid background from the camera module; the video matting unit is further configured to: sampling color values of a plurality of positions of the solid color background to obtain color median values of the color values of the plurality of positions; expanding a color value range with the color median as a center; removing the background image in the color value range; displaying the color median to a user; and adjusting the removed background color range in real time in response to the user manually adjusting the color value range.
A mobile terminal as described above, wherein the video processing module comprises: a template providing unit configured to provide a plurality of background templates; and a video synthesis unit configured to obtain a synthesized dance video using the background template and the user dance video without the background.
The mobile terminal as described above, wherein the video matting unit is further configured to determine boundary pixels by nine-lattice convolution on the solid background image and adjust the transparency values of the boundary pixels according to the convolution values.
A mobile terminal as described above, wherein the camera module is further configured to automatically or manually adjust focus and/or exposure parameters by a user prior to recording a video; when recording video, focus and/or exposure parameters are locked.
According to the method and the device, when the mobile terminal records the video, the customized background template is synthesized in the recorded video in real time, so that the operation steps of replacing the background by the video are simplified, the threshold for replacing the background is reduced, and the method and the device are suitable for most mobile terminals and improve the playability of dance video.
Drawings
Preferred embodiments of the present invention will be described in further detail below with reference to the attached drawing figures, wherein:
FIG. 1 is a schematic diagram of a system for generating a team composite dance video according to one embodiment of the present invention;
FIG. 2 is a schematic diagram of a mobile terminal according to one embodiment of the invention;
FIG. 3 is a schematic diagram of a server according to one embodiment of the invention;
FIG. 4 is a flowchart of a method for a mobile terminal to obtain customized background real-time dance video according to one embodiment of the present invention;
FIGS. 5A-E are graphical user interfaces for a mobile terminal to generate a team combination dance video;
FIG. 6 is a flowchart of a method for generating a dance team combination dance video by a mobile terminal according to one embodiment of the present invention;
FIG. 7 is a team formation schematic diagram according to one embodiment of the present invention;
FIG. 8 is a schematic diagram of a scene of synthesizing dance video according to one embodiment of the invention; and
FIG. 9 is a method for generating a dance video with light and dark environment adaptation at a mobile terminal according to one embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In the following detailed description, reference is made to the accompanying drawings, which form a part hereof, and in which is shown by way of illustration specific embodiments in which the application may be practiced. In the drawings, like reference numerals describe substantially similar components throughout the different views. Various specific embodiments of the present application are described in sufficient detail below to enable those skilled in the art to practice the teachings of the present application. It is to be understood that other embodiments may be utilized or structural, logical, or electrical changes may be made to the embodiments of the present application.
Fig. 1 is a schematic diagram of a system for generating a team combined dance video according to an embodiment of the present invention. As shown, the system includes one or more mobile terminals 10 and a server 20, which are communicatively coupled via a wireless network. The mobile terminal 10 includes a mobile phone, a tablet computer, a personal entertainment device, or a combination thereof, and is used for recording dance video, and changing the background of the video into a background template in real time based on a specific application. After the recording is completed, the user can store the video to a personal account of the server by utilizing a wireless network, and can share the dance video to be shown to more people for watching.
Fig. 2 is a schematic diagram of a mobile terminal according to an embodiment of the present invention. As shown in the figure, the mobile terminal 10 includes a camera module 101, a video processing module 102, a communication module 103, and an interaction module 104, where the video processing module 102 is electrically connected to the camera module 101, the communication module 103, and the interaction module 104, and the camera module 101 is electrically connected to the interaction module 104.
The camera module 101 is one or more cameras, and is configured to obtain a user video in a solid-color background. The camera can be a front camera of a mobile terminal or a rear camera. The camera comprises automatic focusing and manual focusing functions, so that the requirements of different users are met.
The video processing module 102 includes a template providing unit 1021, a video matting unit 1022, a property editing unit 1023, and a video synthesizing unit 1024, wherein the template providing unit 1021 is electrically connected with the interaction module 104 and the property editing unit 1023, respectively; the video matting unit 1022 is electrically connected to the image capturing module 101 and the attribute editing unit 1023, respectively; the property editing unit 1023 is electrically connected to the interaction module 104 and the video composing unit 1024, respectively, and the video composing unit 1024 is electrically connected to the communication module 103.
The template providing unit 1021 is configured to provide a plurality of background templates and/or a team template, where the background templates may be a video or a picture. Background template types include, but are not limited to, scenery, indoor, stage, holiday, atmosphere, and green veil, among others. The background of the green screen type is used for customizing a background template when the video is recorded in real time, can adapt to the whole screen, is synthesized with the scratched background-free user dance video, and cannot see the real recording environment. Dynamic elements such as birds, waterfalls, rowing, etc. can be added to the background templates of the green curtain type, so that video contents are enriched.
The video matting unit 1022 receives an image from the camera module 101, and acquires a background-free user dance video using a color median of a solid background. In some embodiments, the solid background colors include, but are not limited to, green, blue, and red, and those skilled in the art will appreciate that solid background that can be used to capture background-free user dance video may be used in the schemes of the present application.
The property editing unit 1023 receives an instruction from the interaction module 104 to adjust the video properties. Such as adjusting parameters such as hue, contrast, and aspect ratio of the background-free user dance video and/or background templates. In response to a user interaction generating instruction on the interaction module 104, the user in the video is beautified or adjusted to display stature or adjust video hue, and the like.
In some embodiments, the property editing unit 1023 further includes functions to adjust the size, position, distance, and combination team of the background-free user dance video. When a user records a video, the angle, the recording position and the like of the mobile terminal and the non-background user dance video cannot show the optimal shooting effect, so that the size, the position and the distance of the acquired non-background user dance video in a background template can be adjusted, and the fusion with the background template is more natural. The size of the background-free user dance video is adjusted; the distance is the distance effect of simulating and adjusting the camera; and (3) adjusting the position of the background-free user dance video in the background template. The dance team is that when the user shoots dance video, a plurality of images are copied from the dance video of the user without the background, the video of dance of a plurality of people simultaneously is combined according to the station position of the dance team, the movements in dance are uniform, and the observability of the dance video is improved. The team change modes comprise a 1-person team mode, a 2-person team mode, a parallel 3-person team mode, a triangle 3-person team mode and the like.
The video synthesis unit 1024 is configured to obtain a synthesized dance video from the background template, the dance team template, and the non-background user dance video in real time. The video synthesis unit 1024 is electrically connected to the interaction module 104, and sends the synthesized video to the interaction module 104, so that the user can watch the synthesized video on the interaction module 104 in real time. The video synthesizing unit 1024 may further send the synthesized video to the server 20 through the communication module 103, so as to share more people to watch the video made by themselves.
The communication module 103 can include one or more wired or wireless communication interfaces. Such as a communications interface network interface card, a wireless modem, or a wired modem. In one application, the communication module 103 can be a WiFi modem. In other applications, the communication module 103 can be a 3G modem, a 4G modem, an LTE modem, a bluetooth component, a radio frequency receiver, an antenna, or a combination thereof. The communication module 103 is capable of wireless communication with the server 20 and the mobile device 10 is capable of transmitting information or receiving packets or messages using the communication module 103.
The interaction module 104 is configured to present a Graphical User Interface (GUI) for interaction with a user, and may also present functional content to the user, such as displaying a background template and editing a video attribute, or a text box or link, etc. The user may select a background template by a swipe, click, or the like gesture on the interactive module 104. The interaction module 104 may be a touch screen display of a Liquid Crystal Display (LCD), which may be a touch screen display of an Organic Light Emitting Diode (OLED).
Fig. 3 is a schematic diagram of a server according to an embodiment of the present invention. As shown, the server includes one or more processing modules 301, a communication module 302, and a storage module 303.
The processing module 301 can include one or more Central Processing Units (CPUs), graphics Processing Units (GPUs), application Specific Integrated Circuits (ASICs), field Programmable Gate Arrays (FPGAs), or combinations thereof. The processing module 301 is capable of executing software or computer readable instructions stored in the storage module 303 to perform the methods or operations described herein. The processing module 301 can be implemented in a number of different ways. For example, the processing module 301 can include one or more embedded processors, processor cores, microprocessors, logic circuits, hardware Finite State Machines (FSMs), digital Signal Processors (DSPs), or combinations thereof.
The communication module 302 can include one or more wired or wireless communication interfaces. Such as a communications interface network interface card, a wireless modem, or a wired modem. In one application, the communication module 302 can be a WiFi modem. In other applications, the communication module 302 can be a 3G modem, a 4G modem, an LTE modem, a bluetooth component, a radio frequency receiver, an antenna, or a combination thereof.
The storage module 303 can store software, data, logs, or a combination thereof. The storage module 303 can be an internal memory or an external memory. For example, the memory can be volatile memory or nonvolatile memory, such as nonvolatile random access memory (NVRAM), flash memory, disk memory, or volatile memory such as Static Random Access Memory (SRAM).
FIG. 4 is a flowchart of a method for a mobile terminal to obtain customized background real-time dance video according to one embodiment of the present invention; FIGS. 5A-E are graphical user interfaces for a mobile terminal to obtain customized background real-time dance video. In step 410, a user video in a solid background is acquired. Referring to fig. 5A, in response to a user selecting an accompaniment, a camera of a mobile device acquires a user video in a solid background. In some embodiments, the user may adjust the location and shooting distance of the mobile device, etc., according to the user video displayed on the interactive apparatus 104.
In step 420, a background template is selected in response to a user operating on the interactive module. Referring to FIG. 5B, the user selects the "background" button at the interactive module, and a list of background templates appears. The background templates that can be customized are in the type of "green curtain". Not all background templates may be customized background templates, which need to meet certain requirements, such as a background that cannot be transparent, and need to completely cover the interaction module, etc. In response to a user selecting any background template in the "green curtain" background template list, the selected background template is displayed on the interactive module.
At step 430, the color median of the solid background is used to obtain a background-free user dance video. In some embodiments, the specific steps of obtaining the background-free user dance video are:
1. sampling color values of a plurality of positions of the solid color background to obtain color median values of the color values of the plurality of positions;
2. expanding a color value range with the color median as a center; and
3. background images within the range of color values are removed.
In some embodiments, the color values are defined using an HSV color model. The color median may be a median obtained by sorting color values of a plurality of positions according to the sizes and taking a plurality of color values according to the sorting. When a user records a video, sometimes, due to the fact that recording ambient light irradiates on a solid background unevenly, color values of the solid background of the recorded video are inconsistent, and the picture matting effect is affected. Therefore, a plurality of color values of different solid-color background positions are obtained, the median is obtained, and a certain range is expanded, so that the quality of the dance video of the background-free user is improved. Wherein the extended range is the numerical empirical range of a certain color (green/blue/red, etc.) within the HSV color space.
In some embodiments, nine-lattice convolution is performed on the solid background image to determine boundary pixels and the transparent values of the boundary pixels are adjusted according to the convolution values, so that the jaggy feeling of the matting is effectively reduced. According to the method and the device, the convolutional neural network is utilized to carry out boundary judgment on the video pixels, different transparent values are set for different pixels through the convolutional values, and the acquired target user image is more complete and natural.
At step 440, the video attributes are edited in response to the user operating on the interactive module. Referring to fig. 5B, sometimes, the mobile terminal automatically acquires that the quality of the dance video of the background-free user is low, and a part of green background still exists. At this time, the user can click the "green curtain" function key of the screen, enter fig. 5C, and can adjust the slider of the "color key", and as the slider moves, the green part in the screen increases or decreases, and the user stops sliding when the green part is completely disappeared or at least, and at this time, the best background-free user dance video is presented. In some embodiments, the mobile end presents the color median to the user; and adjusting the removed background color range in real time in response to the user manually adjusting the color value. If the sliding block on the color key stays at the color median, the user can present the optimal background-free user dance video only by fine tuning the color value, and the user adjustment range is reduced. Referring to fig. 5D, sliding the sliders with different functions may adjust one or more of the size, position, clipping width, and distance of the background-free user dance video to fit the background template.
In some embodiments, editing the video attributes further includes one or more of beautifying, slimming, and filter operations for a user in the composite dance video. The user can perform personalized adjustment on the user image in the video according to the preference of the user, so that the user in the video is more beautiful.
At step 450, a composite choreography video is obtained in real-time based on the background template and the non-background user choreography video. After the user finishes editing the video attribute, responding to the user to click a shooting function key, starting to record the video by the mobile terminal, synthesizing the background template and the non-background user dance video into the video in real time, and storing the synthesized video in the mobile terminal after the user clicks the shooting function key.
In some embodiments, before recording the video, the mobile terminal can automatically or manually adjust focusing and exposure parameters by a user, so that the camera achieves an optimal shooting effect; when video is recorded, the mobile terminal locks focusing and exposure parameters, and ensures that a background-free user dance video with better quality is obtained in the recording process. If the camera is in the recording process, the automatic focusing or automatic exposure function is opened, the color value of the solid background acquired by the mobile terminal may change, so that the acquired non-background user dance video contains the solid background, and the problem of reduced quality of the matting is caused.
Referring to fig. 5E, in response to a user clicking a "split screen" function key on the left side of the screen, the composite dance video may be edited into a multi-person dance team. In some embodiments, a variety of team templates are available for user selection in the split screen function, with the team position fixed. In other embodiments, the user can customize the positions of the non-background user dance videos, and the user can randomly select the number and the positions of the non-background user dance videos to design the shape of the dance team. Wherein in the custom team, the number of background-free user dance videos is 8 at most.
In order to solve the problem that the background-free user dance video is not natural in lamination in the synthetic dance video, the background-free user dance videos of different layers are calculated by using a pixel mixing mode, so that the overlapping real visual effect is achieved. However, in the process, each pixel of the adjacent layer needs to be computationally rendered, which causes a problem of large computation amount. The method is difficult to popularize at the mobile terminal with limited calculation amount, so the method is designed, and can be used for carrying out mixed optimization on the background-free user dance videos of different levels on the mobile terminal, and has the advantages of small calculation amount, high response speed and the like.
FIG. 6 is a flowchart of a method for generating a dance team combination dance video by a mobile terminal according to one embodiment of the present invention. In step 610, a background-free user dance video is acquired against a solid background. The method for obtaining the dance video of the user without background is the same as the step for obtaining the dance video of the user without background in the method of fig. 4, and will not be described herein.
In step 620, a combination of the background-free user dance video is determined according to the user-selected dance team template. Before starting to record the video, the user can operate on the interaction module of the mobile terminal to select the team template. In some embodiments, a team template list is displayed on the interactive module, and after the user selects the team template, the team form is displayed on the interactive module in real time; in other embodiments, the user clicks a function button on the interactive module, which changes the team formation in real time. The team template comprises a plurality of team forms, different team forms and corresponding layers are different.
The system correspondingly displays the background-free user dance videos in different combination modes. The team formation can be used for realizing three-dimensional space in the video and dividing the space into front, back, left, right and other directions. The team formation can be combined into triangles, pyramids and the like, so that the visual effect of the video is improved.
FIG. 7 is a schematic diagram of a team formation, according to one embodiment of the invention. In general, the more dance people in a team, the more levels. Referring to fig. 7, taking the front 1 and rear 2 dancer groups as an example, the front 1 dancer is level 0, and the rear 2 dancers in parallel are level 1, wherein the display scale of level 0 is greater than level 1. The user watching the video can more clearly enjoy the dance video, and the watching satisfaction degree is improved. The dancers in the level 0 and the level 1 copy the background-removed user dance video into a plurality of dancer videos and distribute the dancer videos at different positions, so that the dancer actions of different levels are completely consistent. It will be appreciated by those skilled in the art that the levels in the team template are not limited to 2 levels, but may be 3 or more levels.
At step 630, one or more overlapping portions of the background-free user dance video are determined. In some embodiments, the priority of each layer formed by the non-background user dance video is determined according to the team formation in the team template, wherein the layers of the non-background user dance video positioned at the back are preferentially drawn. In the team formation, the background-free user dance video needs to be drawn layer by layer. In some embodiments, background-free user dance video may be drawn sequentially in a back-to-front order. The overlapping portion of the background-free user dance video is a portion where background-free user dance videos of different levels are overlapped, such as an overlapping portion 710 and an overlapping portion 720 in fig. 7. Further, the method and the device can determine one or more overlapped part boundaries, thereby achieving the purpose of reducing the calculated amount and improving the rendering speed.
In some embodiments, the pixel coordinate values of the background-free user dance video are utilized to determine the boundaries of one or more overlapping portions. For example, the level 0 and level 1 hierarchical pixels are respectively determined to be coordinate values with respect to the interactive module, and the same coordinates represent the same positions on the interactive module under the same coordinate system. Therefore, by comparing the coordinate values of the user images in the plurality of background-free user dance videos, the boundary of the overlapping portion can be determined.
In other embodiments, the boundaries of one or more overlapping portions are determined using pixel color values between background-free user dance videos. Referring to fig. 7, the background-free user dance video in level 0 has a portrait portion, and pixels of the other portions are transparent. Therefore, by acquiring video pixels of a level adjacent to level 0, the boundary of the overlapping portion of the background-free user dance video can be quickly acquired. Those skilled in the art will appreciate that techniques for determining overlapping portions of dance video may be used in the schemes of the present application.
At step 640, a rendering transition process is performed on the one or more overlapping portions. Wherein the step of rendering the one or more overlapping portions comprises:
determining a boundary extension range based on the one or more overlapping portions;
the boundary extension range is 2-8 pixels extended inwards from one or more overlapped part boundaries; and
and rendering transition processing is carried out on the pixels in the extended range by using the pixel mixed mode.
Wherein the size of the boundary extension range is related to the display scale of the user dance video without background. From the above, the background-free user dance video arranged at the front level has a larger display scale than the background-free user dance video arranged at the rear level. Therefore, the boundary expansion range is determined according to the display proportion of the non-background user dance video, the rendering transition processing range can be thinned, and the quality of the synthesized dance video is improved.
In some embodiments, the rendering transition process is a color mixing special effect process. For example, pixels in an extended range are rendered by using openGL ES, and further pixels in an extended range are processed by using glBlendFunc (gl_one, gl_one_menu_src_alpha) functions. And respectively taking out the colors in the dance videos of different levels within the expansion range, multiplying the colors by a coefficient, and then mixing the colors to obtain a new color, so that the stacking relationship between the dance videos of different levels is more natural and real. The algorithm in the glBlendFunc function can be adjusted according to actual requirements, and is not limited herein.
In other embodiments, the rendering transition process is a Gaussian weighted averaging process. Gaussian weighted averaging is to replace the gray values of the original pixels after weighted averaging processing is performed by using the gray values of a plurality of pixels in the pixel neighborhood, so that the sawtooth and unnatural problems of a plurality of video laminating parts are eliminated, and the laminating parts are smooth, natural and real. It should be understood by those skilled in the art that the above-mentioned technical means for processing overlapping portions of multiple videos is only an exemplary solution of the present application, and any technical solution capable of solving the above-mentioned technical problems may be applied in the present application, which is not limited herein.
FIG. 8 is a schematic diagram of a scene of synthesizing dance video according to one embodiment of the invention. As shown, the dance video includes a background template 810 and one or more non-background user dance videos 820, wherein the background template 810 includes a dance area 811 of the user. The background template 810 is an outdoor landscape template, the light source is correspondingly set as the sun 813, the requirements of the outdoor landscape template are met, and natural scenes such as the white cloud 812, the river 814 and the tree 815 are increased, so that the background template is richer in content. The background templates are divided into various types, and can be divided into indoor scenery and outdoor scenery according to scenes; dynamic and static backgrounds can be classified according to dynamic and static types, wherein elements in the dynamic background include, but are not limited to, rivers, birds, and waterfalls; the landscape background can be classified into a mountain background, a flower cluster background, a human landscape background and the like according to the environment. In addition, the user can select a dance team template to determine the positions of a plurality of dance users in a dance area. In order to synthesize a plurality of background-free user dance videos with a background template more truly, the function of adapting the dance videos through light and shade simulation is added, and the stereoscopic impression of the videos is increased through configuring elements such as light sources in the dance videos. The specific method comprises the following steps:
FIG. 9 is a method for generating a dance video with light and dark environment adaptation at a mobile terminal according to one embodiment of the present invention. In step 910, information of a user video light source is determined according to the user video in a solid background. The method comprises the following specific steps: collecting a plurality of pixel points of solid color backgrounds in a user video under the solid color backgrounds; analyzing a plurality of pixel points of the solid color background by utilizing a neural network model in deep learning to determine a plurality of light paths; information of the user video light source is determined based on the determined plurality of light paths. Wherein the template light source information includes: one or more of brightness, location, number color, and dynamic effect of the light source. For example, a user stands in a green screen background to record video in dance, and a bulb is arranged at the upper left of the user to provide illumination for the video recorded by the user; according to the principle that light propagates along a straight line, light rays striking on a user body should be inclined downwards, and when the light strikes on the user body, the color of a pixel can change in color and brightness, so that a neural network model in deep learning is utilized to analyze pixel points of a plurality of solid-color backgrounds to determine the irradiation directions of a plurality of light paths; the position of the user video light source can be determined according to the irradiation directions of the plurality of light paths. It should be understood by those skilled in the art that only the general information of the video light source of the user needs to be determined in the present application, and no particularly accurate light source information is needed, so long as the technical solution capable of achieving the above objective can be applied in the present application, and no limitation is made herein.
In step 920, configuration information of the template light source is determined according to the team template selected by the user. Wherein, different dance team templates are provided with different template light sources. As described above, in an outdoor scene template, the template light source may be the sun; in the indoor background template, the template light source may be a bulb or the like. And the information such as the color, the lighting of the configuration template light source is different according to the characteristics of different background templates.
In step 930, information of the user video light source or configuration information of the template light source is modified. In some embodiments, the configuration information of the template light source is kept consistent with the information of the user video light source. In order to ensure that the configuration information of the template light source is consistent with the information of the user video light source, the information of the user video light source can be modified, and the configuration information of the template light source can also be modified, wherein the brightness change of the user body in the background-free user dance video is modified in response to the position change of the template light source. Preferably, the light source configuration information with small calculation amount is modified. According to the method and the device, the positions, colors, brightness and the like of the template light source and the user video light source are kept consistent, so that the synthesized dance video is more coordinated and more natural.
In some embodiments, configuration information of the template light source is modified based on the mobile-end system time. The template light source comprises a natural light source and an artificial light source. For example, in response to the user recording dance video using the mobile terminal, the system time of the mobile terminal is 12 am, the sun is set at the sky middle position, and the brightness is adjusted to the highest; when the system time is 8 pm, the sun can be replaced by a street lamp as a light source, and the color tone of the light source is adjusted to be warm. In the scheme of this application, the light that the template light source sent all is directional light, builds a bright and dark line in synthesizing dance video through the light that directional light sent, highlights the authenticity of scene.
In some embodiments, configuration information of the template light sources is modified based on accompaniment of the dance video. Music and lights enhance dance atmosphere and mobilize user emotion. When a user records dance videos, the system can adjust the information such as brightness, position, color, dynamic effect and the like of the template light source according to the playing time of accompanying music. For example, when the user plays the dance accompaniment to the climax part, the template light source flashes at a certain frequency and changes the color of the light source, so that the video interestingness is improved.
In step 940, a team template is combined with the background-free user dance video. And combining the dance team template and the background-free user dance video in response to the information of the adjusted user video light source or the configuration information of the template light source, wherein bright and dark lines change in the combined dance video according to the configuration information of the template light source. Wherein in a plurality of template light sources and/or a plurality of background-free user dance video environments, when directional light is projected on a user, the brightness of an irradiated part on the user is increased. When a plurality of template light sources are arranged in the background template, light rays can be overlapped on the non-background user dance video, so that the brightness of the overlapped part is larger than that of the non-overlapped part. In some embodiments, the color of the light emitted by the template light source is mixed with the color of the light irradiated on the background-free user dance video part by using a pixel mixing mode for rendering.
In addition, the light source creates a correspondingly shaped shadow on the ground surface according to the contours of the light blockage. For example, the light source emits directional light to illuminate the user, and the viewer can see bright portions and shadow portions in the environment, so that the whole environment is more stereoscopic and real. Wherein, the number of the light sources is increased by one, and the generation of shadows is doubled; multiple background-free user dance videos may create multiple shadows.
In some embodiments, the configuration information of the template light source and/or the combination mode of the background-free user dance video are utilized to simulate the user shadows in the background-free user dance videos; dance of multiple background-free users
Generating a shadow layer by using user shadows in the video; and superposing the shadow layer with a plurality of non-background user dance videos 5 and background templates to synthesize. Wherein, the superposition mixing processes a plurality of template light sources and/or a plurality of user shadows generated by the background-free user dance video to generate the color of the superposition part. The method and the device are used for carrying out superposition mixing treatment on superposition of a plurality of light rays and superposition of a plurality of shadow parts, so that the brightness of the whole ambient light is obvious, a very real dance environment is built, and the use experience of a user is improved.
The above-described embodiments are provided for illustrating the present invention and are not intended to be limiting, and various changes and modifications may be made by one of ordinary skill in the pertinent art without departing from the scope of the present invention, so that all equivalent embodiments shall fall within the scope of the present disclosure.
Claims (13)
1. A method for a mobile terminal to obtain a customized background real-time dance video comprises the following steps:
acquiring a user video under a solid background;
responding to interactive operation of a user, and selecting a background template;
acquiring a background-free user dance video by using the color median of the solid background; and
acquiring a synthesized dance video in real time based on a background template and a non-background user dance video;
wherein the obtaining the background-free user dance video by utilizing the color median of the solid background comprises:
sampling color values of a plurality of positions of the solid color background to obtain color median values of the color values of the plurality of positions;
expanding a color value range with the color median as a center;
removing the background image in the color value range;
displaying the color median to a user; and adjusting the removed background color range in real time in response to the user manually adjusting the color value range.
2. The method of claim 1, further comprising: nine-lattice convolution is carried out on the solid background image to determine boundary pixels, and the transparent values of the boundary pixels are adjusted according to the convolution values.
3. The method of claim 1, wherein the mobile terminal automatically or manually adjusts focus and/or exposure parameters by a user prior to recording the video; when recording video, focus and/or exposure parameters are locked.
4. The method of claim 1, further comprising: one or more attributes of the composite dance video are edited in response to user interaction prior to recording the video.
5. The method of claim 4, further comprising: editing the synthesized dance video into a multi-person dance team.
6. The method of claim 4, further comprising: wherein one or more of beautifying, slimming, and filter operations are performed on a user in the composite dance video.
7. The method of claim 4, further comprising: one or more of a size, a position, a clipping width, and a near-far operation of the background-free user dance video are adjusted.
8. The method of claim 1, wherein the color values are defined using an HSV color model.
9. The method of claim 1, wherein the colors of the solid background include green, blue, and red.
10. A mobile terminal, comprising:
the camera module is configured to acquire user videos in a solid-color background;
an interaction module configured to select a background template in response to a user operation;
the video processing module is configured to acquire a background-free user dance video by utilizing the color median of the solid background; the method comprises the steps of obtaining a synthesized dance video in real time based on a background template and a non-background user dance video;
the video processing module comprises a video matting unit which is configured to acquire a background-free user dance video based on the user video under the solid background from the camera module; the video matting unit is further configured to:
sampling color values of a plurality of positions of the solid color background to obtain color median values of the color values of the plurality of positions;
expanding a color value range with the color median as a center; and
removing the background image in the color value range;
displaying the color median to a user; and adjusting the removed background color range in real time in response to the user manually adjusting the color value range.
11. The mobile terminal of claim 10, wherein the video processing module comprises:
a template providing unit configured to provide a plurality of background templates; and
a video synthesis unit configured to obtain a synthesized dance video using the background template and the user dance video without the background.
12. The mobile end of claim 10, wherein the video matting unit is further configured to nine-gram convolve a solid background image to determine boundary pixels and adjust transparency values of boundary pixels according to convolution values.
13. The mobile end of claim 12, wherein the camera module is further configured to automatically or manually adjust focus and/or exposure parameters by a user prior to recording video; when recording video, focus and/or exposure parameters are locked.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211667026.2A CN116055800A (en) | 2020-09-10 | 2020-09-10 | Method for mobile terminal to obtain customized background real-time dance video |
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211667026.2A CN116055800A (en) | 2020-09-10 | 2020-09-10 | Method for mobile terminal to obtain customized background real-time dance video |
| CN202010947691.1A CN112087664A (en) | 2020-09-10 | 2020-09-10 | Method for obtaining customized background real-time dance video by mobile terminal |
Related Parent Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010947691.1A Division CN112087664A (en) | 2020-09-10 | 2020-09-10 | Method for obtaining customized background real-time dance video by mobile terminal |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116055800A true CN116055800A (en) | 2023-05-02 |
Family
ID=73736314
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211667026.2A Pending CN116055800A (en) | 2020-09-10 | 2020-09-10 | Method for mobile terminal to obtain customized background real-time dance video |
| CN202010947691.1A Pending CN112087664A (en) | 2020-09-10 | 2020-09-10 | Method for obtaining customized background real-time dance video by mobile terminal |
Family Applications After (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010947691.1A Pending CN112087664A (en) | 2020-09-10 | 2020-09-10 | Method for obtaining customized background real-time dance video by mobile terminal |
Country Status (1)
| Country | Link |
|---|---|
| CN (2) | CN116055800A (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114286163B (en) * | 2021-12-24 | 2024-02-13 | 苏州亿歌网络科技有限公司 | Sequence chart recording method, device, equipment and storage medium |
| CN114710703A (en) * | 2022-03-29 | 2022-07-05 | 稿定（厦门）科技有限公司 | Live broadcast method and device with variable scenes |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106204426A (en) * | 2016-06-30 | 2016-12-07 | 广州华多网络科技有限公司 | A kind of method of video image processing and device |
| CN110298783B (en) * | 2019-06-05 | 2024-04-19 | 苏州常立科技有限公司 | Image matting method and system |
| CN110335279A (en) * | 2019-07-02 | 2019-10-15 | 武汉瑞宏峰科技有限公司 | Real-time green curtain is scratched as method, apparatus, equipment and storage medium |
| CN110290425B (en) * | 2019-07-29 | 2023-04-07 | 腾讯科技（深圳）有限公司 | Video processing method, device and storage medium |
| CN110503725B (en) * | 2019-08-27 | 2023-07-14 | 百度在线网络技术（北京）有限公司 | Image processing method, device, electronic equipment and computer readable storage medium |
-
2020
- 2020-09-10 CN CN202211667026.2A patent/CN116055800A/en active Pending
- 2020-09-10 CN CN202010947691.1A patent/CN112087664A/en active Pending
Also Published As
| Publication number | Publication date |
|---|---|
| CN112087664A (en) | 2020-12-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Rana et al. | Deep tone mapping operator for high dynamic range images | |
| CN109644224A (en) | System and method for capturing digital picture | |
| US20070146360A1 (en) | System And Method For Generating 3D Scenes | |
| CN113436343A (en) | Picture generation method and device for virtual studio, medium and electronic equipment | |
| CN110090437A (en) | Video acquiring method, device, electronic equipment and storage medium | |
| CN116055800A (en) | Method for mobile terminal to obtain customized background real-time dance video | |
| US20220343590A1 (en) | System and techniques for lighting adjustment for an immersive content production system | |
| CN112087662B (en) | Method for generating dance combination dance video by mobile terminal and mobile terminal | |
| Mirbauer et al. | SkyGAN: Towards Realistic Cloud Imagery for Image Based Lighting. | |
| CN104899913B (en) | A kind of fluid special effect making method true to nature under virtual stage environment | |
| CN112087663B (en) | Method for generating dance video with adaptive light and shade environment by mobile terminal | |
| US11887251B2 (en) | System and techniques for patch color correction for an immersive content production system | |
| US20080247727A1 (en) | System for creating content for video based illumination systems | |
| CN110136239B (en) | Method for enhancing illumination and reflection reality degree of virtual reality scene | |
| JP7387029B2 (en) | Single-image 3D photography technology using soft layering and depth-aware inpainting | |
| CN112866507B (en) | Intelligent panoramic video synthesis method and system, electronic device and medium | |
| CN112969007B (en) | Video post-production method oriented to virtual three-dimensional background | |
| Bredow et al. | Renderman on film | |
| CN114902277A (en) | System and method for processing shadows on portrait image frames | |
| CN115546041B (en) | Training method of light supplementing model, image processing method and related equipment thereof | |
| CN216248725U (en) | Intelligent high-dynamic-range full-color light matrix | |
| US11762481B2 (en) | Light capture device | |
| AU2022202424B2 (en) | Color and lighting adjustment for immersive content production system | |
| CN116245741B (en) | Image processing method and related device | |
| US20220343562A1 (en) | Color and lighting adjustment for immersive content production system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |