WO2024067157A1 - 生成特效视频的方法、装置、电子设备及存储介质 - Google Patents

生成特效视频的方法、装置、电子设备及存储介质 Download PDF

Info

Publication number
WO2024067157A1
WO2024067157A1 PCT/CN2023/119023 CN2023119023W WO2024067157A1 WO 2024067157 A1 WO2024067157 A1 WO 2024067157A1 CN 2023119023 W CN2023119023 W CN 2023119023W WO 2024067157 A1 WO2024067157 A1 WO 2024067157A1
Authority
WO
WIPO (PCT)
Prior art keywords
audio
video frame
target
target object
processed
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2023/119023
Other languages
English (en)
French (fr)
Inventor
马佳欣
温思敬
梁冰雁
王晓婵
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Zitiao Network Technology Co Ltd
Original Assignee
Beijing Zitiao Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Zitiao Network Technology Co Ltd filed Critical Beijing Zitiao Network Technology Co Ltd
Priority to US19/115,753 priority Critical patent/US20260105705A1/en
Publication of WO2024067157A1 publication Critical patent/WO2024067157A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • G06T19/20—Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161—Detection; Localisation; Normalisation
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H1/00—Details of electrophonic musical instruments
    • G10H1/0008—Associated control or indicating means
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H1/00—Details of electrophonic musical instruments
    • G10H1/36—Accompaniment arrangements
    • G—PHYSICS
    • G11—INFORMATION STORAGE
    • G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
    • G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
    • G11B27/02—Editing, e.g. varying the order of information signals recorded on, or reproduced from, record carriers
    • G11B27/031—Electronic editing of digitised analogue information signals, e.g. audio or video signals
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
    • G06T2219/20—Indexing scheme for editing of 3D models
    • G06T2219/2004—Aligning objects, relative positioning of parts
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
    • G10H2210/005—Musical accompaniment, i.e. complete instrumental rhythm synthesis added to a performed melody, e.g. as output by drum machines
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
    • G10H2210/031—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal
    • G10H2210/056—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal for extraction or identification of individual instrumental parts, e.g. melody, chords, bass; Identification or separation of instrumental parts by their characteristic voices or timbres

Definitions

  • the embodiments of the present disclosure relate to image processing technology, for example, to a method, device, electronic device and storage medium for generating special effect videos.
  • the present disclosure provides a method, device, electronic device and storage medium for generating special effect videos, which realize special effect processing of audio, thereby enriching the special effect display effect and further improving the technical effect of user experience.
  • an embodiment of the present disclosure provides a method for generating a special effects video, the method comprising:
  • the mixing condition When it is detected that the mixing condition is met, determining at least one mixing audio corresponding to at least one target object in the video frame to be processed; wherein the video frame to be processed is a video frame collected in real time or a video frame in a recorded video;
  • a special effect video frame corresponding to the video frame to be processed is determined.
  • an embodiment of the present disclosure further provides a device for generating a special effects video, the device comprising:
  • the mixed audio determination module is configured to determine at least one mixed audio corresponding to at least one target object in the video frame to be processed when the mixed audio condition is detected to be met; wherein the video frame to be processed It is a video frame captured in real time or a video frame in a recorded video;
  • a target audio determination module configured to determine a target audio of a video frame to be processed based on at least one mixed audio and audio information of at least one target object
  • the special effect video frame determination module is configured to determine a special effect video frame corresponding to a video frame to be processed based on a target audio and at least one target object.
  • an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:
  • processors one or more processors
  • a storage device for storing one or more programs
  • the one or more processors When one or more programs are executed by one or more processors, the one or more processors implement a method for generating a special effects video as described in any of the embodiments of the present disclosure.
  • the embodiments of the present disclosure further provide a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute a method for generating a special effects video as described in any of the embodiments of the present disclosure.
  • FIG1 is a flow chart of a method for generating special effects video provided by an embodiment of the present disclosure
  • FIG2 is a user display interface of an application program for generating special effect videos provided by an embodiment of the present disclosure
  • FIG3 is a schematic diagram of an interface for generating special effect videos provided by an embodiment of the present disclosure
  • FIG4 is a flow chart of another method for generating special effects video provided by an embodiment of the present disclosure.
  • FIG5 is a flow chart of another method for generating special effects video provided by an embodiment of the present disclosure.
  • FIG6 is a schematic diagram of a display position of at least one target object provided by an embodiment of the present disclosure.
  • FIG7 is a schematic diagram of another display position of at least one target object provided by an embodiment of the present disclosure.
  • FIG8 is a schematic diagram of another display position of at least one target object provided by an embodiment of the present disclosure.
  • FIG9 is a schematic diagram of a display position of a segmented image provided by an embodiment of the present disclosure.
  • FIG10 is a schematic diagram of another segmented image display position provided by an embodiment of the present disclosure.
  • FIG11 is a schematic diagram of another display position of a segmented image provided by an embodiment of the present disclosure.
  • FIG12 is a flow chart of another method for generating special effects video provided by an embodiment of the present disclosure.
  • FIG13 is a schematic diagram of a display position of a three-dimensional (3D) microphone provided in an embodiment of the present disclosure
  • FIG14 is a schematic diagram of the structure of a device for generating special effect videos provided by an embodiment of the present disclosure
  • FIG. 15 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure.
  • the types, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.
  • a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information.
  • the user can autonomously choose whether to submit the operation to the user according to the prompt message.
  • Software or hardware such as electronic devices, applications, servers or storage media provide personal information.
  • the prompt information in response to receiving an active request from the user, may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form.
  • the pop-up window may also carry a selection control for the user to choose "agree” or “disagree” to provide personal information to the electronic device.
  • the data involved in this technical solution shall comply with the requirements of relevant laws, regulations and relevant provisions.
  • the technical solution of the present disclosure can be applied to any scenario that requires special effects display or special effects processing, such as when applied to the video shooting process, special effects processing can be performed on the target object being shot; it can also be applied after the video shooting process, for example, after shooting a video with a camera built into the terminal device, the pre-shot video can be displayed with special effects.
  • the target object can be a user or any object that can send audio information.
  • the technical method provided by the embodiments of the present disclosure can be applied in the scenario of real-time acquisition or in the scenario of post-processing.
  • the scenario of real-time acquisition it can be understood that each time a video frame is acquired, the video frame is used as a video frame to be processed, and the special effect video frame corresponding to the video frame to be processed is determined based on the technical method provided by the embodiments of the present disclosure; in the scenario of post-processing, each video frame in the uploaded video can be used as a video frame to be processed in turn.
  • the processing of a video frame is taken as an example for explanation, and the processing of the remaining video frames can repeat the steps provided by the embodiments of the present disclosure.
  • the device for executing the method for generating special effects video provided by the embodiment of the present disclosure can be integrated into the application software that supports the special effects video processing function, and the software can be installed in the electronic device.
  • the electronic device can be a mobile terminal or a personal computer (PC), etc.
  • the application software can be a type of software for image/video processing. The specific application software will not be described one by one here, as long as the image/video processing can be realized.
  • the device for executing the method for generating special effects video provided by the embodiment of the present disclosure can also be a specially developed application program to realize the software for adding special effects and displaying the special effects, or it can be integrated in the corresponding page, and the user can realize the processing of the special effects video through the page integrated in the PC.
  • FIG1 is a flow chart of a method for generating special effect video provided by an embodiment of the present disclosure.
  • the embodiment of the present disclosure is applicable to the case of performing special effect processing on audio.
  • the method can be executed by a device for generating special effect video.
  • the device can be implemented in the form of software and/or hardware.
  • the device can be implemented in the form of electronic device.
  • the electronic device may be a mobile terminal, a PC or a server, etc.
  • the technical solution provided by the embodiment of the present disclosure may be executed by the server, or by the client, or by the client and the server in cooperation.
  • the method comprises:
  • the audio mixing condition may be understood as a condition for determining whether special effects processing needs to be performed on the audio of the video frame to be processed.
  • the mixing condition may include multiple situations, and whether to process the audio information in the to-be-processed video frame may be determined based on whether the current trigger operation satisfies the corresponding situation.
  • the mixing conditions may include: triggering a special effect prop corresponding to the mixing effect; including at least one target object in the display interface; triggering a shooting control; detecting a recorded video uploaded by a triggered video processing control.
  • the first way to determine the mixing condition is to trigger the special effect props corresponding to the mixing special effect, which can be understood as: based on the technical method provided by the embodiment of the present disclosure, the program code or processing data is compressed and processed so that it is integrated into some application software as a special effect package as a special effect prop.
  • the special effect prop is triggered, it means that the audio in the collected video frame to be processed needs to be processed with special effects, and the mixing condition is met at this time.
  • the second way to determine the sound mixing condition is that the display interface includes at least one target object, whether it is a real-time captured video frame or a non-real-time video frame, as long as the in-camera picture is detected, that is, the target object is included in the video frame to be processed, then the sound mixing condition is considered to be met.
  • the target object can be pre-set.
  • the target object can be a user, and as long as the user is detected in the display interface, the computer considers that the sound mixing condition is met.
  • the third way to determine the mixing condition is to trigger the shooting control.
  • the shooting control can be used as a trigger condition, wherein the shooting control is pre-written.
  • clicking the shooting control indicates that the mixing condition is met.
  • the captured video frame to be processed includes audio content, special effects processing is required for the audio.
  • the fourth way to determine the mixing condition is to detect the recorded video uploaded by the triggered video processing control.
  • This solution can not only achieve the effect of real-time processing, but also perform post-processing.
  • the uploaded recorded video is received, it means that the video needs to be processed, and the video can be processed with special effects based on the method of the embodiment of the present disclosure.
  • the video frame collected in real time and the video frame in the recorded video there are mainly two methods involved: the video frame collected in real time and the video frame in the recorded video.
  • the two determination methods may include: Multiple mixing conditions.
  • the advantage of this setting is that no matter how the user determines the video frame to be processed, the mixed audio corresponding to the target object in the video frame to be processed can be determined through multiple mixing conditions, making the application scope of this solution wider.
  • the video frames to be processed can be determined based on real-time video or non-real-time video. As long as the mixing conditions are met, the video frames of the real-time captured video or uploaded video can be processed in sequence, and each video frame can be used as a video frame to be processed. Another situation is that if some video frames are processed with special effects under selectable conditions, each selected video frame can be used as a video frame to be processed.
  • the target object is the user presented in the video frame to be processed.
  • the number of target objects can be one or more, and the number of target objects can be preset according to the actual situation. For example, if the preset setting is to use all objects in the frame as target objects, the number of target objects corresponds to the number of users in the frame; if only some specific users need to be processed with special effects, the facial image corresponding to the object can be uploaded in advance, so that when multiple display objects are included in the frame, the target object can be determined based on the uploaded facial information and the facial information of the display object.
  • determine the target object based on the trigger operation of the target user on the display interface For example, if there are multiple objects on the display interface, the object triggered and selected by the target user can be used as the target object. That is, only the target object triggered and selected needs to be processed with special effects.
  • Mixing can be understood as integrating sounds from multiple sources into a stereo track or a mono track.
  • the sources of multiple sounds can be audios of different parts corresponding to different users. Therefore, mixed audio can be understood as audio corresponding to different parts of the same song sung by multiple performers. For example, at least one song is pre-set, and multiple mixed audios can be determined based on multiple users.
  • Mixed audios suitable for different users can be pre-made. For example, mixed audios can be distinguished according to age stages, can be distinguished according to gender attributes, or can be distinguished according to pitch.
  • one or more songs can be pre-set and mixed audios corresponding to multiple division criteria can be determined based on the pre-set one or more songs for use by target users.
  • the number of mixed audios can correspond to the number of target objects, or the mixed audio can be selected by triggering.
  • the target user display interface for generating special effect video application program is entered, see FIG2.
  • the control located at the bottom middle of the display interface is a control for calling the camera device of the mobile device.
  • the target user triggers the control named "shoot” the mobile terminal device starts the camera device to shoot.
  • the user image can be shot, and the video frame in the video shot in the mobile terminal device is used as the video frame to be processed, and the user image shot is It can be used as a target object, and the mixed audio corresponding to the target object can be determined.
  • a display interface of such a mixing condition can be shown in Figure 3.
  • the interface can be set to trigger the control for selecting the mixed audio to be selected, such as the control corresponding to "Part 1", “Part 2", “Part 3", and "Part 4" in Figure 3.
  • the target user triggers any control in the control of the mixed audio to be selected, it indicates that the target user has selected the mixed audio corresponding to the control.
  • the target user can trigger all the controls of the mixed audio to be selected displayed in the display interface. If multiple controls are triggered, multiple mixed audios can be determined.
  • the control located at the lower right of the display interface is a control for uploading a pre-shot video.
  • the target user When the target user triggers the control named "Album", it jumps to the album browsing interface, and can find and select a pre-shot video from the album of the mobile device.
  • the selected pre-shot video is displayed as a video frame to be processed in the display interface, and the user in the video frame to be processed can be used as a target object, and the mixed audio corresponding to the target object can be determined.
  • S120 Determine a target audio of a to-be-processed video frame based on at least one mixed audio and audio information of at least one target object.
  • the audio information is based on an audio acquisition module, for example, the audio data corresponding to the target object collected by a microphone array.
  • the target audio can be understood as playing the mixed audio and the audio data corresponding to the target object as dual-track audio. For example, if the determined mixed audio is a child's voice, and the audio information actually collected is the audio of a young person, the child's voice and the young person's audio can be played as dual-track audio and played together as the target audio.
  • the attributes corresponding to each target object are different.
  • the target object can be an elderly person, a middle-aged person, or a child.
  • the audio information and mixed audio of all objects can be used as the target audio as a whole, and the target audio can be played based on the speaker. If you want to reflect the effect of multiple people singing at the same time, you can directly play the audio information and mixed audio of all objects as multiple tracks. If you want to reflect the audio signal of a target object, then you can set a control in the display interface at this time, and the control is used to select which target user's audio information to play.
  • target object A and target object B only want to reflect the audio signal of target object A, then at this time, you can set a control near target object A in the display interface, triggering this control can choose to play only the audio signal of target object A, and the audio signal of target user B can be muted.
  • the song text information corresponding to the mixed audio song can also be displayed in the display interface to guide the target user to read, sing or broadcast based on the song text information.
  • S130 Determine a special effect video frame corresponding to the video frame to be processed based on the target audio and at least one target object.
  • the special effect video frame is a video frame that simultaneously displays the target object and the target audio.
  • the target audio includes the mixed audio and the audio information of the target object, and the target object corresponds to the image information in the video frame. Based on the determined target audio, the target object corresponding to the target audio is simultaneously displayed in the display interface, so that the display screen of the target object is consistent with the target audio, thereby obtaining a special effect video frame.
  • the target audio and the target object are fused to obtain each special effect video frame.
  • multiple special effect video frames are spliced in time to obtain a special effect video.
  • the technical solution of the disclosed embodiment can determine at least one mixed audio corresponding to at least one target object in the video frame to be processed when it is detected that the mixing condition is met, and then based on the determined mixed audio and the audio information of at least one target object, the target audio corresponding to multiple tracks can be determined, and the final special effect video frame can be obtained by fusing the target audio and the target object.
  • the technical effect of not only processing the picture content but also the audio content is achieved, which improves the richness and fun of the special effect display effect, and further improves the technical effect of the target user's use experience.
  • FIG4 is a flow chart of a method for generating special effects video provided by an embodiment of the present disclosure.
  • determining the mixed audio corresponding to the target object in the video frame to be processed can be achieved in a variety of ways.
  • the target audio can be determined according to the volume information corresponding to the audio information.
  • the specific implementation method can refer to the technical solution of this embodiment. Among them, the technical terms that are the same as or corresponding to the above-mentioned embodiment are not repeated here.
  • the method comprises the following steps:
  • S210 Determine at least one mixed audio.
  • a first implementation manner is to determine at least one mixed audio based on a triggering operation on at least one mixing control on a display interface.
  • the method of determining the mixed audio based on the triggering operation of the mixing control on the display interface is applicable to the case where the video frame to be processed is a video frame captured in real time or a video frame in a recorded video.
  • the mixed audio effect corresponding to the control can be directly selected according to the control prompt in the display interface.
  • the target user can select multiple Mixing control, the number of mixed audios determined at this time corresponds to the number of mixing controls triggered by the target user.
  • the target user triggers the Part 1 control in the display interface, and the mixed sound effect can be directly determined as the audio content of Part 1; if the target user triggers the controls corresponding to Part 1, Part 2, and Part 3 in the interface within the preset duration, the audio contents corresponding to Part 1, Part 2, and Part 3 can all be used as mixed audio. It can be pre-set to determine whether the part corresponding to the special effect prop control is selected based on the number of times the control of the special effect prop is triggered.
  • the target user triggers the special effects props control an odd number of times, for example, the target user triggers the special effects props control 1 time or 3 times, then it indicates that the part corresponding to the current control is selected; if the target user triggers the special effects props control an even number of times, for example, the target user triggers the special effects props control 2 times or 4 times, then it means that the user has triggered the special effects props control once, and triggered the same control again on the basis of triggering the control, then it indicates that the target user cancels the part corresponding to the current control, that is, the part corresponding to the current control is not used as the final mixed audio to be displayed.
  • a second implementation manner is to determine at least one mixed audio according to an object attribute of at least one target object.
  • the method of determining the mixed audio according to the object attributes of the target object is applicable to the case where the video frame to be processed is a video frame collected in real time or a video frame in a recorded video.
  • the target object can have multiple attributes, for example, different attributes can be distinguished from the gender aspect, or different attributes can be distinguished from the age stage.
  • the attributes of the target object are different, and the mixed audio determined according to the attributes of the target object is also different.
  • the method of determining at least one mixed audio may include: identifying the object attributes of at least one target object based on a facial detection algorithm; based on the number of attribute categories of the object attributes and the object attributes, determining the mixed audio consistent with the number of attribute categories from at least one pre-made mixed audio to be selected.
  • the mixed audio can be determined based on the total number of attribute categories and the mixed audio of multiple people. For example, if it is detected that the object attributes in the display interface include both a male and a female, the number of attribute categories of the object attributes is 2 at this time.
  • the male's mixed audio, the female's mixed audio, and the multi-person mixed audio can be retrieved.
  • the object attributes in the display interface may be multiple males and multiple females, but at this time the number of attribute categories of the object attributes is still 2. At this time, multiple male mixed audios will not be repeatedly retrieved, and multiple female mixed audios will not be repeatedly retrieved. Only one male's mixed audio, one female's mixed audio, and multi-person mixed audio will be determined.
  • the mixed audio corresponding to the video frame to be processed can be set to a pre-configured child part; if it is detected that the target object in the display interface is an elderly person, the mixed audio corresponding to the video frame to be processed can be set to the pre-configured elderly part; if the pre-made mixed audio to be selected includes a child part, a juvenile part, a youth part, a middle-aged part and an elderly part, when it is detected that the target object in the display interface is a child and an elderly person, the child part and the elderly part are determined from the pre-made mixed audio to be selected as the mixed audio, so the number of determined mixed audios is 2, and the attribute categories of the object attributes include children and the elderly, so the number of attribute categories of the object attributes is 2, and the number of attribute categories of the object attributes is consistent with the number of mixed audios; if the pre-made mixed audio to be selected includes a child part, a juvenile
  • a third implementation manner is to determine at least one mixed audio according to audio information in the video frame to be processed.
  • At least one mixed audio is determined based on the audio information in the video frame to be processed, which is applicable to the case where the video frame to be processed is a video frame in a recorded video.
  • the determined video frame to be processed may contain the original audio information in the video frame, and the original audio information may indicate the content of the song that the target user wants to sing.
  • the audio information in the video frame may be identified first, and the mixed audio associated with the audio information in the video frame may be determined to achieve the effect of meeting the personalized needs of the target user.
  • a harmony melody is determined according to accompaniment information of the audio information in the video frame to be processed and a target part in the harmony; and at least one mixed audio is determined based on pitch information in the harmony melody and pitch information in the audio information.
  • the target voice part can be the high voice part, low voice part, or the harmony melody of a syllable of the harmony in the video frame to be processed, or it can be the voice part corresponding to a pre-calibrated syllable.
  • the harmony melody can be the melody associated with the voice part of the audio information in the video frame to be processed. For example, in the process of music creation, the tune of the song is different, the melody corresponding to the song will also change, and the harmony melody of different voice parts is also different.
  • the harmony of music includes high voice harmony, middle voice harmony and low voice harmony, wherein the harmony melody of the high voice harmony is melody A, the harmony melody of the middle voice harmony is melody B, and the harmony melody of the low voice harmony is melody C, and melody A, melody B and melody C are different melodies.
  • the accompaniment information of the audio information in the video frame to be processed For example, if the audio information in the video frame to be processed is the audio of the user's impromptu humming, the accompaniment information of the audio can be obtained through the accompaniment detection algorithm, and then the corresponding chords are matched for the accompaniment through the chord matching algorithm to obtain the accompaniment information of the audio information in the video frame to be processed. Subsequently, obtain the target part in the harmony of the audio information in the video frame to be processed. The target part can be the corresponding part in the harmony in the video frame to be processed.
  • the target part is the bass part; if the part to be processed is If the part in the harmony of the audio information in the video frame is the middle part, the target part is the middle part; if the part in the harmony of the audio information in the video frame to be processed is the high part, the target part is the high part. Finally, based on the accompaniment information and the target part in the harmony, the harmony melody is determined.
  • the chord position in the accompaniment chord can be lowered to obtain the harmony melody of the low part; if the target part in the harmony is determined to be the high part, the chord position in the accompaniment chord can be increased to obtain the harmony melody of the high part.
  • the pitch information in the harmony melody and the pitch information in the audio information in the video frame to be processed can jointly reflect which song the audio hummed by the original audio information in the video frame to be processed belongs to, and then determine the audio related to this song from the pre-set mixed audio as the mixed audio, and the mixed audio determined at this time is highly correlated with the original audio information in the video frame to be processed.
  • the accompaniment information of the audio is first obtained through the accompaniment detection algorithm, and then the corresponding chords are matched for the accompaniment through the chord matching algorithm to obtain the accompaniment information of song A in the video frame to be processed; then, the target part of song A in the video frame to be processed is obtained as the low part, and the chord position in the accompaniment chord can be lowered at this time to obtain the harmony melody of the low part. Due to the different tones of songs, the melody corresponding to the song will also change, and the harmony melodies of different parts are also different. Therefore, the tone information in the harmony melody can represent the specific song corresponding to the tone information in the audio information in the video frame to be processed. When determining the mixed audio, the audio related to song A will be selected as the mixed audio.
  • At least one mixed audio is determined based on the pitch information in the harmony melody and the pitch information in the audio information.
  • the advantage of this setting is that the mixed audio associated with the actual audio information in the video frame to be processed is determined according to the actual audio information of the target object, which can meet the personalized needs of the target user.
  • determining at least one mixed audio based on the pitch information in the harmony melody and the pitch information in the audio information includes: determining at least one mixed audio based on the pitch information in the harmony melody, the pitch information in the audio information and an object attribute of at least one target object.
  • the object attribute of the target object can also be used as a consideration for determining the mixed audio.
  • song A can be determined based on the pitch information in the harmony melody and the pitch information in the audio information.
  • the mixed audio can contain audio content of song A sung by children's voices.
  • the mixed audio includes at least one vocal harmony accompaniment or the mixed audio includes at least one Audio of the harmony accompaniment and lead vocal tracks of the vocal parts.
  • the mixed audio can be audio in two different ways.
  • One is a harmony accompaniment containing one or more parts; the other is an audio that contains not only a harmony accompaniment of one or more parts, but also a lead vocal track, that is, the content of the mixed audio can be only accompaniment music, or it can be a combination of accompaniment music and lead vocal track.
  • the advantage of this setting is that there are multiple ways to compose mixed audio, providing users with more alternative playback methods, and improving the richness and fun of special effect display effects.
  • a mixed sound effect is determined for the video frame to be processed, and the mixed audio can be determined in a variety of ways.
  • the advantage of this arrangement is that the mixed audio is determined in a variety of ways, making the application scope of this solution wider.
  • S220 Determine the audio to be displayed according to the volume information corresponding to the audio information.
  • the audio volume information corresponding to the multiple target objects is different.
  • the audio track corresponding to the target object in the mixed audio can be determined based on the volume information.
  • the video frame to be processed contains target object A and target object B.
  • Target object A is relatively familiar with the current song, so the volume of target object A singing along is relatively large, while target object B is relatively unfamiliar with the current song, so the volume of target object A singing along is relatively small.
  • the volume information of target object A is stronger than the volume information of target object B, and the audio information of target object A can be used as the audio to be displayed.
  • S230 Use at least one mixed audio and the audio to be displayed as target audio of the video frame to be processed.
  • the determined mixed audio and the audio to be displayed are played in dual tracks. That is to say, the target audio includes not only the mixed audio but also the target audio with relatively large volume information.
  • the advantage of such a setting is that the audio information with large volume can be strengthened and the audio information with small volume can be weakened, so that the played audio is more harmonious and pleasant to listen to.
  • S240 Determine a special effect video frame corresponding to the video frame to be processed based on the target audio and at least one target object.
  • the technical solution of the embodiment of the present disclosure can adopt a variety of methods to determine the mixed audio corresponding to at least one target object, that is, at least one mixed audio can be determined based on the triggering operation of at least one mixed audio control on the display interface; at least one mixed audio can be determined according to the object properties of at least one target object; or at least one mixed audio can be determined according to the audio information in the video frame to be processed.
  • the mixed audio determined by various means is relatively highly adaptable to the user.
  • the target audio determined based on the mixed audio and the audio information of the target object is closest to the actual effect, thereby improving the display effect of the special effects and expanding the scope of application of this solution.
  • FIG5 is a flow chart of a method for generating special effects video provided by an embodiment of the present disclosure.
  • richer display content is displayed in the special effects display interface to create a realistic on-site atmosphere.
  • the technical solution of this embodiment please refer to the technical solution of this embodiment. Among them, the technical terms that are the same as or corresponding to the above-mentioned embodiment are not repeated here.
  • the method includes the following steps:
  • S320 Determine a target audio of a to-be-processed video frame based on at least one mixed audio and audio information of at least one target object.
  • S330 Determine at least one split-screen image corresponding to at least one target object.
  • one or more target objects may be displayed in the video frame to be processed. If there is only one target object in the video frame to be processed, the image content corresponding to the one target object may be copied to obtain a split-screen image, and the split-screen image may be displayed at a preset position in the display interface. If there are multiple target objects in the video frame to be processed, the image content corresponding to the multiple target objects may be copied as a whole to obtain a split-screen image, and the split-screen image may be displayed in the display interface.
  • each split-screen image includes at least one target object, or each split-screen image includes one target object.
  • the split-screen image may include one target object, see Figure 6. If there are multiple target objects in the video frame to be processed, the split-screen image can be obtained in two ways. The first way is: the image content corresponding to the multiple target objects can be cut out as a whole, and the overall cut-out content of the multiple target objects is the split-screen image, see Figure 7. The second way is: the image content corresponding to the multiple target objects can be split and processed, that is, the multiple target objects are split into independent split-screen images, and displayed at preset positions, see Figure 8. The advantage of this setting is that no matter how many target objects there are, the split-screen image can be determined according to the user's choice, which enhances the user experience.
  • the display effect of the target object in the display interface can also be: segmenting at least one target object to determine an object segmentation image; taking at least one target object as the center of the video frame to be processed, and stacking the display object segmentation image on both sides of the center according to a preset zoom ratio to update the special effect video frame.
  • the image corresponding to the target object can be segmented and processed, and then the object segmentation images are stacked and displayed on both sides of the center according to a preset scaling ratio with the target object as the center, as shown in FIG9.
  • the image contents corresponding to the multiple target objects can be segmented as a whole to obtain the object segmentation images of the multiple target objects as a whole, and the object segmentation images of the multiple target objects as a whole are stacked and displayed on both sides of the center according to a preset scaling ratio, as shown in FIG10.
  • multiple target objects can also be stacked and displayed on both sides of the center according to a preset scaling ratio.
  • the target objects are segmented and processed separately.
  • the video frame to be processed includes target object A and target object B, and target object A and target object B are segmented and processed separately.
  • the object segmentation image corresponding to target object A is stacked on the left side of the center according to a preset scaling ratio
  • the object segmentation image corresponding to target object B is stacked on the right side of the center according to a preset scaling ratio, see Figure 11, wherein the scaling ratio can be reduced by 20 percent on the basis of the original image.
  • the advantage of such a setting is that more object segmentation images are displayed in the special effects display page, so that the special effects display effect reflects the scene of the chorus on the scene, which enhances the interest of the special effects display effect.
  • S340 Determine a special effect video frame based on at least one split-screen image, target audio, and a video frame to be processed.
  • the split-screen image, target audio and video frames to be processed are superimposed as a whole to obtain special effects video frames with both audio special effects and image special effects. Subsequently, multiple special effects video frames can be spliced to generate a special effects video that can display the chorus effect.
  • the technical solution of the disclosed embodiment based on the special effects processing of the audio, can determine multiple split-screen images corresponding to the target object based on the target object, and then superimpose the split-screen images, the target audio and the video frame to be processed as a whole to obtain a special effects video frame with both audio special effects and image special effects. That is, in addition to the special effects processing of the audio, the special effects processing is also performed on the image corresponding to at least one target object, so as to achieve synchronous processing of the audio and the image, so as to improve the display content of the special effects screen, so that the special effects display effect reflects the scene of the chorus on the scene, and improve the richness of the screen content.
  • FIG12 is a flowchart of another method for generating special effects video provided by an embodiment of the present disclosure.
  • a 3D microphone is displayed in the special effects display interface, and can be aimed at the target object in real time to create a realistic on-site atmosphere.
  • the technical solution of this embodiment please refer to the technical solution of this embodiment. Among them, the technical terms that are the same as or corresponding to the above-mentioned embodiments are not repeated here.
  • the method specifically includes the following steps:
  • S420 Determine a target audio of a to-be-processed video frame based on at least one mixed audio and audio information of at least one target object.
  • S430 Determine a special effect video frame corresponding to the video frame to be processed based on the target audio and at least one target object.
  • an alignment pair corresponding to the 3D microphone is determined from at least one target object.
  • the display position of the 3D microphone in the special effect video frame is adjusted according to the position information of the aligned object.
  • the position of the 3D microphone in the special effect video frame is shown in Figure 13.
  • the advantage of this setting is that the 3D microphone is displayed in the special effect display page, making the special effect display effect more realistic and enhancing the richness of the special effect display effect.
  • displaying a 3D microphone in a special effect video frame may include the following steps: determining an alignment object corresponding to the 3D microphone from at least one target object; adjusting the microphone display position of the 3D microphone in the special effect video frame according to the target position information of the alignment object; wherein the microphone display position includes the microphone deflection angle and/or the display height of the microphone in the special effect video frame.
  • one is to determine the alignment object based on the depth information of the image, and the other is to determine the alignment object based on the screen display ratio.
  • the implementation method of determining the alignment object based on the screen display ratio is as follows: determine the display ratio of each target object in the video frame in the screen, and the target object with the largest display ratio can be used as the alignment object.
  • Determining the alignment object based on depth information can be as follows: the depth information can represent the distance between the camera and the user. The closer the user is to the camera, the smaller the depth information; the farther the user is from the camera, the larger the depth information.
  • Determine the depth image corresponding to each target object in the video frame to be processed then calculate the depth value corresponding to each point in the portrait of the target object, and then calculate the average depth value of each portrait point, and finally obtain the depth information of each target object, and use the target object with the smallest depth information as the alignment object.
  • the display position of the alignment object in the display interface in the video frame to be processed may have certain changes, for example, there is a certain rotation angle, etc.
  • the display position of the 3D microphone can be adaptively adjusted according to the deflection angle of the alignment object.
  • the target position information of the alignment object can be a pre-set fixed point, for example, it can be a nose tip fixed point of the target object.
  • the process of determining the nose tip fixed point is: firstly, the position information of the nose tip fixed point is tracked in real time based on the face detection algorithm, and then the deflection angle of the 3D microphone is adaptively adjusted according to the position information of the nose tip fixed point and the deflection angle of the pre-defined baseline, so as to achieve the effect that the 3D microphone follows the alignment object in real time.
  • the position information of the nose tip fixed point can be represented by a spatial coordinate point.
  • the normal of the nose tip fixed point can be determined, and the baseline corresponds to a normal line, and then the angle between the normal of the nose tip fixed point and the normal corresponding to the baseline can be calculated.
  • the calculated angle is the deflection angle of the microphone.
  • the microphone adjusts its display position according to the deflection angle.
  • the deflection angle range can be fixed between [-30°, 30°]. That is, the deflection angle of the microphone can be determined based on the deflection angle range and the actual deflection angle.
  • the target user when the target user is shooting a video, the target user may be far away from the camera at different times. At this time, the display position of the target object in the video frame to be processed may move up and down. In this case, the relative display height of the 3D microphone needs to be adjusted.
  • the technical solution of the disclosed embodiment on the basis of synchronous special effects processing of audio and the image of the target object, can also display the 3D microphone in real time in the special effects video frame, and adjust the display position of the 3D microphone in the display interface based on the display position information of the target object, so that the 3D microphone and the target object are matched in real time, thereby achieving the effect of collecting audio information of the target object based on the 3D microphone, improving the realism of the special effects display effect, and further improving the interest of the special effects display.
  • Figure 14 is a structural schematic diagram of a device for generating special effect video provided by an embodiment of the present disclosure. As shown in Figure 14, the device includes: a mixed audio determination module 510, a target audio determination module 520 and a special effect video frame determination module 530.
  • the mixed audio determination module 510 is configured to determine at least one mixed audio corresponding to at least one target object in the video frame to be processed when it is detected that the mixing condition is met; wherein the video frame to be processed is a video frame captured in real time or a video frame in a recorded video; the target audio determination module 520 is configured to determine the target audio of the video frame to be processed based on at least one mixed audio and audio information of at least one target object; the special effect video frame determination module 530 is configured to determine the special effect video frame corresponding to the video frame to be processed based on the target audio and at least one target object.
  • the mixing conditions include at least one of the following: triggering special effect props corresponding to the mixing special effect; including at least one target object in the display interface; triggering the shooting control; detecting the recorded video uploaded by the triggered video processing control.
  • the mixed audio determination module 510 includes: at least one of the following: a trigger operation determination submodule, an object attribute determination submodule and a mixed audio determination submodule.
  • a trigger operation determination submodule is configured to determine at least one mixed audio based on a trigger operation of at least one mixing control on a display interface; wherein at least one mixing control corresponds to at least one mixed audio to be selected; an object property determination submodule is configured to determine at least one mixed audio according to an object property of at least one target object; and a mixed audio determination submodule is configured to determine at least one mixed audio according to audio information in a video frame to be processed.
  • the object attribute determination submodule includes: a facial algorithm recognition unit and an attribute category determination unit.
  • the mixed audio determination submodule includes: a harmony melody determination unit and a mixed audio determination unit.
  • the harmony melody determination unit is configured to determine the harmony melody based on the accompaniment information of the audio information in the video frame to be processed and the target part in the harmony; the mixed audio determination unit is configured to determine at least one mixed audio based on the pitch information in the harmony melody and the pitch information in the audio information.
  • the mixed audio determination unit is configured to determine at least one mixed audio based on the pitch information in the harmony melody, the pitch information in the audio information and the object attribute of at least one target object.
  • the mixed audio includes the harmony accompaniment of at least one part or the mixed audio includes the audio of the harmony accompaniment of at least one part and the lead vocal track.
  • the target audio determination module 520 includes: a volume information determination submodule and a target audio determination submodule.
  • the volume information determination submodule is configured to determine the audio to be displayed according to the volume information corresponding to the audio information; the target audio determination submodule is configured to use at least one mixed audio and the audio to be displayed as the target audio of the video frame to be processed.
  • the special effect video frame determination module 530 includes: a split-screen image determination submodule and a special effect video frame determination submodule.
  • the split-screen image determination submodule is configured to determine at least one split-screen image corresponding to at least one target object; the special effect video frame determination submodule is configured to determine the special effect video frame based on at least one split-screen image, target audio and video frame to be processed.
  • each split-screen image includes at least one target object, or each split-screen image includes one target object.
  • the device further includes: a segmented image determination module and a special effect video update module.
  • a segmented image determination module is configured to perform segmentation processing on at least one target object to determine an object segmentation image
  • a special effect video update module is configured to take at least one target object as the center of a video frame to be processed, and to stack and display the object segmentation image on both sides of the center according to a preset scaling ratio to update the special effect video frame.
  • the device further includes: a microphone display module, configured to display a 3D microphone in a special effect video frame.
  • the microphone display module further includes: an aiming object determination submodule and a microphone position adjustment submodule.
  • the alignment object determination submodule is configured to determine from at least one target object an object corresponding to the 3D microphone. an aiming object; a microphone position adjustment submodule, configured to adjust the microphone display position of the 3D microphone in the special effects video frame according to the target position information of the aiming object; wherein the microphone display position includes a microphone deflection angle and/or a display height of the microphone in the special effects video frame.
  • the technical solution of the disclosed embodiment can determine at least one mixed audio corresponding to at least one target object in the video frame to be processed when it is detected that the mixing condition is met, and then based on the determined mixed audio and the audio information of at least one target object, the target audio corresponding to multiple tracks can be determined, and the final special effect video frame can be obtained by fusing the target audio and the target object.
  • the technical effect of not only processing the picture content but also the audio content is achieved, which improves the richness and fun of the special effect display effect, and further improves the technical effect of the target user's use experience.
  • the device for generating special effects video provided by the embodiments of the present disclosure can execute the method for generating special effects video provided by any embodiment of the present disclosure, and has functional modules and effects corresponding to the execution method.
  • the multiple units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be realized; in addition, the names of the multiple units and modules are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.
  • FIG15 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure.
  • FIG15 shows a schematic diagram of the structure of an electronic device (e.g., a terminal device or server in FIG15 ) 600 suitable for implementing an embodiment of the present disclosure.
  • the terminal device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle-mounted terminals (e.g., vehicle-mounted navigation terminals), etc., and fixed terminals such as digital televisions (TVs), desktop computers, etc.
  • PDAs personal digital assistants
  • PADs Portable Android Devices
  • PMPs portable multimedia players
  • vehicle-mounted terminals e.g., vehicle-mounted navigation terminals
  • fixed terminals such as digital televisions (TVs), desktop computers, etc.
  • TVs digital televisions
  • TVs digital televisions
  • the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which may perform a variety of appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603.
  • a processing device 601 e.g., a central processing unit, a graphics processing unit, etc.
  • RAM random access memory
  • the processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604.
  • An input/output (I/O) interface 605 is also connected to the bus 604.
  • the following devices can be connected to the I/O interface 605: including, for example, a touch screen, a touch pad, a keyboard, Input devices 606 such as a mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 607 such as a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 such as a magnetic tape, a hard disk, etc.; and communication devices 609.
  • the communication device 609 can allow the electronic device 600 to communicate with other devices wirelessly or wired to exchange data.
  • FIG. 15 shows an electronic device 600 with multiple devices, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or provided instead.
  • an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart.
  • the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602.
  • the processing device 601 the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.
  • the electronic device provided in the embodiment of the present disclosure and the method for generating special effect videos provided in the above embodiment belong to the same inventive concept.
  • the technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same effect as the above embodiment.
  • An embodiment of the present disclosure provides a computer storage medium on which a computer program is stored.
  • the program is executed by a processor, the method for generating a special effect video provided by the above embodiment is implemented.
  • the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two.
  • the computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, RAM, ROM, an erasable programmable read-only memory (EPROM) or flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
  • a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or device.
  • a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above.
  • Computer-readable signal medium It can also be any computer-readable medium other than a computer-readable storage medium, and the computer-readable signal medium can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device.
  • the program code contained on the computer-readable medium can be transmitted using any appropriate medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination of the above.
  • the client and the server may communicate using any currently known or future developed network protocol such as HyperText Transfer Protocol (HTTP), and may be interconnected with any form or medium of digital data communication (e.g., a communication network).
  • HTTP HyperText Transfer Protocol
  • Examples of communication networks include a local area network (LAN), a wide area network (WAN), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.
  • the computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
  • the computer-readable medium carries one or more programs.
  • the electronic device determines at least one mixed audio corresponding to at least one target object in the video frame to be processed when it is detected that the mixing condition is met; wherein the video frame to be processed is a video frame captured in real time or a video frame in a recorded video; determines the target audio of the video frame to be processed based on the at least one mixed audio and the audio information of at least one target object; determines the special effect video frame corresponding to the video frame to be processed based on the target audio and at least one target object.
  • Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages.
  • the program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server.
  • the remote computer may be connected to the user's computer via any type of network, including a LAN or WAN, or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
  • each box in the flowchart or block diagram may represent a module, a program segment, or a portion of a code, which contains one or more executable instructions for implementing a specified logical function.
  • the functions marked in the boxes may also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession may actually be substantially parallel. The instructions may be executed sequentially, and they may sometimes be executed in reverse order, depending on the functions involved.
  • each block in the block diagram and/or flow chart, and the combination of blocks in the block diagram and/or flow chart may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.
  • the units and modules involved in the embodiments described in the present disclosure may be implemented by software or hardware.
  • the names of the units and modules do not limit the units and modules themselves.
  • the mixed audio determination module may also be described as "a module for determining at least one mixed audio corresponding to at least one target object in a video frame to be processed when a mixed audio condition is detected to be met".
  • FPGA Field Programmable Gate Array
  • ASIC Application Specific Integrated Circuit
  • ASSP Application Specific Standard Parts
  • SOC System on Chip
  • CPLD Complex Programmable Logic Device
  • a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment.
  • a machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium.
  • a machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, RAM, ROM, EPROM or flash memory, optical fibers, portable CD-ROMs, optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
  • Example 1 provides a method for generating a special effects video, the method comprising: when it is detected that a mixing condition is met, determining at least one mixed audio corresponding to at least one target object in a video frame to be processed; wherein the video frame to be processed is a video frame captured in real time or a video frame in a recorded video; based on at least one mixed audio and audio information of at least one target object, determining the target audio of the video frame to be processed; based on the target audio and at least one target object, determining a special effects video frame corresponding to the video frame to be processed.
  • Example 2 provides a method for generating a special effects video, the method further comprising: optionally, determining at least one mixed audio based on a triggering operation of at least one mixed audio control on a display interface; wherein at least one mixed audio control corresponds to at least one mixed audio to be selected; determining at least one mixed audio according to an object attribute of at least one target object; and At least one mixed audio is determined according to audio information in the video frame to be processed.
  • Example Three provides a method for generating a special effects video, the method further including: optionally, determining at least one mixed audio based on object attributes of at least one target object, including: identifying the object attributes of at least one target object based on a face detection algorithm; based on the number of attribute categories of the object attributes and the object attributes, determining a mixed audio consistent with the number of attribute categories from at least one pre-made mixed audio to be selected.
  • Example 4 provides a method for generating a special effects video, the method also includes: optionally, determining at least one mixed audio based on the audio information in the video frame to be processed, including: determining the harmony melody based on the accompaniment information of the audio information in the video frame to be processed and the target part in the harmony; determining at least one mixed audio based on the pitch information in the harmony melody and the pitch information in the audio information.
  • Example Five provides a method for generating a special effects video, the method also includes: optionally, determining at least one mixed audio based on the pitch information in the harmonic melody and the pitch information in the audio information, including: determining at least one mixed audio based on the pitch information in the harmonic melody, the pitch information in the audio information and the object attributes of at least one target object.
  • Example Six provides a method for generating a special effects video, the method further comprising: optionally, the mixed audio includes a harmony accompaniment of at least one part or the mixed audio includes a harmony accompaniment of at least one part and audio of a lead vocal track.
  • Example 7 provides a method for generating a special effects video, the method also includes: optionally, based on at least one mixed audio and audio information of at least one target object, determining the target audio in the video frame to be processed, including: determining the audio to be displayed according to the volume information corresponding to the audio information; using at least one mixed audio and the audio to be displayed as the target audio of the video frame to be processed.
  • Example Eight provides a method for generating a special effects video, the method also includes: optionally, based on the target audio and at least one target object, determining a special effects video frame corresponding to the video frame to be processed, including: determining at least one split-screen image corresponding to at least one target object; determining the special effects video frame based on at least one split-screen image, the target audio and the video frame to be processed.
  • each split-screen image includes at least one target object, or each split-screen image includes one target object.
  • Example 10 provides a method for generating a special effects video, the method further comprising: optionally, segmenting at least one target object to determine the object segment segmented images; taking at least one target object as the center of the video frame to be processed, and stacking the display object segmented images on both sides of the center according to a preset scaling ratio to update the special effect video frame.
  • Example 11 provides a method for generating a special effects video, the method further comprising: optionally, displaying a 3D microphone in the special effects video frame.
  • Example 12 provides a method for generating a special effects video, the method further comprising: optionally, determining an alignment object corresponding to a 3D microphone from at least one target object; adjusting a microphone display position of the 3D microphone in a special effects video frame according to target position information of the alignment object; wherein the microphone display position includes a microphone deflection angle and/or a display height of the microphone in the special effects video frame.
  • Example 13 provides a method for generating a special effects video, the method also includes: optionally, the mixing condition includes at least one of the following: triggering a special effects prop corresponding to the mixing special effect; including at least one target object in the display interface; triggering a shooting control; detecting a recorded video uploaded by a triggered video processing control.
  • Example 14 provides a device for generating special effects video, which includes: a mixed audio determination module, which is configured to determine at least one mixed audio corresponding to at least one target object in a video frame to be processed when it is detected that a mixing condition is met; wherein the video frame to be processed is a video frame captured in real time or a video frame in a recorded video; a target audio determination module, which is configured to determine the target audio of the video frame to be processed based on at least one mixed audio and audio information of at least one target object; and a special effects video frame determination module, which is configured to determine the special effects video frame corresponding to the video frame to be processed based on the target audio and at least one target object.
  • a mixed audio determination module which is configured to determine at least one mixed audio corresponding to at least one target object in a video frame to be processed when it is detected that a mixing condition is met
  • the video frame to be processed is a video frame captured in real time or a video frame in a recorded video
  • a target audio determination module which is configured

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Acoustics & Sound (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Computation (AREA)
  • Human Computer Interaction (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • Medical Informatics (AREA)
  • General Engineering & Computer Science (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Databases & Information Systems (AREA)
  • Architecture (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • Signal Processing (AREA)
  • Studio Circuits (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)

Abstract

本公开实施例提供了一种生成特效视频的方法、装置、电子设备及存储介质，该方法包括：在检测到满足混音条件的情况下，确定待处理视频帧中至少一个目标对象所对应的至少一个混音音频；基于至少一个混音音频以及至少一个目标对象的音频信息，确定待处理视频帧的目标音频；基于目标音频和至少一个目标对象，确定与待处理视频帧相对应的特效视频帧。

Description

生成特效视频的方法、装置、电子设备及存储介质
本申请要求在2022年09月29日提交中国专利局、申请号为202211204819.0的中国专利申请的优先权，该申请的全部内容通过引用结合在本申请中。
技术领域
本公开实施例涉及图像处理技术，例如涉及一种生成特效视频的方法、装置、电子设备及存储介质。
背景技术
随着网络技术的发展，越来越多的应用程序进入了用户的生活，例如一系列可以拍摄短视频的软件，深受用户的喜爱。
软件开发商可以在应用中添加各种各样的特效道具，以供用户在拍摄视频的过程中使用，然而，这些特效道具的丰富度不足，无法完全匹配用户的需求。
发明内容
本公开提供一种生成特效视频的方法、装置、电子设备及存储介质，实现了对音频进行特效处理，从而丰富特效展示效果，进而提高了用户体验的技术效果。
第一方面，本公开实施例提供了一种生成特效视频的方法，该方法包括：
在检测到满足混音条件的情况下，确定待处理视频帧中至少一个目标对象所对应的至少一个混音音频；其中，待处理视频帧为实时采集的视频帧或录制视频中的视频帧；
基于至少一个混音音频以及至少一个目标对象的音频信息，确定待处理视频帧的目标音频；
基于目标音频和至少一个目标对象，确定与待处理视频帧相对应的特效视频帧。
第二方面，本公开实施例还提供了一种生成特效视频的装置，该装置包括：
混音音频确定模块，设置为在检测到满足混音条件的情况下，确定待处理视频帧中至少一个目标对象所对应的至少一个混音音频；其中，待处理视频帧 为实时采集的视频帧或录制视频中的视频帧；
目标音频确定模块，设置为基于至少一个混音音频以及至少一个目标对象的音频信息，确定待处理视频帧的目标音频；
特效视频帧确定模块，设置为基于目标音频和至少一个目标对象，确定与待处理视频帧相对应的特效视频帧。
第三方面，本公开实施例还提供了一种电子设备，电子设备包括：
一个或多个处理器；
存储装置，用于存储一个或多个程序，
当一个或多个程序被一个或多个处理器执行，使得一个或多个处理器实现如本公开实施例任一的生成特效视频的方法。
第四方面，本公开实施例还提供了一种包含计算机可执行指令的存储介质，计算机可执行指令在由计算机处理器执行时用于执行如本公开实施例任一的生成特效视频的方法。
附图说明
贯穿附图中，相同或相似的附图标记表示相同或相似的元素。应当理解附图是示意性的，原件和元素不一定按照比例绘制。
图1是本公开实施例所提供的一种生成特效视频的方法的流程示意图；
图2是本公开实施例所提供的生成特效视频应用程序的用户显示界面；
图3是本公开实施例所提供的生成特效视频的界面示意图；
图4是本公开实施例所提供的另一种生成特效视频的方法的流程示意图；
图5为本公开实施例所提供的另一种生成特效视频的方法的流程示意图；
图6是本公开实施例所提供的一种至少一个目标对象显示位置的示意图；
图7是本公开实施例所提供的另一种至少一个目标对象显示位置的示意图；
图8是本公开实施例所提供的另一种至少一个目标对象显示位置的示意图；
图9是本公开实施例所提供的一种分割图像显示位置的示意图；
图10是本公开实施例所提供的另一种分割图像显示位置的示意图；
图11是本公开实施例所提供的另一种分割图像显示位置的示意图；
图12为本公开实施例所提供的另一种生成特效视频的方法的流程示意图；
图13是本公开实施例所提供的一种三维(3Dimensional，3D)话筒的显示位置的示意图；
图14是本公开实施例所提供的一种生成特效视频的装置的结构示意图；
图15是本公开实施例所提供的一种电子设备的结构示意图。
具体实施方式
下面将参照附图描述本公开的实施例。虽然附图中显示了本公开的一些实施例，然而应当理解的是，本公开可以通过多种形式来实现，而且不应该被解释为限于这里阐述的实施例。本公开的附图及实施例仅用于示例性作用，并非用于限制本公开的保护范围。
本公开的方法实施方式中记载的多个步骤可以按照不同的顺序执行，和/或并行执行。此外，方法实施方式可以包括附加的步骤和/或省略执行示出的步骤。本公开的范围在此方面不受限制。
本文使用的术语“包括”及其变形是开放性包括，即“包括但不限于”。术语“基于”是“至少部分地基于”。术语“一个实施例”表示“至少一个实施例”；术语“另一实施例”表示“至少一个另外的实施例”；术语“一些实施例”表示“至少一些实施例”。其他术语的相关定义将在下文描述中给出。
本公开中提及的“第一”、“第二”等概念仅用于对不同的装置、模块或单元进行区分，并非用于限定这些装置、模块或单元所执行的功能的顺序或者相互依存关系。
本公开中提及的“一个”、“多个”的修饰是示意性而非限制性的，本领域技术人员应当理解，除非在上下文另有明确指出，否则应该理解为“一个或多个”。
本公开实施方式中的多个装置之间所交互的消息或者信息的名称仅用于说明性的目的，而并不是用于对这些消息或信息的范围进行限制。
在使用本公开的多个实施例公开的技术方案之前，均应当依据相关法律法规通过恰当的方式对本公开所涉及个人信息的类型、使用范围、使用场景等告知用户并获得用户的授权。
例如，在响应于接收到用户的主动请求时，向用户发送提示信息，以明确地提示用户，其请求执行的操作将需要获取和使用到用户的个人信息。从而，使得用户可以根据提示信息来自主地选择是否向执行本公开技术方案的操作的 电子设备、应用程序、服务器或存储介质等软件或硬件提供个人信息。
作为一种可选的但非限定性的实现方式，响应于接收到用户的主动请求，向用户发送提示信息的方式例如可以是弹窗的方式，弹窗中可以以文字的方式呈现提示信息。此外，弹窗中还可以承载供用户选择“同意”或者“不同意”向电子设备提供个人信息的选择控件。
上述通知和获取用户授权过程仅是示意性的，不对本公开的实现方式构成限定，其它满足相关法律法规的方式也可应用于本公开的实现方式中。
本技术方案所涉及的数据(包括但不限于数据本身、数据的获取或使用)应当遵循相应法律法规及相关规定的要求。
在介绍本技术方案之前，先对应用场景进行示例性说明。可以将本公开的技术方案应用在任意需要特效展示或者特效处理的场景中，如应用在视频拍摄过程中，可以对被拍摄目标对象进行特效处理；也可以是应用在视频拍摄过程后，例如，通过终端设备自带摄像机拍摄视频后，将预先拍摄的视频进行特效展示的情况。在本实施中，目标对象可以是用户或者任意可以发出音频信息的对象。
本公开实施例所提供的技术方法可以应用在实时采集的场景下，也可以应用在后处理的场景下。在实时采集的场景下，可以理解为每采集一个视频帧，就将所述视频帧作为待处理视频帧，并基于本公开实施例所提供的技术方法确定待处理视频帧相应的特效视频帧；在后处理的场景下，可以依次将上传的视频中的每一个视频帧均作为待处理视频帧。为了介绍本公开实施例所提供的技术方法，以对一个视频帧处理为例进行说明，对其余视频帧的处理可以重复执行本公开实施例所提供的步骤。
执行本公开实施例提供的生成特效视频的方法的装置，可以集成在支持特效视频处理功能的应用软件中，且该软件可以安装至电子设备中，可选的，电子设备可以是移动终端或者个人计算机(Personal Computer，PC)等。应用软件可以是对图像/视频处理的一类软件，具体的应用软件在此不再一一赘述，只要可以实现图像/视频处理即可。执行本公开实施例提供的生成特效视频的方法的装置还可以是专门研发的应用程序，来实现添加特效并将特效进行展示的软件中，亦或是集成在相应的页面中，用户可以通过PC中集成的页面来实现对特效视频的处理。
图1是本公开实施例所提供的一种生成特效视频的方法的流程示意图，本公开实施例适用于对音频进行特效处理的情形，该方法可以由生成特效视频的装置来执行，该装置可以通过软件和/或硬件的形式实现，可选的，通过电子设 备来实现，该电子设备可以是移动终端、PC或服务器等。本公开实施例所提供的技术方案可以由服务端执行，也可以由客户端执行，还可以由客户端和服务端配合执行。
如图1所示，所述方法包括：
S110、在检测到满足混音条件的情况下，确定待处理视频帧中至少一个目标对象所对应的至少一个混音音频。
混音条件可以理解为确定是否需要对待处理视频帧的音频进行特效处理的条件。
在本公开实施例中，混音条件可以包括多种情形，可以基于当前触发操作是否满足相应的情形，确定是否对待处理视频帧中的音频信息进行处理。
可选的，混音条件包括的情形可以是：触发与混音特效相对应的特效道具；显示界面中包括至少一个目标对象；触发拍摄控件；检测到基于触发的视频处理控件上传的录制视频。
在本实施例中，确定混音条件的第一种方式是触发与混音特效相对应的特效道具，可以理解为：基于本公开实施例所提供的技术方法，对程序代码或者处理数据进行压缩处理，使其作为一个特效包集成于一些应用程序软件中，作为特效道具。当触发特效道具时，说明当前是需要对采集的待处理视频帧中的音频进行特效处理，此时是满足混音条件的。
确定混音条件的第二种方式是显示界面中包括至少一个目标对象，不论是实时采集的视频帧还是非实时的视频帧，只要检测到入镜画面，即待处理视频帧中包括目标对象的，此时便认为满足混音条件。其中，目标对象可以是预先设定的。例如，目标对象可以是用户，只要检测到显示界面中有用户，计算机就认为满足混音条件。
确定混音条件的第三种方式是触发拍摄控件，可以将拍摄控件作为触发条件，其中，拍摄控件是预先编写好的，当基于摄像装置拍摄图像时，点击拍摄控件就说明满足了混音条件，此时只要采集的待处理视频帧中包括音频内容，就需要对音频进行特效处理。
确定混音条件的第四种方式是检测到基于触发的视频处理控件上传的录制视频，本方案不仅可以达到实时处理的效果，还可以进行后处理，当接收到上传的录制视频时，就说明需要对视频进行处理，可以基于本公开实施例的方法对视频进行特效处理。
在本实施例中，在确定待处理视频帧的过程中，主要涉及两种方式：分别是实时采集的视频帧和录制视频中的视频帧，而对于这两种确定方式可以包括 多种混音条件。这样设置的好处在于：无论用户是通过何种方式确定待处理视频帧，都可以通过多种混音条件，确定待处理视频帧中目标对象所对应的混音音频，使得本方案的适用范围更广。
待处理视频帧可以基于实时拍摄视频确定，也可以基于非实时拍摄视频确定。只要满足混音条件，就可以依次对实时采集的视频或者上传的视频的视频帧进行处理，可以将每一个视频帧都作为待处理视频帧。还有一种情形是如果在可选择的条件下对一些视频帧进行特效处理，可以将所选的每一个视频帧作为待处理视频帧。
目标对象为待处理视频帧中的所呈现的用户，目标对象的数量可以是一个或者多个，至于目标对象的数量的个数可以根据实际的情况进行预先设定。例如，如果预先设定是将所有入镜画面中的对象都作为目标对象，则目标对象的数量与入镜画面的用户数量是相对应的；如果仅需要对特定的一些用户进行特效处理，可以预先上传对象所对应的面部图像，以在入镜画面中包括多个显示对象时，可以基于上传的面部信息和显示对象的面部信息，确定目标对象。另外，还有一种方式是：基于目标用户于显示界面的触发操作，确定目标对象。例如，显示界面的对象有多个，可以将目标用户触发选择的对象作为目标对象。即，仅需要对触发选择的目标对象进行特效处理。
混音可以理解为是把多种来源的声音，整合至一个立体音轨或单音音轨中。多种声音的来源可以是不同用户所对应的不同声部的音频。因此，混音音频可以理解为多个演奏者演唱同一首歌曲中不同声部所对应的音频。例如，预先设定至少一首歌曲，可以基于多个用户确定多个混音音频。可以预先制作不同用户相适配的混音音频，例如，混音音频可以按照年龄阶段进行区分，可以按照性别属性进行区分，也可以按照声调进行区分。如果是按照年龄阶段进行区分可以分为：儿童、少年、青年、中年或者老年；如果是按照性别属性进行区分可以分为：男声部或者女声部；如果是按照音调进行区分可以分为：高音部、中音部或者低音部。在实际用过程中，可以预先设置一首歌或者多首歌并基于预先设置的一首歌或者多首歌确定多种划分标准对应的混音音频，以供目标用户使用。混音音频的数量可以是与目标对象的数量相对应，也可以通过触发选定混音音频。
示例性的，当目标用户触发应用软件或应用程序，进入生成特效视频应用程序的目标用户显示界面，参见图2。如图2所示，位于用于显示界面的最下方中间的控件为调用移动设备摄像装置的控件，当目标用户触发名称为“拍摄”的控件时，移动终端设备启动拍摄装置进行拍摄，此时可以拍摄用户图像，将移动终端设备中拍摄的视频中的视频帧作为待处理视频帧，而所拍摄到的用户 可以作为目标对象，可以确定目标对象所对应的混音音频。也可以预先设定混音特效道具对应的控件，通过触发特效道具控件作为混音条件，这种混音条件的一种显示界面可以参见图3。如图3所示，界面中可以设置触发选定待选择混音音频的控件，例如图3中的“声部1”、“声部2”、“声部3”、“声部4”对应的控件，当目标用户触发待选择混音音频的控件中的任意控件，表明目标用户选定了该控件对应的混音音频，在实际应用过程中，目标用户可以触发显示界面中显示的所有待选择混音音频的控件，如果触发多个控件，可以确定多个混音音频。另外，如图2所示，位于显示界面的右下方的控件为上传预先拍摄视频的控件，当目标用户触发名称为“相册”的控件时，跳转到相册浏览界面，可以从移动设备的相册中寻找并选择预先拍摄的视频，将选定的预先拍摄的视频作为待处理视频帧显示界面中，而待处理视频帧中的用户可以作为目标对象，可以确定目标对象所对应的混音音频。
S120、基于至少一个混音音频以及至少一个目标对象的音频信息，确定待处理视频帧的目标音频。
音频信息为基于音频采集模块，例如，麦克风阵列采集的与目标对象对应的音频数据。目标音频可以理解为在确定混音音频与目标对象对应的音频数据之后，将二者进行双轨音频进行播放。例如，确定的混音音频是儿童声部，实际采集的音频信息是青年人的音频，可以将儿童声部与青年人的音频作为双音轨音频，共同作为目标音频进行播放。
示例性的，参见图3，若显示界面中显示的混音控件与声部1、声部2、声部3、声部4等所对应的，基于目标用户对这些控件的触发操作，如触发的是声部1和声部2，则将声部1和声部2作为混音音频，基于声部1、声部2以及目标对象所对应的音频信息共同作为目标音频。
在本实施例中，每个目标对象所对应的属性是有差别的，例如，目标对象可以是老年人、中年人或者儿童，可以将所有对象的音频信息和混音音频整体共同作为目标音频，并基于扬声器播放目标音频。如果想要体现出多人同时唱歌的效果，可以直接将所有对象的音频信息和混音音频作为多音轨进行播放，如果想要体现出一个目标对象的音频信号，那么此时可以在显示界面中设置控件，控件用于选择播放哪位目标用户的音频信息。例如，目标对象A和目标对象B，仅想要体现出目标对象A的音频信号，那么此时可以在显示界面中目标对象A附近设置控件，触发这一控件可以选择仅仅播放目标对象A的音频信号，而目标用户B的音频信号可以做消音处理。
还可以在显示界面中显示与混音音频歌曲所对应的歌曲文本信息，以引导目标用户基于歌曲文本信息进行阅读、演唱或者播报。
S130、基于目标音频和至少一个目标对象，确定与待处理视频帧相对应的特效视频帧。
在本实施例中，特效视频帧为同时展示目标对象与目标音频的视频帧。目标音频中包含混音音频以及目标对象的音频信息，而目标对象对应的是视频帧中的图像信息。基于所确定的目标音频，在显示界面中同时显示与目标音频相对应的目标对象，以使目标对象的展示画面与目标音频是相一致的，从而得到特效视频帧。
对于每一帧待处理视频帧，均将目标音频与目标对象进行融合处理，可以得到每一帧特效视频帧，最终将多个特效视频帧在时间上进行拼接，得到特效视频。
本公开实施例的技术方案，当检测到满足混音条件时，可以确定待处理视频帧中至少一个目标对象所对应的至少一个混音音频，进而基于所确定的混音音频以及至少一个目标对象的音频信息，可以确定多个音轨所对应的目标音频，通过对目标音频和目标对象进行融合处理，可以得到最终的特效视频帧。实现了不仅可以对画面内容进行处理，还可以对音频内容进行处理的技术效果，提升了特效展示效果的丰富性、趣味性，还进一步提升了目标用户使用体验的技术效果。
图4是本公开实施例所提供的一种生成特效视频的方法的流程示意图，在前述实施例的基础上，确定待处理视频帧中目标对象所对应的混音音频可以通过多种方式实现，在确定目标音频的过程中，可以根据音频信息对应的音量信息确定目标音频。具体的实施方式可以参见本实施例技术方案。其中，与上述实施例相同或者相应的技术术语在此不再赘述。
如图4所示，该方法包括如下步骤：
S210、确定至少一个混音音频。
在本公开实施例中，确定至少一个混音音频的方式可以有多种，接下来阐述每一种方式是如何实现的。
第一种实现方式为：基于对显示界面上至少一个混音控件的触发操作，确定至少一个混音音频。
在本实施例中，基于对显示界面上混音控件的触发操作，确定混音音频的方式适用于待处理视频帧为实时采集的视频帧或者录制视频中的视频帧的情形。当目标用户触发显示界面中的混音特效相对应的特效道具时，可以根据显示界面中的控件提示，直接选定控件对应的混音音效，目标用户可以选择多个 混音控件，此时确定的混音音频的数量与目标用户触发混音控件的数量是对应的。例如，在图3中，目标用户触发显示界面中的声部1控件，此时可以直接确定混音音效为声部1的音频内容；如果目标用户在预设时长内触发了界面中声部1、声部2和声部3对应的控件，则可以将声部1、声部2和声部3对应的音频内容均作为混音音频。可以预先设置根据触发特效道具的控件的次数，确定是否选定特效道具控件对应的声部。示例性的，如果目标用户触发特效道具的控件的次数为奇数次，例如，目标用户触发特效道具的控件的次数1次或3次，则表明选中当前控件对应的声部；如果目标用户触发特效道具的控件的次数为偶数次，例如，目标用户触发特效道具的控件的次数2次或4次，此时说明用户已经触发了一次特效道具的控件，在触发控件的基础上又触发了相同的控件，则表明目标用户将当前控件对应的声部进行取消操作，即当前控件对应的声部不作为最终要展示的混音音频。
第二种实现方式为：根据至少一个目标对象的对象属性，确定至少一个混音音频。
在本实施例中，根据目标对象的对象属性确定混音音频的方式，适用于待处理视频帧为实时采集的视频帧或者录制视频中的视频帧的情形。在本实施例中，目标对象可以具有多种属性，例如，可以从性别方面区分不同的属性，也可以从年龄阶段区分不同的属性。目标对象的属性不同，根据目标对象的属性确定的混音音频也是不同的。可选的，根据至少一个目标对象的对象属性，确定至少一个混音音频的方法可以包括：基于面部检测算法识别至少一个目标对象的对象属性；基于对象属性的属性类别数量和对象属性，从预先制作的至少一个待选择混音中，确定出与属性类别数量相一致的混音音频。这样设置的好处在于：基于面部识别算法，结合属性类别数量，确定的混音音频与待处理视频中的目标对象匹配度更高，达到更加逼真的特效展示效果。
在本实施例中，根据面部识别算法，如果检测到显示界面中属性类别数量大于1，可以基于总的属性类别的数量以及多人混音音频确定混音音频。例如，如果检测到显示界面中对象属性同时包括一位男性和一位女性，此时对象属性的属性类别数量为2，在确定混音音频的过程中，可以调取男性的混音音频、调取女性的混音音频以及多人混音音频。在实际应用过程中，检测到显示界面中对象属性可能是多位男性以及多位女性，但是此时对象属性的属性类别数量依然为2，此时不会重复调取多个男性混音音频，以及重复调取多个女性混音音频，只确定一个男性的混音音频、一个女性的混音音频以及多人混音音频。
示例性的，根据面部识别算法，如果检测到显示界面中的目标对象是一位儿童，可以将待处理视频帧对应的混音音频设定为预先配置的儿童声部；如果 检测到显示界面中的目标对象是一位老年人，可以将待处理视频帧对应的混音音频设定为预先配置的老年人声部；如果预先制作的待选择混音包括儿童声部，少年声部、青年声部、中年声部和老年人声部，当检测到显示界面中的目标对象是一位儿童和一位老年人，则从预先制作的待选择混音中确定出儿童声部和老人声部作为混音音频，所以确定的混音音频的数量为2个，而对象属性的属性类别包括儿童与老年人，所以对象属性的属性类别数量为2，此时对象属性的属性类别数量与混音音频的数量是一致的；如果预先制作的待选择混音包括儿童声部，少年声部、青年声部、中年声部、老年人声部以及多人声部，当检测到显示界面中的目标对象是一位儿童和一位老年人，则从预先制作的待选择混音中确定出儿童声部、老人声部以及多人声部作为混音音频，即识别到目标对象是多人，混音音频中需要包括多人声部。
第三种实现方式为：根据待处理视频帧中的音频信息，确定至少一个混音音频。
在本实施例中，根据待处理视频帧中的音频信息确定至少一个混音音频，适用于待处理视频帧为录制视频中的视频帧的情形。确定的待处理视频帧中可以包含视频帧中原始的音频信息，原始的音频信息可以表明目标用户想要演唱的歌曲内容，此时可以先对视频帧中的音频信息进行识别，确定与视频帧中音频信息相关联的混音音频，实现满足目标用户个性化需求的效果。
可选的，根据待处理视频帧中音频信息的伴奏信息和和声中的目标声部，确定和声旋律；基于和声旋律中的音调信息和音频信息中的音调信息，确定至少一个混音音频。
目标声部可以是待处理视频帧中和声的高声部、低声部、或者一个音节的和声旋律，也可以是预先标定的一个音节所对应的声部。和声旋律可以为待处理视频帧中音频信息的声部相关联的旋律。例如，在音乐创作的过程中，歌曲的音调不同，歌曲对应的旋律也会发生改变，不同声部的和声旋律也不同。例如，音乐的和声包括高声部和声、中声部和声和低声部和声，其中，高声部和声的和声旋律为旋律A，中声部和声的和声旋律为旋律B，低声部和声的和声旋律为旋律C，旋律A、旋律B和旋律C为不同的旋律。
首先获取待处理视频帧中音频信息的伴奏信息，例如，若待处理视频帧中音频信息为用户即兴哼唱的音频，则可以通过伴奏检测算法，获取该音频的伴奏信息，进而通过和弦匹配算法，为该伴奏匹配对应的和弦，得到待处理视频帧中音频信息的伴奏信息。随后，获取待处理视频帧中音频信息的和声中的目标声部，目标声部可以为待处理视频帧中和声中对应的声部。例如，若待处理视频帧中音频信息的和声中的声部为低声部，则目标声部为低声部；若待处理 视频帧中音频信息的和声中的声部为中声部，则目标声部为中声部；若待处理视频帧中音频信息的和声中的声部为高声部，则目标声部为高声部。最后，基于伴奏信息和和声中的目标声部，确定和声旋律。例如，若确定和声中的目标声部为低声部，则可以降低伴奏和弦中的和弦位置，进而得到低声部的和声旋律；若确定和声中的目标声部为高声部，则可以提高伴奏和弦中的和弦位置，进而得到高声部的和声旋律。和声旋律中的音调信息和待处理视频帧中音频信息中的音调信息，可以共同反映待处理视频帧中原始的音频信息所哼唱的音频是属于哪首歌曲，进而从预先设定的混音音频中确定与这首歌曲相关的音频作为混音音频，此时确定的混音音频与待处理视频帧中原始的音频信息是高度相关的。
示例性的，假设待处理视频帧中音频信息为歌曲A对应的音频，则首先通过伴奏检测算法获取该音频的伴奏信息，进而通过和弦匹配算法为该伴奏匹配对应的和弦，得到待处理视频帧中歌曲A的伴奏信息；随后，获取待处理视频帧中歌曲A的目标声部为低声部，此时则可以降低伴奏和弦中的和弦位置，进而得到低声部的和声旋律。由于歌曲的音调不同，歌曲对应的旋律也会发生改变，不同声部的和声旋律也不同，因此，和声旋律中的音调信息可以表征待处理视频帧中音频信息中音调信息所对应的具体歌曲，在确定混音音频时，将选取与歌曲A相关的音频作为混音音频。
在上述实施例的基础上，基于和声旋律中的音调信息和音频信息中的音调信息，确定至少一个混音音频。这样设置的好处在于：根据目标对象实际音频信息，进而确定与待处理视频帧中实际音频信息相关联的混音音频，可以满足目标用户的个性化需求。
可选的，基于和声旋律中的音调信息和音频信息中的音调信息，确定至少一个混音音频，包括：基于和声旋律中的音调信息、音频信息中的音调信息和至少一个目标对象的对象属性，确定至少一个混音音频。
在上述实施例的基础上，除了根据和声旋律中的音调信息、音频信息中的音调信息确定至少一个混音音频之外，还可以将目标对象的对象属性作为确定混音音频的考虑因素。例如，根据和声旋律中的音调信息、以及音频信息中的音调信息，可以确定歌曲A，如果目标对象的对象属性为儿童，则混音音频中可以包含儿童声部演唱歌曲A的音频内容。这样设置的好处在于：根据目标对象的对象属性，确定与视频帧中实际音频信息相关联的混音音频，可以在满足目标用户的个性化需求的基础上，还可以使最终播放的目标音频与显示界面中的图像更加匹配。
可选的，混音音频包括至少一个声部的和声伴奏或混音音频包括至少一个 声部的和声伴奏和主唱音轨的音频。
在本实施例中，混音音频可以是两种不同表现方式的音频。一种是包含一个或者多个声部的和声伴奏；另一种是不仅包含一个或者多个声部的和声伴奏，还包括主唱音轨的音频，即混音音频的内容可以是只有伴奏音乐，也可以是伴奏音乐与主唱音轨二者合在一起的音乐。这样设置的好处在于：有多种混音音频的组成方式，为用户提供更多的备选播放方式，提升了特效展示效果的丰富性、趣味性。
在本实施例中，通常在满足一些特效添加条件的情况下，为待处理视频帧确定混音音效，而确定混音音频可以通过多种方式实现。这样设置的好处在于：采用多种方式确定混音音频，使得本方案的适用范围更广。
S220、根据音频信息所对应的音量信息，确定待展示音频。
在本实施例中，如果音频信息中录制了多位目标对象的音频内容，多位目标对象对应的音频音量信息是有差异的，此时可以基于音量信息确定混音音频中目标对象所对应的音轨。示例性的，待处理视频帧中包含目标对象A和目标对象B，目标对象A对当前的歌曲相对比较熟悉，那么目标对象A跟唱的音量是相对较大的，而目标对象B对当前的歌曲相对比较陌生，那么目标对应A跟唱的音量是相对较小的，此时目标对象A的音量信息强于目标对象B的音量信息，可以将目标对象A的音频信息作为待展示音频。
S230、将至少一个混音音频和待展示音频均作为待处理视频帧的目标音频。
在本实施例中，将所确定的混音音频与待展示音频进行双音轨播放。也就是说目标音频中，不仅包含混音音频，而且还包含音量信息相对较大目标对象音频。这样设置的好处在于：可以强化音量大的音频信息，弱化音量小的音频信息，以使播放的音频更加的和谐动听。
S240、基于目标音频和至少一个目标对象，确定与待处理视频帧相对应的特效视频帧。
本公开实施例的技术方案，可以采用多种方式，确定与至少一个目标对象所对应的混音音频，即可以基于对显示界面上至少一个混音控件的触发操作，确定至少一个混音音频；可以根据至少一个目标对象的对象属性，确定至少一个混音音频；也可以根据待处理视频帧中的音频信息，确定至少一个混音音频。
通过多种方式确定出的混音音频与用户之间的适配性相对较高，相应的，基于混音音频与目标对象的音频信息确定出的目标音频，与实际效果是最接近的，从而提高了特效的展示效果，也扩大了本方案的适用范围。
图5为本公开实施例所提供的一种生成特效视频的方法的流程示意图，在前述实施例的基础上，在特效展示界面中显示更加丰富的展示内容，营造逼真的现场氛围。具体的实施方式可以参见本实施例技术方案。其中，与上述实施例相同或者相应的技术术语在此不再赘述。如图5所示，该方法包括如下步骤：
S310、在检测到满足混音条件的情况下，确定待处理视频帧中至少一个目标对象所对应的至少一个混音音频。
S320、基于至少一个混音音频以及至少一个目标对象的音频信息，确定待处理视频帧的目标音频。
S330、确定与至少一个目标对象对应的至少一个分屏图像。
在本实施例中，待处理视频帧中可以显示一个或者多个目标对象。如果待处理视频帧中仅有一个目标对象，可以将这一个目标对象所对应的图像内容进行复制，得到分屏图像，并将分屏图像显示于显示界面中的预设位置。如果待处理视频帧中有多个目标对象，可以将多个目标对象所对应的图像内容进行整体的复制，得到分屏图像，并将分屏图像显示于显示界面中。
可选的，每个分屏图像中包括至少一个目标对象，或，每个分屏图像中包括一个目标对象。
在本实施例中，如果待处理视频帧中仅有一个目标对象，分屏图像中可以包括一个目标对象，参见图6。而如果待处理视频帧中有多个目标对象，分屏图像可以是通过两种方式得到，第一种方式是：可以将多个目标对象所对应的图像内容进行整体的抠图，多个目标对象的整体抠图内容为分屏图像，参见图7。第二种方式是：可以将多个目标对象所对应的图像内容进行拆分处理，即将多个目标对象分别拆分成独立的分屏图像，并显示在预设的位置上，参见图8。这样设置的好处在于：无论目标对象的数量是多少，都可以根据用户的选择确定分屏图像，增强了用户的使用体验。
目标对象在显示界面中的展示效果还可以是：对至少一个目标对象分割处理，确定对象分割图像；将至少一个目标对象作为待处理视频帧的中心，并按照预设缩放比例在中心的两侧堆叠显示对象分割图像，以更新特效视频帧。
在本实施例中，如果待处理视频帧中仅有一个目标对象，可以分割处理目标对象所对应的图像，随后以目标对象为中心，将对象分割图像按照预设缩放比例在中心的两侧堆叠显示对象分割图像，参见图9。如果待处理视频帧中有多个目标对象，可以将多个目标对象所对应的图像内容进行整体的分割处理，得到多个目标对象整体的对象分割图像，并按照预设缩放比例在中心的两侧堆叠显示多个目标对象整体的对象分割图像，参见图10。另外，也可以是将多个目 标对象的分别进行分割处理。示例性的，待处理视频帧中包括目标对象A和目标对象B，将目标对象A和目标对象B分别进行分割处理，以目标对象A和目标对象B的整体图像为中心，按照预设缩放比例在中心的左侧堆叠目标对象A所对应的对象分割图像，按照预设缩放比例在中心的右侧堆叠目标对象B所对应的对象分割图像，参见图11，其中，缩放比例可以是在原有图像的基础上缩小百分之二十。这样设置的好处在于：在特效展示页面中显示更多的对象分割图像，使得特效展示效果体现出现场合唱的情景，增强了特效展示效果的趣味性。
S340、基于至少一个分屏图像、目标音频以及待处理视频帧，确定特效视频帧。
在本实施例中，将分屏图像、目标音频以及待处理视频帧进行整体的叠加，得到既有音频特效又有图像特效的特效视频帧，随后可以基于多个特效视频帧进行拼合，生成一个可以展示合唱效果的特效视频。
本公开实施例的技术方案，在对音频进行特效处理的基础上，基于目标对象，可以确定目标对象所对应的多个分屏图像，进而将分屏图像、目标音频以及待处理视频帧进行整体的叠加，得到既有音频特效又有图像特效的特效视频帧。即除了对音频进行特效处理，还对至少一个目标对象对应的图像进行特效处理，实现对音频以及图像进行同步处理，以提高特效画面的显示内容，使得特效展示效果体现出现场合唱的情景，提高了画面内容的丰富度。
图12为本公开实施例所提供的另一种生成特效视频的方法的流程示意图，在前述实施例的基础上，在特效展示界面中显示3D话筒，且可以实时对准目标对象，营造逼真的现场氛围。具体的实施方式可以参见本实施例技术方案。其中，与上述实施例相同或者相应的技术术语在此不再赘述。如图12所示，该方法具体包括如下步骤：
S410、在检测到满足混音条件的情况下，确定待处理视频帧中至少一个目标对象所对应的至少一个混音音频。
S420、基于至少一个混音音频以及至少一个目标对象的音频信息，确定待处理视频帧的目标音频。
S430、基于目标音频和至少一个目标对象，确定与待处理视频帧相对应的特效视频帧。
S440、在特效视频帧中显示3D话筒。
在本实施例中，从至少一个目标对象中确定一个与3D话筒相对应的对准对 象，根据对准对象的位置信息调整特效视频帧中3D话筒的显示位置。
示例性的，3D话筒在特效视频帧中的位置，参见图13。这样设置的好处在于：在特效展示页面中显示3D话筒，使得特效展示效果更加逼真，增强了特效展示效果的丰富性。
可选的，在特效视频帧中显示3D话筒可以包括以下步骤：从至少一个目标对象中确定与3D话筒相对应的对准对象；根据对准对象的目标位置信息，调整3D话筒在特效视频帧中的话筒显示位置；其中，话筒显示位置包括话筒偏转角度和/或话筒于特效视频帧中的显示高度。这样设置的好处在于：话筒的显示位置可以根据目标对象的位移进行调整，提高了话筒与对准对象之间的匹配度，从而增强了特效展示效果的丰富性和趣味性。
在实际应用过程中，确定对准对象的方式可以包括两种，一种是基于图像的深度信息确定对准对象，另一种是基于画面显示比例确定对准对象。
基于画面显示比例确定对准对象的实现方式为：确定视频帧中每个目标对象在画面中的显示比例，可以将显示比例最大的目标对象作为对准对象。基于深度信息确定对准对象可以是：深度信息可以表征摄像头与用户之间的距离，距离摄像头越近的用户，深度信息越小；距离摄像头越远的用户，深度信息越大。确定待处理视频帧中每个目标对象所对应深度图像，随后，计算目标对象人像中每个点所对应的深度值，进而计算各人像点深度值的平均值，最后得到每个目标对象的深度信息，将深度信息最小的目标对象作为对准对象。
在本实施例中，待处理视频帧中对准对象于显示界面中的显示位置可以存在一定的变化，例如，有一定的旋转角度等，此时可以根据对准对象的偏转角度适应性的调整3D话筒的显示位置。对准对象的目标位置信息可以是预先设定的定点，例如可以是目标对象的鼻尖定点。鼻尖定点的确定过程为：首先基于面部检测算法实时的追踪鼻尖定点的位置信息，进而根据鼻尖定点的位置信息与预先定义的基准线的偏转角度，适应性的调整3D话筒的偏转角度，以达到3D话筒实时追随对准对象的效果。
示例性的，鼻尖定点的位置信息可以用空间坐标点表征，基于空间坐标可以确定鼻尖定点的法线，而基准线对应有一条法线，进而可以计算鼻尖定点的法线与基准线对应法线的夹角，所计算的夹角即为话筒偏转角度。话筒根据偏转角度，调整其显示位置。可选的，偏转角度范围可以固定在[-30°，30°]之间的。即，可以基于偏转角度范围和实际偏转角度确定话筒的偏转角度。
在实际用过程中，目标用户在拍摄视频的过程中，目标用户可能距离摄像头时远时近，此时目标对象在待处理视频帧中的显示位置可能存在上下移动的 情形，需要调整3D话筒的相对显示高度。
本公开实施例的技术方案，在对音频以及目标对象的图像进行同步特效处理的基础上，还可以在特效视频帧中实时显示3D话筒，并基于目标对象的显示位置信息，调整3D话筒于显示界面中的显示位置，以使3D话筒与目标对象是实时相匹配的，从而达到基于3D话筒采集目标对象音频信息的效果，提升了特效展示效果的逼真性，进一步提升特效展示的趣味性。
图14为本公开实施例所提供的一种生成特效视频的装置的结构示意图，如图14所示，装置包括：混音音频确定模块510、目标音频确定模块520和特效视频帧确定模块530。
混音音频确定模块510，设置为在检测到满足混音条件的情况下，确定待处理视频帧中至少一个目标对象所对应的至少一个混音音频；其中，待处理视频帧为实时采集的视频帧或录制视频中的视频帧；目标音频确定模块520，设置为基于至少一个混音音频以及至少一个目标对象的音频信息，确定待处理视频帧的目标音频；特效视频帧确定模块530，设置为基于目标音频和至少一个目标对象，确定与待处理视频帧相对应的特效视频帧。
在上述技术方案的基础上，混音条件包括下述至少一种：触发与混音特效相对应的特效道具；显示界面中包括至少一个目标对象；触发拍摄控件；检测到基于触发的视频处理控件上传的录制视频。
在上述技术方案的基础上，混音音频确定模块510包括：以下至少之一：触发操作确定子模块、对象属性确定子模块和混音音频确定子模块。
触发操作确定子模块，设置为基于对显示界面上至少一个混音控件的触发操作，确定至少一个混音音频；其中，至少一个混音控件对应于至少一个待选择混音音频；对象属性确定子模块，设置为根据至少一个目标对象的对象属性，确定至少一个混音音频；混音音频确定子模块，设置为根据待处理视频帧中的音频信息，确定至少一个混音音频。
在上述技术方案的基础上，对象属性确定子模块包括：面部算法识别单元和属性类别确定单元。
面部算法识别单元，设置为基于面部检测算法识别至少一个目标对象的对象属性；属性类别确定单元，设置为基于对象属性的属性类别数量和对象属性，从预先制作的至少一个待选择混音中，确定出与属性类别数量相一致的混音音频。
在上述技术方案的基础上，混音音频确定子模块包括：和声旋律确定单元 和混音音频确定单元。
和声旋律确定单元，设置为根据待处理视频帧中音频信息的伴奏信息和和声中的目标声部，确定和声旋律；混音音频确定单元，设置为基于和声旋律中的音调信息和音频信息中的音调信息，确定至少一个混音音频。
在上述技术方案的基础上，混音音频确定单元是设置为基于和声旋律中的音调信息、音频信息中的音调信息和至少一个目标对象的对象属性，确定至少一个混音音频。
在上述各技术方案的基础上，混音音频包括至少一个声部的和声伴奏或混音音频包括至少一个声部的和声伴奏和主唱音轨的音频。
在上述技术方案的基础上，目标音频确定模块520包括：音量信息确定子模块和目标音频确定子模块。
音量信息确定子模块，设置为根据音频信息所对应的音量信息，确定待展示音频；目标音频确定子模块，设置为将至少一个混音音频和待展示音频均作为待处理视频帧的目标音频。
在上述技术方案的基础上，特效视频帧确定模块530包括：分屏图像确定子模块和特效视频帧确定子模块。
分屏图像确定子模块，设置为确定与至少一个目标对象对应的至少一个分屏图像；特效视频帧确定子模块，设置为基于至少一个分屏图像、目标音频以及待处理视频帧，确定特效视频帧。
在上述各技术方案的基础上，每个分屏图像中包括至少一个目标对象，或，每个分屏图像中包括一个目标对象。
在上述技术方案的基础上，装置还包括：分割图像确定模块和特效视频更新模块。
分割图像确定模块，设置为对至少一个目标对象分割处理，确定对象分割图像；特效视频更新模块，设置为将至少一个目标对象作为待处理视频帧的中心，并按照预设缩放比例在中心的两侧堆叠显示对象分割图像，以更新特效视频帧。
在上述技术方案的基础上，装置还包括：话筒显示模块，设置为在特效视频帧中显示3D话筒。
在上述各技术方案的基础上，话筒显示模块还包括：对准对象确定子模块和话筒位置调整子模块。
对准对象确定子模块，设置为从至少一个目标对象中确定与3D话筒相对应 的对准对象；话筒位置调整子模块，设置为根据对准对象的目标位置信息，调整3D话筒在特效视频帧中的话筒显示位置；其中，话筒显示位置包括话筒偏转角度和/或话筒于特效视频帧中的显示高度。
本公开实施例的技术方案，当检测到满足混音条件时，可以确定待处理视频帧中至少一个目标对象所对应的至少一个混音音频，进而基于所确定的混音音频以及至少一个目标对象的音频信息，可以确定多个音轨所对应的目标音频，通过对目标音频和目标对象进行融合处理，可以得到最终的特效视频帧。实现了不仅可以对画面内容进行处理，还可以对音频内容进行处理的技术效果，提升了特效展示效果的丰富性、趣味性，还进一步提升了目标用户使用体验的技术效果。
本公开实施例所提供的生成特效视频装置可执行本公开任意实施例所提供的生成特效视频的方法，具备执行方法相应的功能模块和效果。
上述装置所包括的多个单元和模块只是按照功能逻辑进行划分的，但并不局限于上述的划分，只要能够实现相应的功能即可；另外，多个单元和模块的名称也只是为了便于相互区分，并不用于限制本公开实施例的保护范围。
图15为本公开实施例所提供的一种电子设备的结构示意图。下面参考图15，图15示出了适于用来实现本公开实施例的电子设备(例如图15中的终端设备或服务器)600的结构示意图。本公开实施例中的终端设备可以包括但不限于诸如移动电话、笔记本电脑、数字广播接收器、个人数字助理(Personal Digital Assistan，PDA)、平板电脑(Portable Android Device，PAD)、便携式多媒体播放器(Portable Media Player，PMP)、车载终端(例如车载导航终端)等等的移动终端以及诸如数字电视(television，TV)、台式计算机等等的固定终端。图15示出的电子设备仅仅是一个示例，不应对本公开实施例的功能和使用范围带来任何限制。
如图15所示，电子设备600可以包括处理装置(例如中央处理器、图形处理器等)601，其可以根据存储在只读存储器(Read-Only Memory，ROM)602中的程序或者从存储装置608加载到随机访问存储器(Random Access Memory，RAM)603中的程序而执行多种适当的动作和处理。在RAM 603中，还存储有电子设备600操作所需的多种程序和数据。处理装置601、ROM 602以及RAM 603通过总线604彼此相连。输入/输出(Input/Output，I/O)接口605也连接至总线604。
通常，以下装置可以连接至I/O接口605：包括例如触摸屏、触摸板、键盘、 鼠标、摄像头、麦克风、加速度计、陀螺仪等的输入装置606；包括例如液晶显示器(Liquid Crystal Display，LCD)、扬声器、振动器等的输出装置607；包括例如磁带、硬盘等的存储装置608；以及通信装置609。通信装置609可以允许电子设备600与其他设备进行无线或有线通信以交换数据。虽然图15示出了具有多种装置的电子设备600，但是应理解的是，并不要求实施或具备所有示出的装置。可以替代地实施或具备更多或更少的装置。
根据本公开的实施例，上文参考流程图描述的过程可以被实现为计算机软件程序。例如，本公开的实施例包括一种计算机程序产品，其包括承载在非暂态计算机可读介质上的计算机程序，该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中，该计算机程序可以通过通信装置609从网络上被下载和安装，或者从存储装置608被安装，或者从ROM 602被安装。在该计算机程序被处理装置601执行时，执行本公开实施例的方法中限定的上述功能。
本公开实施方式中的多个装置之间所交互的消息或者信息的名称仅用于说明性的目的，而并不是用于对这些消息或信息的范围进行限制。
本公开实施例提供的电子设备与上述实施例提供的生成特效视频的方法属于同一发明构思，未在本实施例中详尽描述的技术细节可参见上述实施例，并且本实施例与上述实施例具有相同的效果。
本公开实施例提供了一种计算机存储介质，其上存储有计算机程序，该程序被处理器执行时实现上述实施例所提供的生成特效视频的方法。
本公开上述的计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、RAM、ROM、可擦式可编程只读存储器(Erasable Programmable Read-Only Memory，EPROM)或闪存、光纤、便携式紧凑磁盘只读存储器(Compact Disc Read-Only Memory，CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本公开中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。而在本公开中，计算机可读信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读信号介质 还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读信号介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：电线、光缆、射频(Radio Frequency，RF)等等，或者上述的任意合适的组合。
在一些实施方式中，客户端、服务器可以利用诸如超文本传输协议(HyperText Transfer Protocol，HTTP)之类的任何当前已知或未来研发的网络协议进行通信，并且可以与任意形式或介质的数字数据通信(例如，通信网络)互连。通信网络的示例包括局域网(Local Area Network，LAN)，广域网(Wide Area Network，WAN)，网际网(例如，互联网)以及端对端网络(例如，ad hoc端对端网络)，以及任何当前已知或未来研发的网络。
上述计算机可读介质可以是上述电子设备中所包含的；也可以是单独存在，而未装配入该电子设备中。
上述计算机可读介质承载有一个或者多个程序，当上述一个或者多个程序被该电子设备执行时，使得该电子设备：在检测到满足混音条件的情况下，确定待处理视频帧中至少一个目标对象所对应的至少一个混音音频；其中，待处理视频帧为实时采集的视频帧或录制视频中的视频帧；基于至少一个混音音频以及至少一个目标对象的音频信息，确定待处理视频帧的目标音频；基于目标音频和至少一个目标对象，确定与待处理视频帧相对应的特效视频帧。
可以以一种或多种程序设计语言或其组合来编写用于执行本公开的操作的计算机程序代码，上述程序设计语言包括但不限于面向对象的程序设计语言—诸如Java、Smalltalk、C++，还包括常规的过程式程序设计语言—诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络——包括LAN或WAN—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。
附图中的流程图和框图，图示了按照本公开的多种实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，该模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实际上可以基本并行 地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
描述于本公开实施例中所涉及到的单元和模块可以通过软件的方式实现，也可以通过硬件的方式来实现。其中，单元的名称和模块在并不构成对该单元和模块本身的限定，例如，混音音频确定模块还可以被描述为“在检测到满足混音条件的情况下，确定待处理视频帧中至少一个目标对象所对应的至少一个混音音频的模块”。
本文中以上描述的功能可以至少部分地由一个或多个硬件逻辑部件来执行。例如，非限制性地，可以使用的示范类型的硬件逻辑部件包括：现场可编程门阵列(Field Programmable Gate Array，FPGA)、专用集成电路(Application Specific Integrated Circuit，ASIC)、专用标准产品(Application Specific Standard Parts，ASSP)、片上系统(System on Chip，SOC)、复杂可编程逻辑设备(Complex Programmable Logic Device，CPLD)等等。
在本公开的上下文中，机器可读介质可以是有形的介质，其可以包含或存储以供指令执行系统、装置或设备使用或与指令执行系统、装置或设备结合地使用的程序。机器可读介质可以是机器可读信号介质或机器可读储存介质。机器可读介质可以包括但不限于电子的、磁性的、光学的、电磁的、红外的、或半导体系统、装置或设备，或者上述内容的任何合适组合。机器可读存储介质的更具体示例会包括基于一个或多个线的电气连接、便携式计算机盘、硬盘、RAM、ROM、EPROM或快闪存储器、光纤、便捷式CD-ROM、光学储存设备、磁储存设备、或上述内容的任何合适组合。
根据本公开的一个或多个实施例，【示例一】提供了一种生成特效视频的方法，该方法包括：在检测到满足混音条件的情况下，确定待处理视频帧中至少一个目标对象所对应的至少一个混音音频；其中，待处理视频帧为实时采集的视频帧或录制视频中的视频帧；基于至少一个混音音频以及至少一个目标对象的音频信息，确定待处理视频帧的目标音频；基于目标音频和至少一个目标对象，确定与待处理视频帧相对应的特效视频帧。
根据本公开的一个或多个实施例，【示例二】提供了一种生成特效视频的方法，该方法，还包括：可选的，基于对显示界面上至少一个混音控件的触发操作，确定至少一个混音音频；其中，至少一个混音控件对应于至少一个待选择混音音频；根据至少一个目标对象的对象属性，确定至少一个混音音频；根 据待处理视频帧中的音频信息，确定至少一个混音音频。
根据本公开的一个或多个实施例，【示例三】提供了一种生成特效视频的方法，该方法，还包括：可选的，根据至少一个目标对象的对象属性，确定至少一个混音音频，包括：基于面部检测算法识别至少一个目标对象的对象属性；基于对象属性的属性类别数量和对象属性，从预先制作的至少一个待选择混音中，确定出与属性类别数量相一致的混音音频。
根据本公开的一个或多个实施例，【示例四】提供了一种生成特效视频的方法，该方法，还包括：可选的，根据待处理视频帧中的音频信息，确定至少一个混音音频，包括：根据待处理视频帧中音频信息的伴奏信息和和声中的目标声部，确定和声旋律；基于和声旋律中的音调信息和音频信息中的音调信息，确定至少一个混音音频。
根据本公开的一个或多个实施例，【示例五】提供了一种生成特效视频的方法，该方法，还包括：可选的，基于和声旋律中的音调信息和音频信息中的音调信息，确定至少一个混音音频，包括：基于和声旋律中的音调信息、音频信息中的音调信息和至少一个目标对象的对象属性，确定至少一个混音音频。
根据本公开的一个或多个实施例，【示例六】提供了一种生成特效视频的方法，该方法，还包括：可选的，混音音频包括至少一个声部的和声伴奏或混音音频包括至少一个声部的和声伴奏和主唱音轨的音频。
根据本公开的一个或多个实施例，【示例七】提供了一种生成特效视频的方法，该方法，还包括：可选的，基于至少一个混音音频以及至少一个目标对象的音频信息，确定待处理视频帧中目标音频，包括：根据音频信息所对应的音量信息，确定待展示音频；将至少一个混音音频和待展示音频均作为待处理视频帧的目标音频。
根据本公开的一个或多个实施例，【示例八】提供了一种生成特效视频的方法，该方法，还包括：可选的，基于目标音频和至少一个目标对象，确定与待处理视频帧相对应的特效视频帧，包括：确定与至少一个目标对象性对应的至少一个分屏图像；基于至少一个分屏图像、目标音频以及待处理视频帧，确定特效视频帧。
根据本公开的一个或多个实施例，【示例九】提供了一种生成特效视频的方法，该方法，还包括：可选的，每个分屏图像中包括至少一个目标对象，或，每个分屏图像中包括一个目标对象。
根据本公开的一个或多个实施例，【示例十】提供了一种生成特效视频的方法，该方法，还包括：可选的，对至少一个目标对象分割处理，确定对象分 割图像；将至少一个目标对象作为待处理视频帧的中心，并按照预设缩放比例在中心的两侧堆叠显示对象分割图像，以更新特效视频帧。
根据本公开的一个或多个实施例，【示例十一】提供了一种生成特效视频的方法，该方法，还包括：可选的，在特效视频帧中显示3D话筒。
根据本公开的一个或多个实施例，【示例十二】提供了一种生成特效视频的方法，该方法，还包括：可选的，从至少一个目标对象中确定与3D话筒相对应的对准对象；根据对准对象的目标位置信息，调整3D话筒在特效视频帧中的话筒显示位置；其中，话筒显示位置包括话筒偏转角度和/或话筒于特效视频帧中的显示高度。
根据本公开的一个或多个实施例，【示例十三】提供了一种生成特效视频的方法，该方法，还包括：可选的，混音条件包括下述至少一种：触发与混音特效相对应的特效道具；显示界面中包括至少一个目标对象；触发拍摄控件；检测到基于触发的视频处理控件上传的录制视频。
根据本公开的一个或多个实施例，【示例十四】提供了一种生成特效视频装置，该装置包括：混音音频确定模块，设置为在检测到满足混音条件的情况下，确定待处理视频帧中至少一个目标对象所对应的至少一个混音音频；其中，待处理视频帧为实时采集的视频帧或录制视频中的视频帧；目标音频确定模块，设置为基于至少一个混音音频以及至少一个目标对象的音频信息，确定待处理视频帧的目标音频；特效视频帧确定模块，设置为基于目标音频和至少一个目标对象，确定与待处理视频帧相对应的特效视频帧。
虽然采用特定次序描绘了多个操作，但是这不应当理解为要求这些操作以所示出的特定次序或以顺序次序执行来执行。在一定环境下，多任务和并行处理可能是有利的。同样地，虽然在上面论述中包含了具体实现细节，但是这些不应当被解释为对本公开的范围的限制。在单独的实施例的上下文中描述的一些特征还可以组合地实现在单个实施例中。在单个实施例的上下文中描述的多种特征也可以单独地或以任何合适的子组合的方式实现在多个实施例中。
尽管已经采用特定于结构特征和/或方法逻辑动作的语言描述了本主题，但是应当理解所附权利要求书中所限定的主题未必局限于上面描述的特定特征或动作。上面所描述的特定特征和动作仅仅是实现权利要求书的示例形式。

Claims (16)

  1. 一种生成特效视频的方法，包括：
    在检测到满足混音条件的情况下，确定待处理视频帧中至少一个目标对象所对应的至少一个混音音频；其中，待处理视频帧为实时采集的视频帧或录制视频中的视频帧；
    基于至少一个混音音频以及至少一个目标对象的音频信息，确定待处理视频帧的目标音频；
    基于目标音频和至少一个目标对象，确定与待处理视频帧相对应的特效视频帧。
  2. 根据权利要求1的方法，其中，确定至少一个混音音频包括下述至少一种方式：
    基于对显示界面上至少一个混音控件的触发操作，确定至少一个混音音频；其中，至少一个混音控件对应于至少一个待选择混音音频；
    根据至少一个目标对象的对象属性，确定至少一个混音音频；
    根据待处理视频帧中的音频信息，确定至少一个混音音频。
  3. 根据权利要求2的方法，其中，根据至少一个目标对象的对象属性，确定至少一个混音音频，包括：
    基于面部检测算法识别至少一个目标对象的对象属性；
    基于对象属性的属性类别数量和所述对象属性，从预先制作的至少一个待选择混音音频中，确定出至少一个混音音频。
  4. 根据权利要求2的方法，其中，根据待处理视频帧中的音频信息，确定至少一个混音音频，包括：
    根据待处理视频帧中的音频信息的伴奏信息和和声中的目标声部，确定和声旋律；
    基于和声旋律中的音调信息和音频信息中的音调信息，确定至少一个混音音频。
  5. 根据权利要求4的方法，其中，基于和声旋律中的音调信息和音频信息中的音调信息，确定至少一个混音音频，包括：
    基于和声旋律中的音调信息、音频信息中的音调信息和至少一个目标对象的对象属性，确定至少一个混音音频。
  6. 根据权利要求1-5中任一项的方法，其中，每个混音音频包括至少一个声部的和声伴奏或每个混音音频包括至少一个声部的和声伴奏和主唱音轨的音 频。
  7. 根据权利要求1的方法，其中，基于至少一个混音音频以及至少一个目标对象的音频信息，确定待处理视频帧中目标音频，包括：
    根据音频信息所对应的音量信息，确定待展示音频；
    将至少一个混音音频和待展示音频均作为待处理视频帧的目标音频。
  8. 根据权利要求1的方法，其中，基于目标音频和至少一个目标对象，确定与待处理视频帧相对应的特效视频帧，包括：
    确定与至少一个目标对象对应的至少一个分屏图像；
    基于至少一个分屏图像、目标音频以及待处理视频帧，确定特效视频帧。
  9. 根据权利要求8的方法，其中，每个分屏图像中包括至少一个目标对象，或，每个分屏图像中包括一个目标对象。
  10. 根据权利要求1的方法，还包括：
    对至少一个目标对象分割处理，确定对象分割图像；
    将至少一个目标对象作为待处理视频帧的中心，并按照预设缩放比例在中心的两侧堆叠显示对象分割图像，以更新特效视频帧。
  11. 根据权利要求1的方法，还包括：
    在特效视频帧中显示3D话筒。
  12. 根据权利要求11的方法，还包括：
    从至少一个目标对象中确定与3D话筒相对应的对准对象；
    根据对准对象的目标位置信息，调整3D话筒在特效视频帧中的话筒显示位置；
    其中，话筒显示位置包括以下至少之一：话筒偏转角度和话筒于特效视频帧中的显示高度。
  13. 根据权利要求1的方法，其中，混音条件包括下述至少一种：
    触发与混音特效相对应的特效道具；
    显示界面中包括至少一个目标对象；
    触发拍摄控件；
    检测到基于触发的视频处理控件上传的录制视频。
  14. 一种生成特效视频的装置，包括：
    混音音频确定模块，设置为在检测到满足混音条件的情况下，确定待处理视频帧中至少一个目标对象所对应的至少一个混音音频；其中，待处理视频帧为实时采集的视频帧或录制视频中的视频帧；
    目标音频确定模块，设置为基于至少一个混音音频以及至少一个目标对象的音频信息，确定待处理视频帧的目标音频；
    特效视频帧确定模块，设置为基于目标音频和至少一个目标对象，确定与待处理视频帧相对应的特效视频帧。
  15. 一种电子设备，包括：
    至少一个处理器；
    存储装置，设置为存储至少一个程序，
    当所述至少一个程序被所述一个或多个处理器执行，使得所述至少一个处理器实现如权利要求1-13中任一所述的生成特效视频的方法。
  16. 一种包含计算机可执行指令的存储介质，所述计算机可执行指令在由计算机处理器执行时用于执行如权利要求1-13中任一所述的生成特效视频的方法。
PCT/CN2023/119023 2022-09-29 2023-09-15 生成特效视频的方法、装置、电子设备及存储介质 Ceased WO2024067157A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US19/115,753 US20260105705A1 (en) 2022-09-29 2023-09-15 Method of generating effect video electronic device and storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202211204819.0A CN115623146B (zh) 2022-09-29 2022-09-29 一种生成特效视频的方法、装置、电子设备及存储介质
CN202211204819.0 2022-09-29

Publications (1)

Publication Number Publication Date
WO2024067157A1 true WO2024067157A1 (zh) 2024-04-04

Family

ID=84860655

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/119023 Ceased WO2024067157A1 (zh) 2022-09-29 2023-09-15 生成特效视频的方法、装置、电子设备及存储介质

Country Status (3)

Country Link
US (1) US20260105705A1 (zh)
CN (1) CN115623146B (zh)
WO (1) WO2024067157A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115623146B (zh) * 2022-09-29 2025-11-04 北京字跳网络技术有限公司 一种生成特效视频的方法、装置、电子设备及存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160057316A1 (en) * 2011-04-12 2016-02-25 Smule, Inc. Coordinating and mixing audiovisual content captured from geographically distributed performers
CN107888843A (zh) * 2017-10-13 2018-04-06 深圳市迅雷网络技术有限公司 用户原创内容的混音方法、装置、存储介质及终端设备
CN114220409A (zh) * 2021-12-14 2022-03-22 腾讯音乐娱乐科技（深圳）有限公司 一种音频处理方法及计算机装置
CN114245036A (zh) * 2021-12-21 2022-03-25 北京达佳互联信息技术有限公司 视频制作方法及装置
CN114630057A (zh) * 2022-03-11 2022-06-14 北京字跳网络技术有限公司 确定特效视频的方法、装置、电子设备及存储介质
CN115623146A (zh) * 2022-09-29 2023-01-17 北京字跳网络技术有限公司 一种生成特效视频的方法、装置、电子设备及存储介质

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112073748B (zh) * 2019-06-10 2022-03-18 北京字节跳动网络技术有限公司 全景视频的处理方法、装置及存储介质

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160057316A1 (en) * 2011-04-12 2016-02-25 Smule, Inc. Coordinating and mixing audiovisual content captured from geographically distributed performers
CN107888843A (zh) * 2017-10-13 2018-04-06 深圳市迅雷网络技术有限公司 用户原创内容的混音方法、装置、存储介质及终端设备
CN114220409A (zh) * 2021-12-14 2022-03-22 腾讯音乐娱乐科技（深圳）有限公司 一种音频处理方法及计算机装置
CN114245036A (zh) * 2021-12-21 2022-03-25 北京达佳互联信息技术有限公司 视频制作方法及装置
CN114630057A (zh) * 2022-03-11 2022-06-14 北京字跳网络技术有限公司 确定特效视频的方法、装置、电子设备及存储介质
CN115623146A (zh) * 2022-09-29 2023-01-17 北京字跳网络技术有限公司 一种生成特效视频的方法、装置、电子设备及存储介质

Also Published As

Publication number Publication date
CN115623146A (zh) 2023-01-17
US20260105705A1 (en) 2026-04-16
CN115623146B (zh) 2025-11-04

Similar Documents

Publication Publication Date Title
WO2022121558A1 (zh) 一种直播演唱方法、装置、设备和介质
WO2022152064A1 (zh) 视频生成方法、装置、电子设备和存储介质
WO2020259130A1 (zh) 精选片段处理方法、装置、电子设备及可读介质
CN110324718A (zh) 音视频生成方法、装置、电子设备及可读介质
CN115633206A (zh) 媒体内容展示方法、装置、设备及存储介质
WO2022042035A1 (zh) 视频制作方法、装置、设备及存储介质
US12555607B2 (en) Audio data processing method and apparatus, and device and storage medium
US11272136B2 (en) Method and device for processing multimedia information, electronic equipment and computer-readable storage medium
WO2020259133A1 (zh) 录制热门片段方法、装置、电子设备和可读介质
WO2024104181A1 (zh) 确定音频的方法、装置、电子设备及存储介质
WO2024032635A1 (zh) 媒体内容获取方法、装置、设备、可读存储介质及产品
CN112312163B (zh) 视频生成方法、装置、电子设备及存储介质
WO2024067157A1 (zh) 生成特效视频的方法、装置、电子设备及存储介质
CN117459776A (zh) 一种多媒体内容的播放方法、装置、电子设备及存储介质
JP2014123085A (ja) カラオケにおいて歌唱に合わせて視聴者が行う身体動作等をより有効に演出し提供する装置、方法、およびプログラム
US20240404548A1 (en) Method, apparatus, device and storage medium for video recording
WO2023174073A1 (zh) 视频生成方法、装置、设备、存储介质和程序产品
CN116647714A (zh) 视频生成方法、装置、电子设备以及存储介质
CN110225384A (zh) 状态消息的推送方法、交互内容的切换方法、装置及设备
JP6443205B2 (ja) コンテンツ再生システム、コンテンツ再生装置、コンテンツ関連情報配信装置、コンテンツ再生方法、及びコンテンツ再生プログラム
US20250008166A1 (en) Optimization of live stream productions
CN115774539B (zh) 和声处理方法、装置、设备及介质
CN117714813A (zh) 视频生成方法、装置、介质及设备
CN114979800B (zh) 交互式录屏方法、电子设备及可读存储介质
CN117059121A (zh) 音频处理方法、装置、设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23870391

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 14.07.2025)

122 Ep: pct application non-entry in european phase

Ref document number: 23870391

Country of ref document: EP

Kind code of ref document: A1