HK40028958B - Method and apparatus for processing video special effect and electronic device - Google Patents

Description

Methods, devices, and electronic equipment for processing video special effects

Technical Field

This invention relates to digital multimedia technology, and more particularly to a method, apparatus, electronic device, and computer-readable storage medium for processing video special effects.

Background Technology

The development of the internet, especially the mobile internet, has led to an unprecedented use of video as a medium for information dissemination. To enhance the expressiveness of the information conveyed by videos and increase viewership, related technologies often add video effects.

For example, in related technologies, special effects animations can be designed based on professional video editing and design software such as AE (Adobe After Effects). AE can use Airbnb's popular open-source animation library and Portable Animated Graphics (PAG) solutions.

However, the duration of video effects designed by video editing software is fixed, making it difficult to adapt to diverse application scenarios. If various video effects with different playback durations are generated for various possible scenarios, it will not only waste computing resources but also affect the real-time performance of the video presentation.

Summary of the Invention

This invention provides a method, apparatus, electronic device, and computer-readable storage medium for processing video special effects, which can achieve free scaling of video special effects duration.

The technical solution of this invention is implemented as follows:

This invention provides a method for processing video special effects, the method comprising:

Obtain video effects files and extract duration scaling logic from the video effects files;

Obtain the target playback duration required to be achieved by the video effects file, wherein the target playback duration is different from the original playback duration of the video effects file;

Based on the duration scaling logic, determine the special effects frames in the video special effects file that correspond to the target timeline;

The length of the target timeline is the same as the target playback duration;

Rendering is performed based on the effect frames corresponding to the target timeline to form a video effect playback process that conforms to the target playback duration.

This invention provides a video effects processing apparatus, the apparatus comprising:

The file acquisition module is used to acquire video effects files and extract duration scaling logic from the video effects files.

The duration acquisition module is used to acquire the target playback duration required by the video effects file, wherein the target playback duration is different from the original playback duration of the video effects file;

The special effects frame determination module is used to determine the special effects frame in the video special effects file corresponding to the target timeline according to the duration scaling logic;

The length of the target timeline is the same as the target playback duration;

The rendering module is used to render special effects frames corresponding to the target timeline to form a video special effects playback process that conforms to the target playback duration.

In the above scheme, the file acquisition module is further configured to:

Perform one of the following processes to obtain the encoded export file corresponding to the special effects object:

The multiple layer structures of the special effects object are encoded to obtain the corresponding encoded export file of the special effects object;

Encode multiple effect frames of the effect object to obtain the corresponding encoded export file of the effect object;

Multiple effect frames of the effect object are subjected to video format compression processing, and the resulting video format compression processing result is encoded to obtain the encoded export file corresponding to the effect object;

The duration scaling type and duration scaling range are encapsulated in the encoding export file to obtain the video effects file corresponding to the effects object.

In the above scheme, the duration scaling logic includes duration scaling intervals and corresponding duration scaling types; the file acquisition module is further used for:

The video effects file is decoded to obtain at least one duration scaling interval and the corresponding duration scaling type for the video effects file.

The duration scaling type includes any one of the following types: linear time scaling type; time repetition type; and time reverse repetition type.

In the above solution, the duration acquisition module is further used for:

When there are multiple duration extension intervals, the video effect file is split into multiple video effect sub-files, which is the same number as the duration extension interval, and the target playback duration for each video effect sub-file is obtained.

When the number of duration scaling intervals is one, the overall target playback duration for the video effects file is obtained.

In the above solution, the duration acquisition module is further configured to: after acquiring the target playback duration required by the video effects file,

When there are multiple duration scaling intervals, the following processing is performed for each video effects sub-file:

Obtain the original timeline of the corresponding effect object from the video effects sub-file;

Keeping the frame rate of the original timeline unchanged, the original timeline is subjected to duration scaling to obtain a target timeline corresponding to the target playback duration;

When the number of duration scaling intervals is one, the following processing is performed on the video effects file:

Obtain the original timeline of the corresponding effect object from the video effects file;

Keeping the frame rate of the original timeline unchanged, the original timeline is subjected to duration scaling to obtain a target timeline corresponding to the target playback duration.

In the above scheme, when the number of duration expansion intervals is multiple, the special effects frame determination module is further configured to:

Perform the following processing for each duration scaling interval:

Obtain multiple effect frames including the effect object from the video effect sub-file, and the timestamp corresponding to each effect frame on the original timeline, as the original effect frame timestamp for each effect frame;

Based on the duration scaling interval and the original effect frame timestamp corresponding to each of the effect frames, the effect frame corresponding to each timestamp on the target timeline is determined from the plurality of effect frames.

In the above scheme, when the number of duration expansion intervals is one, the special effects frame determination module is further configured to:

Obtain multiple effect frames including the effect object from the video effect file, and the timestamp corresponding to each effect frame on the original timeline, as the original effect frame timestamp for each effect frame;

Based on the duration scaling interval and the original effect frame timestamp corresponding to each of the effect frames, the effect frame corresponding to each timestamp on the target timeline is determined from the plurality of effect frames.

In the above scheme, the special effects frame determination module is further used for:

Each timestamp on the target timeline is sequentially used as the target timestamp to perform the following processing:

Based on the duration scaling interval, determine the timestamp corresponding to the target timestamp on the original timeline;

When the timestamp corresponding to the target timestamp on the original timeline overlaps with any of the original special effects frame timestamps, the special effects frame corresponding to the overlapping original special effects frame timestamp is determined as the special effects frame corresponding to the target timestamp.

When the timestamp corresponding to the target timestamp on the original timeline does not overlap with any of the original effect frame timestamps, the original timestamp with the smallest distance from the timestamp corresponding to the target timestamp on the original timeline is determined, and the effect frame corresponding to the determined original effect frame timestamp is determined as the effect frame corresponding to the target timestamp.

In the above scheme, the special effects frame determination module is further used for:

For each of the aforementioned duration scaling intervals, the following processing is performed:

When the target timestamp is not greater than the starting timestamp of the duration expansion interval, the target timestamp is determined as the original timestamp corresponding to the original timeline;

When the target timestamp is greater than the start timestamp of the duration expansion interval and less than the end timestamp of the duration expansion interval, the target timestamp is mapped based on the duration expansion type to obtain the corresponding original timestamp.

When the target timestamp is greater than or equal to the termination timestamp and less than the target playback duration, a first difference between the original playback duration and the target playback duration is determined. The first difference is summed with the target timestamp, and the summation result is used to determine the timestamp corresponding to the target timestamp on the original timeline.

In the above scheme, the special effects frame determination module is further used for:

When the duration scaling type is the time linear reduction type or the time linear stretch type, the second difference between the target playback duration and the original playback duration is determined as the scaling length, and the scaling length is summed with the length of the duration scaling interval;

The length of the time-scaling interval is compared with the summation result to obtain the scaling factor;

Determine a third difference between the target timestamp and the starting timestamp, and multiply the third difference by the scaling factor;

The result of the multiplication process is summed with the starting timestamp to obtain the corresponding original timestamp.

In the above scheme, the special effects frame determination module is further used for:

When the duration stretching type is the time repetition type, determine the fourth difference between the target timestamp and the starting timestamp, and perform a remainder operation on the fourth difference and the length of the duration stretching interval.

The remainder result is summed with the starting timestamp to obtain the corresponding original timestamp.

In the above scheme, the special effects frame determination module is further used for:

When the duration stretching type is the time reverse repeating type, determine the fifth difference between the target timestamp and the starting timestamp, and perform a remainder operation on the fifth difference and the length of the duration stretching interval.

The ratio result of the corresponding remainder processing is rounded to obtain the rounded result;

When the rounding result is even, the remainder result is summed with the starting timestamp to obtain the corresponding original timestamp;

When the rounding result is odd, the sixth difference between the length of the duration expansion interval and the result of the remainder processing is determined, and the sixth difference is summed with the starting timestamp to obtain the corresponding original timestamp.

This invention provides an electronic device, comprising:

Memory, used to store executable instructions;

The processor, when executing executable instructions stored in the memory, implements the video effects processing method provided in the embodiments of the present invention.

This invention provides a computer-readable storage medium storing executable instructions that, when executed by a processor, implement the video effects processing method provided in this invention.

The embodiments of the present invention have the following beneficial effects:

By encapsulating duration scaling logic in video effects files, a single video effects file can be freely scaled to the playback duration required by different application scenarios, making it universally applicable. Rendering is then performed on this basis to form the video effects playback process, saving the huge consumption of computing and time resources that would otherwise be required to create a large number of video effects files with different playback durations.

Attached Figure Description

Figure 1 is a schematic diagram of an optional structure of the video effects processing system provided in an embodiment of the present invention;

Figure 2 is a schematic diagram of an optional structure of the video effects processing device provided in an embodiment of the present invention;

Figures 3A-3E are schematic diagrams of an optional process for processing video special effects provided in an embodiment of the present invention;

Figure 4 is a schematic diagram illustrating the application effect of the video effects processing method provided in the embodiment of the present invention;

Figure 5 is a flowchart of the video effects processing system provided in an embodiment of the present invention;

Figures 6A-6C are schematic diagrams illustrating the video effects processing method provided in the embodiments of the present invention;

Figure 7 is a timeline diagram of the video effects processing method provided in the embodiments of the present invention.

Detailed Implementation

To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of the invention described herein can be implemented in an order other than that illustrated or described herein.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

Before providing a further detailed description of the embodiments of the present invention, the nouns and terms involved in the embodiments of the present invention will be explained, and the nouns and terms involved in the embodiments of the present invention shall be interpreted as follows.

1) AE: Short for Adobe After Effects, it is a graphics and video processing software launched by Adobe. It is suitable for organizations engaged in design and video special effects, including TV stations, animation production companies, personal post-production studios and multimedia studios. It belongs to the layer-type post-production software.

2) Video effects files, which are binary files that carry effects content, such as PAG files, are sticker animations stored in binary file format.

3) Original timeline: The timeline corresponding to the entire video effects file, or the timeline corresponding to the effects portion of a video effects sub-file when played.

4) Target timeline: The timeline corresponding to the playback of a complete special effects object in a video effects file after scaling, or the timeline corresponding to the playback of a partial special effects object in a video effects sub-file after scaling.

Among related technologies, mobile internet clients using Adobe After Effects (AE) to implement animations include Airbnb's open-source Lottie solution and the PAG solution. Both solutions streamline the workflow from AE animation design to mobile presentation. Designers create animations in AE and export them as animation files via an export plugin, which are then loaded and rendered on the mobile device using an SDK, significantly reducing development costs. However, the animation duration in the AE-designed animation files is fixed in both solutions. During the implementation of this invention, the applicant discovered that in some user interface animations and video editing scenarios, it is necessary to externally control the duration of the animation files. For example, some intervals of the animation file can be fixed, while others can be linearly stretched or looped. For instance, if the sticker animation is 2 seconds long but the actual required animation length is 4 seconds, the external system needs to stretch the sticker animation to 4 seconds or redraw the sticker animation, i.e., play the sticker animation twice.

To address the technical problem of the contradiction between fixed video effects files and arbitrary target playback duration requirements in related technologies, this invention provides a video effects processing method that supports time scaling of fixed animation files. External application platforms only need to set the target playback time of the animation file, and the animation file can then scale according to the user-configured scaling logic. The playback duration scaling of the video effects file is controlled by the duration scaling logic within the video effects file. After decoding the video effects file, processing and rendering according to the duration scaling logic achieves the desired video effects playback duration. This method can be directly applied to various applications and platforms, is not limited by the platform's operating system, and has an extremely simple implementation process.

The following describes exemplary applications of the electronic device provided in the embodiments of the present invention. The electronic device provided in the embodiments of the present invention can be implemented as various types of terminal devices such as laptops, tablets, desktop computers, set-top boxes, and mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), or as a server. The following will describe exemplary applications when the electronic device is implemented as a terminal, with reference to FIG1.

Referring to Figure 1, which is an optional structural diagram of the video effects processing system provided in an embodiment of the present invention, the terminal 400 is connected to the server 200 through the network 300, which can be a wide area network or a local area network, or a combination of both.

Server 200 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services. Terminals can be smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, etc., but are not limited to these. Terminals and servers can be connected directly or indirectly via wired or wireless communication, and this application does not impose any restrictions on this connection.

Cloud computing is a computing model that distributes computing tasks across a resource pool consisting of a large number of computers, enabling various application systems to access computing power, storage space, and information services as needed. The network providing these resources is called the "cloud." From the user's perspective, the resources in the "cloud" are infinitely scalable, readily available, on-demand, expandable, and pay-as-you-go.

The following describes the application of the video effects processing method provided in the embodiments of the present invention in different application scenarios.

In one application scenario, the designer uses terminal 500 or calls the server's cloud service to design video effects files, and sends the video effects files to the client's server 200 (i.e., the backend server). Server 200 stores the received video effects files, or stores them in database 600 or a file system. The user uses a client running on terminal 400. The client can be various types of applications, such as games, social networking, short video, or online shopping. During the user's use, the business logic of server 200 to send videos is triggered. For example, server 200 periodically sends business usage reports, such as weekly battle reports and monthly consumption video reports in games. The specific content of the business usage reports is related to the client's business, and triggers server 200 to send video effects files that enhance the video's expressiveness. These video effects files can be pre-stored by the client.

In some application scenarios, the client running on terminal 400 uses the duration scaling logic in the video effects file as the target playback duration, taking the playback duration of the video sent by server 200 as the target playback duration. While rendering the video, it also renders the video effects contained in the video effects file to form a video effects playback process that conforms to the target playback duration, thereby achieving the effect of synchronous display of video effects and video. The specific process is as follows: Terminal 400 receives the original video (weekly battle report video) and video effects file returned from server 200, obtains the original video duration (duration of weekly battle report video), uses the original video duration as the target playback duration, decodes the video effects file to perform the corresponding duration scaling processing, so that the duration-scaling processed effects are adapted to the playback duration of the original video, and finally renders the effects and displays them simultaneously with the original video to display the weekly battle report video with effects.

In some application scenarios, the server distributes multiple videos, and video effects serve as transition animations between these videos to allow for sequential playback. The duration of the transition animation can be specified when the server distributes the videos, for example, based on the user's account level (the higher the level, the shorter the transition time). When the client running on terminal 400 finishes playing a video, it renders the video effects contained in the video effects file to form a video effects playback process that conforms to the target playback duration. The video effects playback process actually realizes the function of transition animations, making the connection between multiple videos more natural. The specific process is as follows: Terminal 400 obtains the video effects file corresponding to a certain effect from server 200, decodes it, and performs corresponding duration scaling processing based on the specified target playback duration (the duration of the transition animation), and then renders the duration-scaled effects between several original videos.

The playback duration of videos sent to different users or to the same user varies. The same video effects file can be reused in the playback of many videos at the same time, reducing the computational resource consumption of the server in repeatedly generating videos and reducing the waiting latency on the user side.

In another application scenario, the client running on terminal 400 is a social networking client or a video sharing client, with video capture, editing, and sharing functions. The client captures video and uses effects files downloaded from the server for compositing, such as image stitching (both displayed simultaneously) or timeline stitching (i.e., using video effects files to connect multiple captured videos). For the former, the target playback duration is the video's length. The specific process is as follows: Terminal 400 receives the user-captured native video, obtains the native video's length, retrieves the corresponding video effects file from server 200, decodes it to use the native video's length as the target playback duration, performs corresponding duration scaling for the video effects to adapt them to the native video, performs image stitching between the effects and the native video, renders in real-time to preview the final editing effect, and after previewing, can encode the image stitching result to create a new video file to share with other users. For the latter, the target playback duration is set by the user or the client's default transition animation duration. After playing a video, the video effects contained in the video effects file are rendered to form a video effects playback process that conforms to the target playback duration. The video effects playback process actually realizes the function of transition animation to make the connection between multiple videos more natural. The specific process is as follows: Terminal 400 obtains the video effects file corresponding to a certain effect from server 200, decodes it, and performs corresponding duration scaling processing based on the specified target playback duration (transition animation duration). The duration-scaling processed effects are then rendered between several native videos. The duration-scaling processed effects are then time-stitched together with several native videos, and then encoded to obtain a new video file for sharing with other users.

In the video editing scenario described above, users can continue to adjust the target playback duration and re-render the original audio video and video effects until the final preview result meets the requirements. Then, the client (video editing client) combines the original video and video effects files to encode a complete video file, which can then be shared.

It should be noted that the functions used to implement special effects processing in the client mentioned above can be native to the client or implemented by the client through the implantation of corresponding plugins such as SDKs. There is no limitation on the specific form of video special effects processing implemented in the client.

In addition, as an alternative to client-side rendering, when the computing resources (processor and RAM) consumed by rendering exceed the terminal's capacity (for example), the client can request the server to render and present the video effects playback process based on the rendering data returned by the server.

The following description continues using the terminal described above as an example of the electronic device provided in this embodiment of the invention. Referring to Figure 2, Figure 2 is an optional structural schematic diagram of the terminal 400 provided in this embodiment of the invention. The terminal 400 shown in Figure 2 includes: at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the terminal 400 are coupled together through a bus system 440. It can be understood that the bus system 440 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 440 in Figure 2.

The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and/or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this embodiment is intended to include any suitable type of memory.

In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

The network communication module 452 is used to reach other computing devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

Presentation module 453 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with user interface 430;

The input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.

In some embodiments, the video effects processing apparatus provided in this invention can be implemented in software. Figure 2 shows a video effects processing apparatus 455 stored in a memory 450, which can be software in the form of programs and plugins, including the following software modules: file acquisition module 4551, duration acquisition module 4552, effects frame determination module 4553, and rendering module 4554. These modules are logically related, and therefore can be arbitrarily combined or further split according to the functions implemented.

The following will illustrate the exemplary application and implementation of the video effects processing method provided in the embodiments of the present invention on the terminal described above. It should be noted that the following description is from the perspective of the terminal. However, it is understood that, based on the specific application scenarios described above, the video effects processing described below can be performed by a client running on the terminal. Therefore, the terminal mentioned below can specifically refer to a client running on the terminal. Examples of clients have already been described above and will not be repeated.

Referring to Figure 3A, which is an optional flowchart of the video effects processing method provided in the embodiment of the present invention, the steps shown in Figure 3A will be described in conjunction with the steps shown in Figure 3A.

In step 101, the terminal obtains the video effects file and extracts the duration scaling logic from the video effects file.

As an example, the main way to obtain video effects files is through plugins to export them. The video effects files can be PAG format sticker animation files. In order to read the animation characteristic data in the project file, you can choose one of the following methods to export PAG binary files: vector export, bitmap sequence frame export, or video sequence frame export, depending on the specific needs. The client or server decodes the exported PAG binary files. Here, we take the terminal as an example. That is, the terminal decodes the exported PAG binary files and then renders them through the rendering module. The decoding and rendering process on the terminal can be achieved by calling the rendering SDK. The purpose of decoding is to deserialize the PAG binary files into data objects that the client can operate on. The decoded data structure can refer to the PAG data structure.

In some embodiments, obtaining the video effects file in step 101 can be achieved through the following technical solutions: performing one of the following processes to obtain the encoded export file of the corresponding effects object: encoding multiple layer structures of the effects object to obtain the encoded export file of the corresponding effects object; encoding multiple effects frames of the effects object to obtain the encoded export file of the corresponding effects object; performing video format compression processing on multiple effects frames of the effects object, and encoding the obtained video format compression processing result to obtain the encoded export file of the corresponding effects object; encapsulating the duration extension type and duration extension interval in the encoded export file to obtain the video effects file of the corresponding effects object.

As an example, the encoded export file of the special effects object is first obtained through one of the following three methods: vector export, bitmap sequence frame export, and video sequence frame export. Vector export supports most After Effects features and produces a very small export file. It is typically used in user interfaces or scenes where content is editable. Vector export restores the layer structure of After Effects animation. The vector export method restores the layer structure of After Effects animation through the SDK provided by After Effects and uses dynamic bit storage technology during the export process, which greatly reduces the file size. Bitmap sequence frame export and video sequence frame export can support all After Effects features, but the export file is larger. They are typically used in video compositing or scenes with special requirements for animation effects. Bitmap sequence frame export produces bitmap data. Bitmap sequence frame export converts each frame of the complex animation effect designed by the designer into an image format for storage. More specifically, considering that most After Effects animations are continuous and have small differences between frames, a certain frame is selected as a keyframe. The data of each subsequent frame is compared with it to obtain the position information, width and height data of the difference bitmap, and the difference bitmap information is extracted and stored, thereby reducing the file size. Meanwhile, bitmap sequence frames support exporting multiple versions (with different scaling factors, frame rates, and sizes) to meet the needs of different scenarios. The advantage of this processing method is that it can support all After Effects features, but the disadvantage is that the exported file will be larger and it cannot perform image replacement or text editing operations in After Effects animations. It is suitable for handling complex effects such as masks and shadows and is mainly used in web applications. The video sequence frame export method uses the H.264 compression format in the video field. Compared with bitmap sequence frames, the decoding speed is faster and it is mainly used in mobile applications. The video sequence frame export method compresses the captured images into video format. The video sequence frame export method is an optimization to address the problems of large file size and low decoding efficiency of bitmap sequence frame exported images. From a performance perspective, the vector export method can achieve a very optimized limit in terms of file size and performance, but designers need to spend extra effort to continuously optimize it. For PAG video effect files generated by the sequence frame export method, the rendering performance is increased by additional image decoding time, but the overall time is only related to the size of the sequence frame image.

As an example, encapsulating the user-input duration scaling type and duration scaling range in the encoded export file actually modifies the data structure of the PAG sticker animation file. The duration scaling type and duration scaling range can be added at the root path level of the file, ultimately resulting in the video effect file of the corresponding effect object. In the specific implementation process, the execution order of the encapsulation steps and the encoding steps is not limited. That is, the duration scaling type and duration scaling range can be added at the root path level first, and then the encoding export process of the above three methods can be performed, or the encoding export process can be performed first, and then the duration scaling type and duration scaling range can be added at the root path level.

Through the above embodiments, the development process of designers designing animations through After Effects and then providing animation data characteristics to terminal development engineers to realize the animation function is reduced to designers designing animations through After Effects and exporting PAG sticker animation files, and the terminal directly loading and displaying the PAG sticker animation files. This greatly reduces the workload of terminal development and is also compatible with the development needs of various platforms.

In some embodiments, the duration scaling logic includes duration scaling intervals and corresponding duration scaling types; the extraction of duration scaling logic from the video effects file in step 101 can be achieved through the following technical solution: decoding the video effects file to obtain at least one duration scaling interval and corresponding duration scaling type of the corresponding video effects file; wherein, the duration scaling type includes any one of the following types: time linear scaling type; time repeating type; time reverse repeating type.

As an example, the video effects file can be the aforementioned PAG sticker animation file, which is a description file for the effects object. After decoding, the duration extension interval and corresponding duration extension type based on user configuration can be extracted. The number of duration extension intervals is usually one, but in more complex application scenarios, there are multiple duration extension intervals. For each duration extension interval, there is a corresponding duration extension type configured by the user. For the user configuration function, a configuration entry can be provided to the user through the terminal to receive the duration extension interval and duration extension type input by the user. See Figure 6A. Figure 6A is a schematic diagram of the video effects processing method provided by the embodiment of the present invention. That is, the duration extension interval and duration extension type input by the user are received through the configuration entry 601 and 602 provided by the client in various application scenarios. After receiving the duration extension logic, a page jump can also occur to continue to receive the target playback duration corresponding to each duration extension interval, or after setting the duration extension logic. After processing, the received configuration information is sent to the AE client via inter-process communication. This information is then combined with the special effects data obtained from the AE client for encoding processing to obtain the final video effects file. After extracting the scaling logic from the video effects file, a configuration entry 603 for the target playback duration is provided. This implementation allows clients in any application scenario to flexibly set the scaling logic for animation effects files. The AE client can also annotate the duration scaling range and duration scaling type. See Figures 6B-6C, which are schematic diagrams illustrating the video effects processing method provided in this embodiment. Specifically, the duration scaling range and duration scaling type are directly received from the user through the AE client and combined with the special effects data obtained from the AE client for encoding processing to obtain the final video effects file. This implementation alleviates the development and optimization tasks for clients in various application scenarios, allowing each client to directly call the rendering SDK to obtain the video effects file and execute subsequent logic.

In step 102, the terminal obtains the target playback duration required by the video effects file, wherein the target playback duration is different from the original playback duration of the video effects file.

Based on Figure 3A, and referring to Figure 3B, Figure 3B is an optional flowchart of the video effects processing method provided in the embodiment of the present invention. In step 102, obtaining the target playback duration to be achieved by the video effects file can be achieved through steps 1021 to 1022, which will be explained in conjunction with each step.

In step 1021, when there are multiple time-scaling intervals, the video effects file is split into multiple video effects sub-files of the same number, and the target playback duration for each video effects sub-file is obtained.

As an example, the method of splitting the video effects file is not limited during implementation. It only needs to ensure that each sub-file contains one and only one duration scaling interval. The target playback duration for each duration scaling interval can be obtained by allocating the target playback duration of the video effects file. For example, if the target playback duration of the video effects file is 10 seconds, and the file contains two duration scaling intervals: a first duration scaling interval of 1 second to 2 seconds and a second duration scaling interval of 3 seconds to 4 seconds, then the first target playback duration for the first duration scaling interval only needs to ensure... The second target playback duration, combined with the second duration scaling interval, satisfies the target playback duration of the video effects file. That is, if the first duration scaling type is a repeating type, then the first target playback duration is at least greater than 2 seconds (0 seconds - 2 seconds), and the sum of the first target playback duration and the second target playback duration is 10 seconds. Since there are few restrictions, namely, only the sum of the target playback durations of each corresponding sub-file is limited to be the target playback duration of the video effects file, and the target playback durations of each sub-file satisfy the corresponding duration scaling interval, the trouble of user settings is avoided without the need for user intervention, thus providing diverse and random rendering effects.

As an example, the target playback duration of each video effects sub-file actually involves the allocation of the target playback duration of the video effects file. In the allocation process, in addition to arbitrary allocation under the constraints of the above-mentioned conditions, the user can also be provided with an allocation scheme configuration function. In the user configuration entry in Figure 6C, relevant entries can also be provided to facilitate the user to input the target playback duration configured for each duration expansion interval. For example, the user can set the target playback duration of the video effects sub-file corresponding to each duration expansion interval. This implementation method is conducive to the user flexibly controlling the rendering effect of each sub-file in a more granular way, thereby controlling the rendering effect of the entire file.

As an example, when there are multiple duration scaling intervals and there are native videos that need to be adapted to the special effects, the duration of the native video can be directly used as the target playback duration of the video special effects file. The duration of the native video is allocated for different scaling intervals so that the target playback duration of each scaling interval meets the above constraints. If the allocated target playback duration does not meet the duration scaling type in the duration scaling logic of the corresponding special effects sub-file, then other video special effects files are selected. In the material library, there can be video special effects files with the same special effects object corresponding to different duration scaling logics.

In step 1022, when the number of time-scaling intervals is one, the overall target playback duration for the video effects file is obtained.

As an example, when there is only one duration scaling interval, there is no need to allocate the target playback duration of the video effects file. The target playback duration configured by the user is directly used as the overall target playback duration of the video effects file. When the target playback duration entered by the user does not conform to the duration scaling type of the duration scaling interval, an error message will be returned to the user, and the entry will be reopened to receive the target playback duration entered by the user.

As an example, when there is only one duration scaling interval and there is a native video that needs to be adapted to the special effect, the duration of the native video can be directly used as the target playback duration of the special effect. If the duration of the native video does not meet the duration scaling type in the duration scaling logic of the corresponding special effect file, then other video special effect files are selected. In the material library, there can be video special effect files with the same special effect object but different duration scaling logic.

Since the requirements for target playback duration vary in different application scenarios, the same video effects file can be reused in the playback and editing processes of many videos simultaneously. This can reduce the computational resource consumption of the server in repeatedly generating videos and reduce the waiting latency on the user side.

In some embodiments, after obtaining the target playback duration required by the video effects file in step 102, the following technical solutions can also be implemented: When there are multiple duration scaling intervals, the following processing is performed for each video effects sub-file: the original timeline of the corresponding effect object is obtained from the timeline of the video effects sub-file, that is, the part of the timeline in which the effect object appears in the timeline of the sub-file; the frame rate of the original timeline is kept constant, and the original timeline is subjected to duration scaling processing to obtain the target timeline of the corresponding target playback duration; when there is only one duration scaling interval, the following processing is performed for the video effects file: the original timeline of the corresponding effect object is obtained from the video effects file; the frame rate of the original timeline is kept constant, and the original timeline is subjected to duration scaling processing to obtain the target timeline of the corresponding target playback duration.

As an example, when there are multiple time-scaling intervals, the portion of the timeline containing the effect object in the video effect sub-file's timeline is used as the original timeline. The frame rate of the original timeline remains constant, and the time-scaling process is applied to this original timeline to obtain the target timeline for the corresponding playback duration. When there is only one time-scaling interval, the original timeline of the corresponding effect object in the video effect file is directly scaled to obtain the target timeline for the corresponding playback duration. In both of these scenarios, the frame rate remains constant during the scaling process, ensuring that the minimum time unit remains unchanged. Therefore, the effect of the effect's time-scaling process is a change in playback progress, not a change in the playback frame rate.

In step 103, the terminal determines the special effects frame in the video effects file that corresponds to the target timeline according to the duration scaling logic; wherein the length of the target timeline is consistent with the target playback duration.

Based on Figure 3A, and referring to Figure 3C, Figure 3C is an optional flowchart of the video effects processing method provided in the embodiment of the present invention. When there are multiple time-scaling intervals, in step 103, according to the time-scaling logic, determining the effect frame corresponding to the target time axis in the video effect file can be achieved by executing steps 1031A to 1032A for each time-scaling interval. The following will be explained in conjunction with each step.

In step 1031A, multiple effect frames including the effect object and the timestamp corresponding to each effect frame on the original timeline are obtained from the video effect sub-file, so as to serve as the original effect frame timestamp for each effect frame.

As an example, when no duration scaling is required, the rendering logic relies on the timestamps of each effect frame in the video effect sub-file on the original timeline. For example, there are effect frames 1 to 24, with a frame rate of 24 frames per second, meaning each 1/24th of a second is a timestamp, such as 0, 1/24, 2/24, ..., 23/24. Effect frames 1 to 24 are presented at these 24 timestamps. The timestamps 0, 1/24, 2/24, ..., 23/24 are the original effect frame timestamps of effect frames 1 to 24.

In step 1032A, based on the duration scaling interval and the original effect frame timestamp corresponding to each effect frame, the effect frame corresponding to each timestamp on the target timeline is determined among multiple effect frames.

As an example, the process of duration scaling is actually to determine the effect frame corresponding to each timestamp on the target timeline. Assuming the original timeline is 1 second and the frame rate is 24 frames per second, that is, each 1/24 second is a timestamp, and the target timeline is 2 seconds, the frame rate is still 24 frames per second, then each timestamp on the target timeline is 0, 1/24, 2/24, ..., 23/24, 24/24, ..., 47/24. When scaling, the mapping relationship and mapping range between the timestamps on the target timeline and the timestamps on the original timeline can be determined by the duration scaling interval and the corresponding duration scaling type. Based on the mapping relationship and mapping range, the timestamp on the original timeline corresponding to each timestamp on the target timeline is determined. Some timestamps on the original timeline are original effect frame timestamps, and some timestamps do not present effect frames, such as timestamp 1/48. For timestamps that do not present effect frames, the nearest principle is adopted to determine the effect frame that needs to be presented.

Based on Figure 3A, and referring to Figure 3D, Figure 3D is an optional flowchart of the video effects processing method provided in the embodiment of the present invention. When the number of time-scaling intervals is one, in step 103, the determination of the effect frame corresponding to the target time axis in the video effect file according to the time-scaling logic can be achieved through steps 1031B to 1032B, which will be explained in conjunction with each step.

In step 1031B, multiple effect frames including the effect object and the timestamp corresponding to each effect frame on the original timeline are obtained from the video effect file, so as to serve as the original effect frame timestamp for each effect frame.

As an example, when no duration scaling is required, the rendering logic relies on the timestamps corresponding to each effect frame in the video effects file on the original timeline. For example, there are effect frames 1 to 24, with a frame rate of 24 frames per second, meaning each 1/24th of a second is a timestamp, such as 0, 1/24, 2/24, ..., 23/24. Effect frames 1 to 24 are presented at these 24 timestamps. The timestamps 0, 1/24, 2/24, ..., 23/24 are the original effect frame timestamps for effect frames 1 to 24.

In step 1032B, based on the duration scaling interval and the original effect frame timestamp corresponding to each effect frame, the effect frame corresponding to each timestamp on the target timeline is determined among multiple effect frames.

As an example, the process of duration scaling is actually to determine the effect frame corresponding to each timestamp on the target timeline. Assuming the original timeline is 1 second and the frame rate is 24 frames per second, that is, each 1/24 second is a timestamp, and the target timeline is 2 seconds, the frame rate is still 24 frames per second, then each timestamp on the target timeline is 0, 1/24, 2/24, ..., 23/24, 24/24, ..., 47/24. When scaling, the mapping relationship and mapping range between the timestamps on the target timeline and the timestamps on the original timeline can be determined by the duration scaling interval and the corresponding duration scaling type. Based on the mapping relationship and mapping range, the timestamp on the original timeline corresponding to each timestamp on the target timeline is determined. Some timestamps on the original timeline are original effect frame timestamps, and some timestamps do not present effect frames, such as timestamp 1/48. For timestamps that do not present effect frames, the nearest principle is adopted to determine the effect frame that needs to be presented.

In some embodiments, the step 1032A or 1032B, which determines the effect frame corresponding to each time stamp on the target timeline among multiple effect frames based on the duration scaling interval and the original effect frame timestamp corresponding to each effect frame, can be implemented through the following technical solution: Each time stamp on the target timeline is sequentially used as the target time stamp to perform the following processing: Based on the duration scaling interval, the time stamp corresponding to the target time stamp on the original timeline is determined; when the time stamp corresponding to the target time stamp on the original timeline overlaps with any original effect frame timestamp, the effect frame corresponding to the overlapping original effect frame timestamp is determined as the effect frame corresponding to the target time stamp; when the time stamp corresponding to the target time stamp on the original timeline does not overlap with any original effect frame timestamp, the original time stamp with the smallest distance from the time stamp corresponding to the target time stamp on the original timeline is determined, and the effect frame corresponding to the determined original effect frame timestamp is determined as the effect frame corresponding to the target time stamp.

As an example, suppose the original timeline is 1 second and the frame rate is 24 frames per second, meaning each 1/24th of a second is a timestamp. There are effect frames 1 to 24, with a frame rate of 24 frames per second, also with timestamps of 1/24th of a second, such as 0, 1/24, 2/24, ..., 23/24. Effect frames 1 to 24 are presented at these 24 timestamps. These timestamps 0, 1/24, 2/24, ..., 23/24 are the original effect frame timestamps for effect frames 1 to 24. The target timeline is 2 seconds, and the frame rate remains 24 frames per second. Therefore, each time on the target timeline... The timestamps are 0, 1/24, 2/24, ..., 23/24, 24/24, ..., 47/24. Each timestamp on the target timeline is taken as the target timestamp, and the corresponding original timestamps are determined separately. The corresponding original timestamps are 0, 0, 1/24, 1/24, 2/24, 2/24, ..., 23/24, 23/24. Since the timestamps corresponding to the target timestamps on the original timeline overlap with the timestamps of the original effect frames, the effect frames corresponding to these original effect frame timestamps are determined as the effect frames on each target timestamp. Each timestamp on the target timeline is 0, 1/24, 2/24, ..., 23/24, 24/24, ..., 47/24, corresponding to effect frames 1, 24, ..., 24 respectively. However, this transformation is an ideal scenario. In some cases, the original timestamp on the timeline corresponding to each target timestamp on the target timeline may not be the original effect frame timestamp. For example, suppose the target timestamp 1/24 corresponds to the original timestamp 1/48, but at a frame rate of 24 frames per second, the original timestamp... Since there is no corresponding effect frame for timestamp 1/48 on the timeline, the nearest effect frame A is chosen as the effect frame for timestamp 1/48, and thus determined as the effect frame corresponding to the target timestamp 1/24. Because there are two closest timestamps, effect frame A can be the effect frame at timestamp 0 or the effect frame at timestamp 1/24. If the target timestamp 1/24 corresponds to timestamp 1/36 on the original timeline, then the closest timestamp is timestamp 1/48, and effect frame A is the effect frame for timestamp 1/24.

In some embodiments, the determination of the target timestamp on the original timeline based on the duration scaling interval can be achieved through the following technical solution: For each duration scaling interval, the following processing is performed: when the target timestamp is not greater than the start timestamp of the duration scaling interval, the target timestamp is determined as the corresponding original timestamp on the original timeline; when the target timestamp is greater than the start timestamp of the duration scaling interval and less than the end timestamp of the duration scaling interval, the target timestamp is mapped based on the duration scaling type to obtain the corresponding original timestamp; when the target timestamp is greater than or equal to the end timestamp and less than the target playback duration, a first difference between the original playback duration and the target playback duration is determined, the first difference is summed with the target timestamp, and the summation result is used to determine the timestamp corresponding to the target timestamp on the original timeline.

As an example, since the duration scaling interval in the duration scaling logic has already defined the interval that needs to be scaled, for different timestamps on the target timeline, the corresponding timestamps on the original timeline will be mapped according to different mapping relationships. See Figure 7. The length of the original timeline is m, the length of the target timeline is n, the duration scaling interval is from a to b, and the length of the duration scaling interval is b-a. After duration scaling, the starting timestamp of the duration scaling interval is a, and the ending timestamp is n-(m-b). If the target timestamp t is between 0 and a, then the corresponding timestamp on the original timeline is also t, because this time period belongs to the fixed interval of the opening sequence. If the target timestamp t is between n-(m-b) and n, then the corresponding timestamp on the original timeline is m-n+t. If the target timestamp t is between a and n-m+b, then mapping processing is required according to different duration scaling types.

In some embodiments, the above-mentioned mapping of the target timestamp based on the duration scaling type to obtain the corresponding original timestamp can be achieved through the following technical solution: when the duration scaling type is a linear time reduction type or a linear time stretching type, the second difference between the target playback duration and the original playback duration is determined as the scaling length, and the scaling length is summed with the length of the duration scaling interval; the length of the duration scaling interval is compared with the summation result to obtain the scaling coefficient; the third difference between the target timestamp and the starting timestamp is determined, and the third difference is multiplied with the scaling coefficient; the multiplication result is summed with the starting timestamp to obtain the corresponding original timestamp.

As an example, when the target timestamp t is between a and n-m+b, mapping processing is required according to different duration scaling types. If the duration scaling type is linear time scaling, the second difference between the target playback duration n and the original playback duration m is determined as the scaling length, and the scaling length is summed with the length of the duration scaling interval b-a. The scaling coefficient k is obtained by comparing the length of the duration scaling interval b-a with the summation result. The third difference between the target timestamp t and the starting timestamp a is determined, and the third difference is multiplied with the scaling coefficient k. The multiplication result is summed with the starting timestamp a to obtain the corresponding original timestamp. The specific calculation principle can be found in the following formula: f(t)＝a+k(t-a), a＜t＜n-m+b.

In some embodiments, the above-mentioned mapping of the target timestamp based on the duration scaling type to obtain the corresponding original timestamp can be achieved by the following technical solution: when the duration scaling type is the time repetition type, the fourth difference between the target timestamp and the starting timestamp is determined, and the remainder of the fourth difference and the length of the duration scaling interval is processed; the remainder processing result is summed with the starting timestamp to obtain the corresponding original timestamp.

As an example, when the target timestamp t is between a and n-m+b, mapping processing is required according to different duration scaling types. If the duration scaling type is time repetition, the fourth difference between the target timestamp t and the starting timestamp a is determined, and the remainder of the fourth difference and the length of the duration scaling interval (b-a) is calculated. The remainder result is then summed with the starting timestamp a to obtain the corresponding original timestamp. The specific calculation principle can be found in the following formula: f(t)＝a+(t-a)％(b-a), a＜t＜n-m+b.

In some embodiments, the above-described mapping of the target timestamp based on the duration scaling type to obtain the corresponding original timestamp can be achieved through the following technical solution: when the duration scaling type is a time-reverse repeating type, the fifth difference between the target timestamp and the starting timestamp is determined, and the remainder of the fifth difference and the length of the duration scaling interval is calculated; the ratio of the remainder result is rounded to obtain the rounded result; when the rounded result is even, the remainder result is summed with the starting timestamp to obtain the corresponding original timestamp; when the rounded result is odd, the sixth difference between the length of the duration scaling interval and the remainder result is determined, and the sixth difference is summed with the starting timestamp to obtain the corresponding original timestamp.

As an example, when the target timestamp t is between a and n-m+b, mapping processing is required according to different duration scaling types. If the duration scaling type is a time-reverse repetition type, the fifth difference between the target timestamp t and the starting timestamp a is determined, and the remainder of the fifth difference and the length of the duration scaling interval (b-a) is calculated. For example, the remainder of 8 and 3 is 2, and the corresponding ratio is 8/3. After rounding, the rounded result is 2. The corresponding remainder processing results are then used to calculate the remainder. The ratio of the results is rounded to obtain the rounded result. When the rounded result is even, the remainder result is summed with the starting timestamp a to obtain the corresponding original timestamp. When the rounded result is odd, the sixth difference between the length of the duration expansion interval (b-a) and the remainder result is determined. The sixth difference is summed with the starting timestamp a to obtain the corresponding original timestamp. The specific calculation principle can be found in the following formula: f(t)＝a+(t-a)％(b-a), a＜t＜n-m+b.

In step 104, the terminal renders the special effects frames corresponding to the target timeline to form a video special effects playback process that conforms to the target playback duration.

As an example, the terminal performs rendering processing on the corresponding timestamp based on the effect frames of each timestamp on the target timeline, thereby obtaining a video effect playback process that conforms to the target playback duration.

As an example, in a short video editing scenario, the terminal receives a native video shot by the user or returned from the server, obtains the native video duration, decodes the video effect file corresponding to a certain effect in the media library to perform corresponding duration scaling, uses the native video duration as the target playback duration to make the effect fit the native video, and then splices the effect with the native video for real-time rendering as a preview of the final effect. After previewing, the splicing result is encoded to form a new video file to be shared with other users.

As an example, in the scenario of short video editing, the special effects in the video effects file can also serve as a connecting animation between several native videos. The terminal decodes the video effects file corresponding to a certain effect in the material library and receives the setting operation for the target playback duration to perform the corresponding duration scaling processing, so that the effect after duration scaling processing is placed between several native videos. After the effect and the native videos are spliced together on the timeline, they can be rendered in real time as a preview of the final effect. After previewing, the splicing result is encoded to form a new video file to be shared with other users.

As an example, in scenarios that do not involve file sharing, such as game battle reports, the terminal receives the original video and special effects video files returned from the server, obtains the original video duration, decodes the video special effects file to perform corresponding duration scaling, uses the original video duration as the target playback duration, so that the special effects are adapted to the playback duration of the original video, and renders and displays the special effects and the original video simultaneously.

Referring to Figure 3E, which is an optional flowchart of the video effects processing method provided in this embodiment of the invention, the above embodiment executes the specific processing flow only through the terminal. Alternatively, the processing flow can be implemented by combining a terminal and a server. In step 201, the terminal sends a rendering request to the server. In step 202, the server obtains the video effects file and extracts the duration scaling logic from the video effects file. In step 203, the server returns the extracted duration scaling logic to the terminal. In step 204, the terminal receives the input target playback duration. In step 205, the terminal sends the target playback duration to the server. In step 206, the server determines the desired playback duration based on the duration scaling logic. The server defines the effect frames in the video effect file that correspond to the target timeline. In step 207, the server renders the effect frames corresponding to the target timeline to form a video effect playback process that conforms to the target playback duration. In step 208, the server returns the video effect to the terminal. In step 209, the terminal presents the video effect. The above process involves the interaction between the terminal and the server. The rendering processing, which requires a lot of computing resources, is assigned to the server. The terminal is only responsible for receiving the user's configuration requirements and presenting the rendered video effect. In other embodiments, the logic completed by the server can also be completed by calling the rendering SDK or by remotely calling cloud server resources through the rendering SDK.

This invention provides a method for processing video effects, which supports time scaling of fixed animation files. External application platforms only need to set the target playback time of the animation file, and the animation file can then scale according to the user-configured scaling logic. The playback duration scaling of the video effect file is controlled by the duration scaling logic within the video effect file. After decoding the video effect file, processing and rendering according to the duration scaling logic achieves the desired playback duration of the video effect. This method can be directly applied to various applications and platforms, and is not limited by the platform's operating system; the implementation process is extremely simple.

The following will describe an exemplary application of the video effects processing method provided in this embodiment of the invention in a real-world application scenario.

The video effects processing method provided in this embodiment of the invention has wide applications in game weekly battle report videos. Referring to Figure 4, Figure 4 is a schematic diagram illustrating the application effect of the video effects processing method provided in this embodiment of the invention in a game weekly battle report video scenario. The terminal needs to present the horizontal video (native video) in the middle area and the sticker animations (effect objects corresponding to the video effects file) in the upper and lower areas. The sticker animations in the upper and lower areas are achieved through PAG sticker animations. This is a vertical animation, presented in the upper and lower edge areas of the video, accompanied by periodic animation effects. In other application scenarios, there are also cases where the beginning and end of the sticker animation are fixed, and the content in the middle is stretched according to requirements. The requirement can be a target playback duration. The duration of the horizontal video (the duration of the original video) can be used. The application effect diagram in Figure 4 can be implemented through the following steps: The client triggers the logic of requesting the server to send the weekly battle report video, so that the server sends the weekly battle report video to the client. The client responds by receiving the weekly battle report video (original video) and the corresponding video effect file returned from the server 200. It obtains the duration of the weekly battle report video as the target playback duration, and decodes the video effect file based on the target playback duration to perform the corresponding duration scaling processing, so that the duration scaling processing effect object is adapted to the duration of the weekly battle report video. Finally, the effect object is rendered and displayed simultaneously with the weekly battle report video. The displayed effect is the weekly battle report video with the effect object.

Referring to Figure 5, which is a flowchart of the video effects processing system provided in this embodiment of the invention, firstly, the scaling range and scaling type of the sticker animation are set in AE through a plugin. Referring to Figure 6B, which is a schematic diagram of the video effects processing method provided in this embodiment of the invention, annotations are added to the overall composition of the sticker animation. The plugin in AE can support adding annotations to layers. (1) By clicking on the blank area, no layer is selected. (2) In response to clicking on the layer control, the layer menu is presented. (3) In response to adding a mark, the mark addition process is realized. Thus, in combination with the specific usage scenario, relevant labels can be added, such as the duration scaling range and duration scaling type, so that subsequent rendering processing can be performed according to the duration scaling range and duration scaling type by the rendering SDK. Referring to Figure 6C, which is a schematic diagram of the video effects processing method provided in this embodiment of the invention, the duration scaling range and four duration scaling types can be set. The specific setting process is as follows: (1) In response to the double-clicking of the annotation, the setting page is presented. (2) The duration scaling type is received in the setting page. (i.e., the content filled in), (3) receive the modified start time in the settings page, (4) receive the confirmation and save operation in the settings page. The duration scaling types include the following: 1. No scaling type: indicates that no duration scaling is required; 2. Linear scaling type: when the target playback duration of the entire PAG sticker animation is set to be longer than the original playback duration of the original PAG sticker animation, linear scaling is performed in the duration scaling range; when the target playback duration of the entire PAG sticker animation is set to be shorter than the original playback duration of the original PAG sticker animation, linear scaling is performed in the duration scaling range. 3. Repeat type: When the time stretching type is repeat type, if the target playback duration of the entire PAG sticker animation is set to be longer than the original playback duration of the original PAG sticker animation, the time stretching will be performed periodically within the duration stretching range; 4. Repeat reverse type: When the target playback duration of the entire PAG sticker animation is set to be longer than the original playback duration of the original PAG sticker animation, the time stretching will be performed periodically in reverse order within the duration stretching range, that is, first play in forward order, then play in reverse order, then play in forward order again, then play in reverse order again, and so on.

The PAG sticker animation file is obtained by encoding the special effects object. After the duration scaling range and type are successfully set, the PAG sticker animation file can be exported. By modifying the data structure of the PAG sticker animation file, the duration scaling type and duration scaling range are added at the level of the file root path to facilitate encoding. When used on the platform side, the decoding module of the rendering SDK needs to decode and read the corresponding data. The rendering SDK can be a client SDK or a server SDK. The client SDK completes the rendering on the client side, and the server SDK completes the rendering on the server side.

To add duration scaling support to PAG sticker animations, and because the rendering logic of animations designed based on After Effects is quite complex, including effects such as trajectory calculation and time easing, it would be too complex to implement the animation duration scaling function by modifying the animation characteristics of specific layers in the PAG sticker animation. Therefore, it is not advisable to modify the specific animation characteristics of the layers in the PAG sticker animation. Instead, the rendering side can encapsulate the original rendering time calculation logic and implement the duration scaling function by changing the rendering progress of the original animation file, calculating the specific rendering progress of the duration scaling interval.

Referring to Figure 7, Figure 7 is a timeline diagram of the video effects processing method provided in the embodiment of the present invention. The minimum time unit of the original timeline and the target timeline is a frame. If the frame rate is 24 frames per second, it means that 24 frames are presented in one second. The minimum time unit is 1/24 second. The original playback duration of the PAG sticker animation file is m, which includes a duration scaling interval (a, b). If the duration scaling type is no scaling type, the rendering logic is consistent with the previous logic and no duration scaling processing is performed. If the duration scaling type is one of the other types, and the target playback duration after scaling is n, the specific rendering progress calculation process is as follows: First, calculate the time scaling coefficient k, k = (b-a)/(n-m+b-a); t is the rendering time point after scaling, that is, the target timestamp on the target timeline. f(t) is the original timestamp of the PAG sticker animation on the original timeline for actual rendering. When the time scaling type is linear scaling type, the original timestamp of the PAG sticker animation on the original timeline for actual rendering is calculated according to the following formula (1):

When the time scaling type is repeating, the original timestamp for actual rendering on the original timeline of the PAG sticker animation is calculated according to the following formula (2):

When the time scaling type is repeating and reversing, and when a < t < n-m+b, the original timestamp for actual rendering on the original timeline of the PAG sticker animation is calculated in two cases: when the result of (t-a)/(b-a) is an even number, f(t) = a + (t-a)%(b-a); when the result of (t-a)/(b-a) is an odd number, f(t) = b - (t-a)%(b-a). When t is in other ranges, the calculation is the same as above. When there are multiple time scaling intervals in the PAG sticker animation, the calculation method is similar, and calculations need to be performed for multiple time scaling intervals. After f(t) is calculated using the above formula, the rendering module in the rendering SDK can render the animation screen corresponding to the final effect according to the corresponding original timestamp. Finally, the animation screen of the effect object is displayed simultaneously with the weekly battle report video, and the displayed effect is the weekly battle report video with the effect object.

This invention provides a method for processing video special effects, which can effectively solve the contradiction between the variable duration requirements of special effects animations in user interface animations (e.g., video editing) and server-side special effects video rendering scenarios and the fixed duration of sticker animation files designed by designers. When designing sticker animation effects, once the designer sets the duration stretching range and duration stretching type, any platform only needs to set the target playback duration of the sticker animation to achieve the time stretching effect of the animation.

The following description continues to illustrate the exemplary structure of the video effects processing device 455 provided in the embodiments of the present invention as a software module. In some embodiments, as shown in FIG2, the software module stored in the video effects processing device 455 in the memory 450 may include: a file acquisition module 4551, used to acquire a video effects file and extract duration scaling logic from the video effects file; a duration acquisition module 4552, used to acquire the target playback duration to be achieved by the video effects file, wherein the target playback duration is different from the original playback duration of the video effects file; an effects frame determination module 4553, used to determine the effects frame in the video effects file corresponding to the target timeline according to the duration scaling logic; wherein the length of the target timeline is consistent with the target playback duration; and a rendering module 4554, used to render according to the effects frame corresponding to the target timeline to form a video effects playback process that conforms to the target playback duration.

In some embodiments, the file acquisition module 4551 is further configured to: perform one of the following processes to obtain an encoded export file of the corresponding special effects object: encode multiple layer structures of the special effects object to obtain an encoded export file of the corresponding special effects object; encode multiple special effects frames of the special effects object to obtain an encoded export file of the corresponding special effects object; perform video format compression processing on multiple special effects frames of the special effects object, and encode the obtained video format compression processing result to obtain an encoded export file of the corresponding special effects object; encapsulate the duration extension type and duration extension interval in the encoded export file to obtain a video special effects file of the corresponding special effects object.

In some embodiments, the duration scaling logic includes duration scaling intervals and corresponding duration scaling types; the file acquisition module 4551 is further configured to: decode the video effects file to obtain at least one duration scaling interval and corresponding duration scaling type of the corresponding video effects file; wherein, the duration scaling type includes any one of the following types: time linear scaling type; time repeating type; time reverse repeating type.

In some embodiments, the duration acquisition module 4552 is further configured to: when the number of duration scaling intervals is multiple, split the video effect file into multiple video effect sub-files of the same number, and acquire the target playback duration for each video effect sub-file respectively; when the number of duration scaling intervals is one, acquire the overall target playback duration for the video effect file.

In some embodiments, the duration acquisition module 4552 is further configured to: after acquiring the target playback duration to be achieved by the video effects file, when the number of duration scaling intervals is multiple, perform the following processing for each video effects sub-file: acquire the original timeline of the corresponding effects object from the video effects sub-file; keep the frame rate of the original timeline unchanged, and perform duration scaling processing on the original timeline to obtain the target timeline of the corresponding target playback duration; when the number of duration scaling intervals is one, perform the following processing for the video effects file: acquire the original timeline of the corresponding effects object from the video effects file; keep the frame rate of the original timeline unchanged, and perform duration scaling processing on the original timeline to obtain the target timeline of the corresponding target playback duration.

In some embodiments, when the number of duration scaling intervals is multiple, the effect frame determination module 4553 is further configured to: perform the following processing for each duration scaling interval: obtain multiple effect frames including effect objects and the timestamp corresponding to each effect frame on the original timeline from the video effect sub-file, as the original effect frame timestamp of each effect frame; and determine the effect frame corresponding to each timestamp on the target timeline among the multiple effect frames based on the duration scaling interval and the original effect frame timestamp corresponding to each effect frame.

In some embodiments, when the number of duration scaling intervals is one, the special effects frame determination module 4553 is further configured to: obtain from the video special effects file a plurality of special effects frames including special effects objects, and the timestamp corresponding to each special effects frame on the original timeline, as the original special effects frame timestamp of each special effects frame; and determine the special effects frame corresponding to each timestamp on the target timeline among the plurality of special effects frames based on the duration scaling intervals and the original special effects frame timestamps corresponding to each special effects frame.

In some embodiments, the special effects frame determination module 4553 is further configured to: sequentially use each timestamp on the target timeline as the target timestamp to perform the following processing: based on the duration scaling interval, determine the timestamp corresponding to the target timestamp on the original timeline; when the timestamp corresponding to the target timestamp on the original timeline overlaps with any original special effects frame timestamp, determine the special effects frame corresponding to the overlapping original special effects frame timestamp as the special effects frame corresponding to the target timestamp; when the timestamp corresponding to the target timestamp on the original timeline does not overlap with any original special effects frame timestamp, determine the original timestamp with the smallest distance from the timestamp corresponding to the target timestamp on the original timeline, and determine the special effects frame corresponding to the determined original special effects frame timestamp as the special effects frame corresponding to the target timestamp.

In some embodiments, the special effects frame determination module 4553 is further configured to: perform the following processing for each duration scaling interval: when the target timestamp is not greater than the start timestamp of the duration scaling interval, determine the target timestamp as the corresponding original timestamp on the original timeline; when the target timestamp is greater than the start timestamp of the duration scaling interval and less than the end timestamp of the duration scaling interval, perform a mapping based on the duration scaling type on the target timestamp to obtain the corresponding original timestamp; when the target timestamp is greater than or equal to the end timestamp and less than the target playback duration, determine the first difference between the original playback duration and the target playback duration, sum the first difference with the target timestamp, and determine the timestamp corresponding to the target timestamp on the original timeline using the summation result.

In some embodiments, the special effects frame determination module 4553 is further configured to: when the duration scaling type is a time linear reduction type or a time linear stretch type, determine the second difference between the target playback duration and the original playback duration as the scaling length, and sum the scaling length with the length of the duration scaling interval; perform ratio processing on the length of the duration scaling interval and the summation processing result to obtain the scaling coefficient; determine the third difference between the target timestamp and the starting timestamp, and multiply the third difference with the scaling coefficient; sum the multiplication processing result with the starting timestamp to obtain the corresponding original timestamp.

In some embodiments, the special effects frame determination module 4553 is further configured to: when the duration extension type is a time repetition type, determine the fourth difference between the target timestamp and the starting timestamp, perform a remainder operation on the fourth difference and the length of the duration extension interval; and sum the remainder operation result with the starting timestamp to obtain the corresponding original timestamp.

In some embodiments, the special effects frame determination module 4553 is further configured to: when the duration extension type is a time-reverse repetition type, determine the fifth difference between the target timestamp and the starting timestamp, and perform a remainder operation on the fifth difference and the length of the duration extension interval; perform a rounding operation on the ratio of the corresponding remainder operation result to obtain a rounded result; when the rounded result is even, sum the remainder operation result with the starting timestamp to obtain the corresponding original timestamp; when the rounded result is odd, determine the sixth difference between the length of the duration extension interval and the remainder operation result, and sum the sixth difference with the starting timestamp to obtain the corresponding original timestamp.

This invention provides a computer-readable storage medium storing executable instructions. When the executable instructions are executed by a processor, the processor will execute the electronic red envelope sending method provided in this invention, such as the video effects processing method shown in Figures 3A-3D.

In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hypertext Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

In summary, through the embodiments of the present invention, by encapsulating duration scaling logic in video effects files, a single video effects file can be freely scaled to the playback duration required by different application scenarios, thus possessing universal applicability. Furthermore, rendering is performed on this basis to form the video effects playback process, saving the enormous consumption of computational and time resources associated with creating numerous video effects files with varying playback durations.

The above are merely embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of the present invention are included within the scope of protection of the present invention.

Claims (15)

1. A method for processing video special effects, characterized in that the method includes: Obtain a video effects file and extract duration scaling logic from the video effects file, wherein the duration scaling logic includes duration scaling intervals and corresponding duration scaling types; Obtain the target playback duration that needs to be achieved when applying the video effects file to the design scene, wherein the target playback duration is different from the original playback duration of the video effects file; Based on the duration scaling logic, determine the special effects frames in the video special effects file that correspond to the target timeline; The length of the target timeline is the same as the target playback duration; Rendering is performed based on the effect frames corresponding to the target timeline to form a video effect playback process that conforms to the target playback duration. 2. The method according to claim 1, wherein obtaining the video effects file includes: Perform one of the following processes to obtain the coded export file for the corresponding effect object: The multiple layer structures of the special effects object are encoded to obtain the corresponding encoded export file of the special effects object; Encode multiple effect frames of the effect object to obtain the corresponding encoded export file of the effect object; Multiple effect frames of the effect object are subjected to video format compression processing, and the resulting video format compression processing result is encoded to obtain the encoded export file corresponding to the effect object; The duration scaling type and duration scaling range are encapsulated in the encoding export file to obtain the video effects file corresponding to the effects object. 3. The method according to claim 1, characterized in that, The logic for extracting duration scaling from the video effects file includes: The video effects file is decoded to obtain at least one duration scaling interval and the corresponding duration scaling type for the video effects file. The duration scaling type includes any one of the following types: linear time scaling type; time repetition type; and time reverse repetition type. 4. The method according to claim 3, characterized in that, obtaining the target playback duration to be achieved when applying the video effects file to the design scene includes: When the number of duration expansion intervals is multiple, the video effect file is split into multiple video effect sub-files with the same number of intervals, and the target playback duration for each video effect sub-file is obtained. When the number of duration scaling intervals is one, the overall target playback duration for the video effects file is obtained. 5. The method according to claim 4, characterized in that, after obtaining the target playback duration to be achieved when applying the video effects file to the design scene, the method further includes: When there are multiple duration extension intervals, the following processing is performed for each video effects sub-file: Obtain the original timeline of the corresponding effect object from the video effects sub-file; Keeping the frame rate of the original timeline unchanged, the original timeline is subjected to duration scaling to obtain a target timeline corresponding to the target playback duration; When the number of duration scaling intervals is one, the following processing is performed on the video effects file: Obtain the original timeline of the corresponding effect object from the video effects file; Keeping the frame rate of the original timeline unchanged, the original timeline is subjected to duration scaling to obtain a target timeline corresponding to the target playback duration. 6. The method according to claim 5, characterized in that, When there are multiple duration scaling intervals, determining the effect frame corresponding to the target timeline in the video effect file according to the duration scaling logic includes: Perform the following processing for each duration scaling interval: Obtain multiple effect frames including the effect object from the video effect sub-file, and the timestamp corresponding to each effect frame on the original timeline, as the original effect frame timestamp for each effect frame; Based on the duration scaling interval and the original effect frame timestamp corresponding to each of the effect frames, the effect frame corresponding to each timestamp on the target timeline is determined from the plurality of effect frames. 7. The method according to claim 5, characterized in that, When the number of duration scaling intervals is one, determining the effect frame corresponding to the target timeline in the video effect file according to the duration scaling logic includes: Obtain multiple effect frames including the effect object from the video effect file, and the timestamp corresponding to each effect frame on the original timeline, as the original effect frame timestamp for each effect frame; Based on the duration scaling interval and the original effect frame timestamp corresponding to each of the effect frames, the effect frame corresponding to each timestamp on the target timeline is determined from the plurality of effect frames. 8. The method according to claim 6 or 7, characterized in that, determining the effect frame corresponding to each timestamp on the target timeline from the plurality of effect frames based on the duration scaling interval and the original effect frame timestamp corresponding to each effect frame includes: Each timestamp on the target timeline is sequentially used as the target timestamp to perform the following processing: Based on the duration scaling interval, determine the timestamp corresponding to the target timestamp on the original timeline; When the timestamp corresponding to the target timestamp on the original timeline overlaps with any of the original special effects frame timestamps, the special effects frame corresponding to the overlapping original special effects frame timestamp is determined as the special effects frame corresponding to the target timestamp. When the timestamp corresponding to the target timestamp on the original timeline does not overlap with any of the original effect frame timestamps, the original effect frame timestamp with the smallest distance from the timestamp corresponding to the target timestamp on the original timeline is determined, and the effect frame corresponding to the determined original effect frame timestamp is determined as the effect frame corresponding to the target timestamp. 9. The method according to claim 8, wherein determining the timestamp corresponding to the target timestamp on the original timeline based on the time-scaling interval comprises: For each of the aforementioned duration scaling intervals, the following processing is performed: When the target timestamp is not greater than the starting timestamp of the duration expansion interval, the target timestamp is determined as the timestamp corresponding to the original timeline; When the target timestamp is greater than the start timestamp of the duration expansion interval and less than the end timestamp of the duration expansion interval, the target timestamp is mapped based on the duration expansion type to obtain the timestamp corresponding to the target timestamp on the original time axis. When the target timestamp is greater than or equal to the termination timestamp and less than the target playback duration, a first difference between the original playback duration and the target playback duration is determined. The first difference is summed with the target timestamp, and the summation result is used to determine the timestamp corresponding to the target timestamp on the original timeline. 10. The method according to claim 9, wherein the step of mapping the target timestamp based on the duration scaling type to obtain the timestamp corresponding to the target timestamp on the original timeline comprises: When the duration scaling type is a linear time reduction type or a linear time stretch type, the second difference between the target playback duration and the original playback duration is determined as the scaling length, and the scaling length is summed with the length of the duration scaling interval. The length of the time-scaling interval is compared with the summation result to obtain the scaling factor; Determine a third difference between the target timestamp and the starting timestamp, and multiply the third difference by the scaling factor; The result of the multiplication process is summed with the starting timestamp to obtain the timestamp corresponding to the target timestamp on the original time axis. 11. The method according to claim 9, wherein the step of mapping the target timestamp based on the duration scaling type to obtain the timestamp corresponding to the target timestamp on the original timeline comprises: When the duration stretching type is the time repetition type, determine the fourth difference between the target timestamp and the starting timestamp, and perform a remainder operation on the fourth difference and the length of the duration stretching interval. The remainder result is summed with the starting timestamp to obtain the timestamp corresponding to the target timestamp on the original timeline. 12. The method according to claim 9, wherein the step of mapping the target timestamp based on the duration scaling type to obtain the timestamp corresponding to the target timestamp on the original timeline comprises: When the duration stretching type is the time reverse repeating type, determine the fifth difference between the target timestamp and the starting timestamp, and perform a remainder operation on the fifth difference and the length of the duration stretching interval. The ratio result of the corresponding remainder processing is rounded to obtain the rounded result; When the rounding result is even, the remainder processing result is summed with the starting timestamp to obtain the timestamp corresponding to the target timestamp on the original time axis; When the rounding result is odd, the sixth difference between the length of the duration expansion interval and the result of the remainder processing is determined, and the sixth difference is summed with the starting timestamp to obtain the timestamp corresponding to the target timestamp on the original time axis. 13. A video special effects processing apparatus, characterized in that the apparatus comprises: The file acquisition module is used to acquire video effects files and extract duration scaling logic from the video effects files, wherein the duration scaling logic includes duration scaling intervals and corresponding duration scaling types. The duration acquisition module is used to acquire the target playback duration that needs to be achieved when the video effects file is applied to the design scene, wherein the target playback duration is different from the original playback duration of the video effects file; The special effects frame determination module is used to determine the special effects frame in the video special effects file corresponding to the target timeline according to the duration scaling logic; The length of the target timeline is the same as the target playback duration; The rendering module is used to render special effects frames corresponding to the target timeline to form a video special effects playback process that conforms to the target playback duration. 14. An electronic device, characterized in that it comprises: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the video effects processing method according to any one of claims 1 to 12. 15. A computer-readable storage medium, characterized in that it stores executable instructions for implementing the video effects processing method according to any one of claims 1 to 12 when executed by a processor.