WO2022002082A1

WO2022002082A1 - Method and apparatus for displaying video special effect, and electronic device and storage medium

Info

Publication number: WO2022002082A1
Application number: PCT/CN2021/103299
Authority: WO
Inventors: 武珊珊; 肖逸
Original assignee: 北京达佳互联信息技术有限公司
Priority date: 2020-07-01
Filing date: 2021-06-29
Publication date: 2022-01-06
Also published as: CN111954055B; CN111954055A

Abstract

The present disclosure relates to a method and apparatus for displaying a video special effect, and an electronic device and a storage medium, which relate to the technical field of software applications. The method comprises: on the basis of a key point trajectory special effect opening instruction, acquiring the current frame image and a key point to be tracked in the current frame image; acquiring position information of said key point in the current frame image; generating auxiliary position information of said key point according to the position information of said key point in the current frame image and position information of said key point in the previous frame image; performing key point trajectory rendering on the current frame image according to the position information of said key point in the current frame and in each frame image before the current frame, and the auxiliary position information of said key point generated each time; and displaying the rendered current frame image. In the present disclosure, auxiliary position information is generated on the basis of a linear interpolation mode, so as to ensure that a key point trajectory is smooth and continuous, and trajectory rendering is directly performed on a key point in each frame image, thereby realizing fast rendering.

Description

Display method, device, electronic device and storage medium for video special effects

CROSS-REFERENCE TO RELATED APPLICATIONS

This disclosure claims the priority of a Chinese patent application with an application number of 202010624935.2 and an invention titled "Display method, device, electronic device and storage medium for video special effects" filed with the China Patent Office on July 1, 2020, the entire contents of which are obtained through References are incorporated in this disclosure.

technical field

The present disclosure relates to the technical field of video processing, and in particular, to a method, device, electronic device and storage medium for displaying video special effects.

Background technique

With the rapid development of mobile terminal technology, application software with various functions emerges as the times require, which can bring convenience and entertainment to users. Among them, the game special effects, especially the key point trajectory special effects, are very popular among users.

In the related art, the position of key points in each frame of image is usually recorded, saved in an array, and all points in the trajectory are rendered into the screen in each frame of image, thereby realizing the special effect of the key point trajectory. exhibit. However, the display method of the existing video special effects is not complete due to the incomplete recording of the key point trajectories, resulting in gaps in the displayed key point trajectories. At the same time, if there are many key points that need to be rendered in each frame of image, the rendering time of each frame is relatively short. It is too long, which makes it impossible to achieve fast rendering, which seriously affects the user experience. Therefore, how to quickly add smooth and continuous key point trajectory effects to videos has become an urgent problem to be solved.

SUMMARY OF THE INVENTION

The present disclosure provides a method, device, electronic device, and storage medium for displaying video special effects, so as to at least solve the problem in the related art that a smooth and continuous key point trajectory special effect cannot be quickly added to a video. The technical solutions of the present disclosure are as follows:

According to a first aspect of the embodiments of the present disclosure, there is provided a method for displaying video special effects, including: obtaining a current frame image and a key point to be tracked in the current frame image based on a key point track special effect opening instruction; obtaining the current frame image The position information of the key point to be tracked in the frame image; according to the position information of the key point to be tracked in the current frame image and the position information of the key point to be tracked in the previous frame image, the to-be-tracked key point is generated. The auxiliary position information of the key point; according to the position information of the key point to be tracked in the current frame and each frame image before the current frame and the auxiliary position information of the key point to be tracked generated each time, the current frame image is processed. Keypoint track rendering; and display the current frame image after rendering.

According to an embodiment of the present disclosure, the generation of the key point to be tracked is generated according to the position information of the key point to be tracked in the current frame image and the position information of the key point to be tracked in the previous frame image. Auxiliary position information, including: performing linear interpolation processing on the position information of the key point to be tracked in the current frame image and the position information of the key point to be tracked in the previous frame image to generate the key point to be tracked Auxiliary location information for the point.

According to an embodiment of the present disclosure, performing linear interpolation processing on the position information of the key point to be tracked in the current frame image and the position information of the key point to be tracked in the previous frame image, to generate the The auxiliary position information of the key point to be tracked includes: obtaining the current frame image according to the position information of the key point to be tracked in the current frame image and the position information of the key point to be tracked in the previous frame image. The pixel distance between the key point to be tracked in the frame image and the key point to be tracked in the previous frame image; according to the pixel distance and a preset interpolation gap, determine the key to be tracked to be interpolated The number of auxiliary position information of the point; according to the position information of the key point to be tracked in the current frame image, the position information of the key point to be tracked in the previous frame image, and the assistance of the key point to be tracked The quantity of position information determines the auxiliary position information of the key point to be tracked.

According to an embodiment of the present disclosure, according to the position information of the key point to be tracked in the current frame and each frame of images before the current frame and the auxiliary position information of the key point to be tracked generated each time, Performing key point trajectory rendering on the frame image, including: rendering the stored texture according to the position information of the key point to be tracked in the current frame image and the auxiliary position information of the key point to be tracked generated this time, wherein , the stored texture is generated by rendering according to the position information of the key points to be tracked in each frame image before the current frame and the auxiliary position information of the key points to be tracked generated each time before this time; and Render the rendered texture onto the current frame image.

According to an embodiment of the present disclosure, when the current frame image is the first frame image, the stored texture is a blank texture.

According to an embodiment of the present disclosure, the size of the stored texture is consistent with the size of the video.

According to an embodiment of the present disclosure, the method further includes: based on the video recording start instruction, clearing the stored position information of the key point to be tracked and the auxiliary position information of the key point to be tracked, or replacing the stored texture with blank texture, and continue to perform the step of acquiring the current frame image and the key points to be tracked in the current frame image.

According to an embodiment of the present disclosure, the method further includes: based on a special effect clearing instruction, clearing the stored position information of the key point to be tracked and the auxiliary position information of the key point to be tracked, or replacing the stored texture with blank texture, and display the re-acquired current frame image.

According to an embodiment of the present disclosure, the method further includes: continuing to perform the step of obtaining the current frame image and the key points to be tracked in the current frame image based on the special effect restoration instruction.

According to a second aspect of the embodiments of the present disclosure, there is provided an apparatus for displaying video special effects, including: a first obtaining unit, configured to execute an instruction for opening special effects based on a key-point trajectory, and obtain a current frame image and an image in the current frame image a key point to be tracked; a second acquisition unit, configured to perform acquisition of the position information of the key point to be tracked in the current frame image; a generating unit, configured to perform according to the current frame image of the key point to be tracked The position information of the point and the position information of the key point to be tracked in the previous frame image, generate auxiliary position information of the key point to be tracked; the rendering unit is configured to perform according to the current frame and each frame image before the current frame. The position information of the key points to be tracked and the auxiliary position information of the key points to be tracked generated each time, perform key point trajectory rendering on the current frame image; frame image.

According to an embodiment of the present disclosure, the generating unit includes: a generating subunit, configured to perform the analysis of the position information of the key point to be tracked in the current frame image and the to-be-tracked key point in the previous frame image Linear interpolation processing is performed on the position information of the tracking key points to generate auxiliary position information of the to-be-tracked key points.

According to an embodiment of the present disclosure, the generation subunit includes: an acquisition module configured to execute the process according to the position information of the key points to be tracked in the current frame image and the to-be-tracked key points in the previous frame image Tracking the position information of the key point, and obtaining the pixel distance between the key point to be tracked in the current frame image and the key point to be tracked in the previous frame image; the first determining module is configured to execute according to The pixel distance and the preset interpolation gap determine the amount of auxiliary position information of the to-be-tracked key point that needs to be interpolated; a second determination module is configured to perform a calculation according to the to-be-tracked key point in the current frame image. The number of position information of the key point to be tracked, the position information of the key point to be tracked and the auxiliary position information of the key point to be tracked in the previous frame of image, to determine the auxiliary position information of the key point to be tracked.

According to an embodiment of the present disclosure, the rendering unit includes: a first rendering sub-unit, configured to execute the current frame image according to the position information of the to-be-tracked key point and the to-be-tracked generated this time Auxiliary position information of key points to render the stored texture, wherein the stored texture is based on the position information of the key points to be tracked in each frame of the image before the current frame and the generated data each time before this time. The auxiliary position information of the key point to be tracked is generated by rendering; and a second rendering subunit is configured to perform rendering of the rendered texture onto the current frame image.

According to an embodiment of the present disclosure, the first obtaining unit is further configured to perform: based on the video recording start instruction, clear the stored position information of the key point to be tracked and the auxiliary position information of the key point to be tracked, Or replace the stored texture with a blank texture, and continue to perform the step of acquiring the current frame image and the key points to be tracked in the current frame image.

According to an embodiment of the present disclosure, the first obtaining unit is further configured to execute: based on a special effect clearing instruction, clear the stored position information of the key point to be tracked and the auxiliary position information of the key point to be tracked, or Replace the stored texture with a blank texture, and display the re-acquired current frame image.

According to an embodiment of the present disclosure, the first obtaining unit is further configured to execute: based on the special effect restoration instruction, continue to execute the step of obtaining the current frame image and the key points to be tracked in the current frame image.

According to a third aspect of embodiments of the present disclosure, there is provided an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to achieve The method for displaying video special effects provided by the embodiments of the first aspect of the present disclosure.

According to a fourth aspect of the embodiments of the present disclosure, there is provided a storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, the electronic device can perform the operations provided by the embodiments of the first aspect of the present disclosure. How to display video effects.

The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects: by obtaining the source short video and calling the first display page, the target operation of the music material is monitored, so as to generate a call instruction after the target operation is monitored, Jump to the music editing page, load the target music material on the music editing page, and then in response to the editing operation, edit the target music material and load it into the source short video to generate the target short video. Therefore, the present disclosure simplifies the interaction scheme, so that after the source short video is acquired, the first display page can be directly called, and the target operation can be monitored on the first display page, and then after the target operation is monitored, the target operation can be directly monitored. Jump from the first display page to the music editing page, edit the target music material, and then obtain the target short video, thereby reducing the number of interactions, shortening the operation time, reducing the learning cost, and greatly improving the user experience.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Description of drawings

The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the principles of the present disclosure and do not unduly limit the present disclosure.

Fig. 1 is a flow chart of a method for displaying video special effects according to an exemplary embodiment.

Fig. 2 is a schematic diagram of a main interface according to an exemplary embodiment.

Fig. 3 is a schematic diagram of a special effect preview interface according to an exemplary embodiment.

FIG. 4 is a schematic diagram of displaying a current frame image on a special effect preview interface according to an exemplary embodiment.

Fig. 5 is a schematic diagram showing a result of selecting a target object according to an exemplary embodiment.

Fig. 6 is a schematic diagram of a rendered current frame image according to an exemplary embodiment.

Fig. 7 is a flow chart of another method for displaying video special effects according to an exemplary embodiment.

Fig. 8 is a flow chart of yet another method for displaying video special effects according to an exemplary embodiment.

Fig. 9 is a progressive schematic diagram of a keypoint trajectory changing process according to an exemplary embodiment.

Fig. 10 is a flow chart of yet another method for displaying video special effects according to an exemplary embodiment.

Fig. 11 is a flowchart illustrating another method for displaying video special effects according to an exemplary embodiment.

Fig. 12 is a block diagram of an apparatus for displaying video special effects according to an exemplary embodiment.

Fig. 13 is a block diagram of another apparatus for displaying video special effects according to an exemplary embodiment.

Fig. 14 is a block diagram of an electronic device according to an exemplary embodiment.

detailed description

In order to make those skilled in the art better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

It should be noted that the terms "first", "second" and the like in the description and claims of the present disclosure and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used may be interchanged under appropriate circumstances such that the embodiments of the disclosure described herein can be practiced in sequences other than those illustrated or described herein. The implementations described in the illustrative examples below are not intended to represent all implementations consistent with this disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as recited in the appended claims.

Fig. 1 is a flow chart of a method for displaying video special effects according to an exemplary embodiment. It should be noted that the execution subject of the method for displaying video special effects of the present disclosure is a display device for video special effects. The method for displaying video special effects in the embodiment of the present disclosure may be executed by the apparatus for displaying video special effects in the embodiment of the present disclosure, and the apparatus for displaying video special effects may specifically be a hardware device, or software in the hardware device. Among them, the hardware devices are, for example, terminal devices, servers, and the like. As shown in FIG. 1 , the method for displaying video special effects proposed in this embodiment includes the following steps:

In step 101, the current frame image and the to-be-tracked key points in the current frame image are acquired based on the key point track special effect opening instruction.

The key point track special effect may be a special effect in which any display effect is superimposed on a track moving along the key point. For example, you can superimpose a yellow beam or a white beam along the trajectory of the keypoint movement.

The key point may be any key position of the target object in the current frame image. For example, parts such as the nose, eyes, and fingertips of the target object can be used as key points to be tracked. For another example, objects such as the target object's glasses, a handheld microphone, and a worn video can be used as key points to be tracked.

In the embodiment of the present disclosure, when attempting to add special effects to a video, the user may issue an instruction for adding special effects by clicking on the special effect control, inputting voice information, or the like. Correspondingly, after detecting an instruction for adding special effects issued by the user, at least one special effect identifier may be loaded in the special effect preview interface for the user to select the special effect identifier according to requirements. The special effect identifiers include key point trajectory special effects, such as special effect identifiers such as magic nose, magic eyes, and magic fingers.

As a possible implementation, when trying to detect the special effect addition instruction, the position of the target operation performed by the user on the main interface can be obtained, and when it is detected that the position of the target operation is the operation area where special effects are added, the user can Make sure that the Add Effects command is detected.

Further, it is possible to continue to detect the instruction issued by the user, and after detecting the instruction to open the key point trajectory special effect, add the key point trajectory special effect to the video.

As a possible implementation, when trying to detect the opening command of the key point trajectory special effect, the position of the target operation performed by the user on the special effect preview interface can be obtained, and the position where the target operation is detected is the key point trajectory special effect. When the special effect identifies the operation area, it can be determined that the key point track special effect opening command is detected.

It should be noted that, in the present disclosure, in order to simplify the interaction process and improve the user experience, various functional controls can also be displayed on the special effect preview interface. For example, recording controls can be displayed on the special effect preview interface to facilitate the user to select after previewing. After selecting the required special effects, you can directly perform the operation of adding key point track special effects to the recorded video.

For example, as shown in FIG. 2 , the user can trigger the adding special effect function by clicking on the adding special effect control 12 on the main interface 11 . Correspondingly, after detecting that the user clicks on the adding special effect control 12 , as shown in FIG. 3 , you can In the special effect preview interface 13 , the special effect signs 14 - 1 to 14 - 4 , 4 special effect signs in total, and the shooting control 15 are loaded. Further, when it is detected that the position of the target operation performed by the user on the special effect preview interface is the special effect identification operation area of the key point trajectory special effect, it is determined that the key point trajectory special effect opening instruction is detected.

In the embodiment of the present disclosure, the current frame image may be acquired by a video, image or other acquisition device based on the key point trajectory special effect opening instruction, and the acquired current frame image may be displayed on the special effect preview interface.

For example, the current frame image 16 as shown in FIG. 4 can be acquired through a camera, and the acquired current frame image 16 can be displayed on the special effect preview interface 13 .

It should be noted that, after the current frame image is obtained in the above manner, it can be first determined whether the obtained current frame image includes only one object, and if it is recognized that the current frame image includes only one object, the object can be used as the target object; If it is recognized that the current frame image includes more than one object, one of the objects can be selected as the target object. The target object is an object to which video special effects need to be added later.

It should be noted that the selection method of the target object is not limited in the present disclosure, and can be selected according to the actual situation. For example, the target object can be selected according to the recognition integrity and confidence of the joint points of each object. For example, as shown in FIG. 5 , after the current frame image is acquired, it is recognized that the current frame image includes 17-1 and 17-2, a total of 2 objects, and the object 17-1 can be the target object.

Further, after the current frame image is acquired, the key points to be tracked in the current frame image may be acquired based on the characteristics of the target object.

It should be noted that, the method for obtaining the key points to be tracked is not limited in the present disclosure, and can be selected according to the actual situation. For example, straight line-like regions can be detected by using the contour image, and key points to be tracked in the current frame image can be identified by combining various straight line regions. For another example, the key points to be tracked may be described by color features and/or shape features, and then the key points to be tracked in the current frame image are acquired by means of feature matching.

In step 102, the position information of the key points to be tracked in the current frame image is acquired.

In the embodiment of the present disclosure, after acquiring the key points to be tracked in the current frame image, the position information of the key points to be tracked may be acquired.

As a possible implementation, the image position of the key point to be tracked in the current frame image can be determined, and the position mapping between the current frame image and the special effect preview interface preset in the present disclosure can be queried according to the image position, and the to-be-tracked key point can be obtained. The position information of the key point on the special effect preview interface.

Optionally, the coordinates of the key points to be tracked in the current frame image can be determined according to the acquired key points to be tracked in the current frame image, and based on the pre-established same reference coordinate system of the current frame image and the special effects preview interface, obtain The coordinates of the key points to be tracked in the special effect preview interface, and then the position information of the key points to be tracked on the special effect preview interface is obtained.

In step 103, the auxiliary position information of the key point to be tracked is generated according to the position information of the key point to be tracked in the current frame image and the position information of the key point to be tracked in the previous frame image.

The auxiliary position information refers to the position information obtained by performing interpolation processing on the key points to be tracked.

In the embodiment of the present disclosure, after obtaining the position information of the key points to be tracked in the current frame image, the position information of the key points to be tracked in the previous frame image can be read, and the key points to be tracked in the current frame image can be read. Perform linear interpolation processing on the position information of the key point to be tracked and the position information of the key point to be tracked in the previous frame image to generate auxiliary position information of the key point to be tracked.

In step 104, key point trajectory rendering is performed on the current frame image according to the position information of the key points to be tracked in the current frame and each frame image before the current frame and the auxiliary position information of the key points to be tracked generated each time.

The key point trajectory rendering for the current frame image refers to the process of generating a rendering result (ie, the current frame image) according to information such as texture and lighting information.

In the embodiment of the present disclosure, when trying to perform keypoint trajectory rendering on the current frame image, the position information of the keypoints to be tracked in the current frame and each frame image before the current frame and the position information of the keypoints to be tracked generated each time can be used. Auxiliary position information and pre-stored textures are used to perform keypoint trajectory rendering on the current frame image to obtain the rendered current frame image.

The pre-stored texture is a texture whose size is the same as that of the video.

In step 105, the rendered current frame image is displayed.

In the embodiment of the present disclosure, after the keypoint trajectory rendering is performed on the current frame image, the rendered current frame image can be obtained and displayed on the screen to realize the display of video special effects.

For example, as shown in FIG. 6 , based on the key point track special effect opening instruction, add key point track special effects to the to-be-tracked key point "target user's nose 19" in the current frame image 18-1, so as to obtain the rendered current Frame image 18-2 and show it. The rendered current frame image 18-2 includes a black light 20 superimposed along the trajectory of the target user's nose 19 moving.

It should be noted that after the rendered current frame image is obtained, it is not displayed on the screen first. After the background is processed to obtain a new texture, the texture is superimposed on the current frame image, and then the superimposed image is displayed. on the screen.

The method for displaying video special effects provided by the embodiments of the present disclosure can obtain the current frame image and the key points to be tracked in the current frame image based on the key point track special effect opening instruction, and obtain the position information of the key points to be tracked in the current frame image and the position information of the key points to be tracked in the previous frame image to generate auxiliary position information of the key points to be tracked, and then according to the position information of the key points to be tracked in the current frame and each frame image before the current frame and each generation The auxiliary position information of the key points to be tracked, the key point trajectory rendering is performed on the current frame image, and the rendered current frame image is displayed to realize the display of video special effects. Therefore, the present disclosure can generate auxiliary position information by performing linear interpolation processing on the position information of the key point to be tracked in the current frame image and the position information of the key point to be tracked in the previous frame image, and then according to the position of the key point to be tracked information and auxiliary position information, perform key point trajectory rendering on the current frame image, so that the displayed trajectory of the key points in the rendered current frame image is smooth and continuous, and there is no gap. Further, the present disclosure can directly render the key point trajectory through the key points obtained in each frame of image, which can realize the rapid rendering of the key point trajectory, avoid the technical problem of rendering stuck, and greatly shorten the display of video special effects. time-consuming process.

It should be noted that, in the present disclosure, in order to ensure that the trajectories of the key points in the displayed rendered current frame image are smooth and continuous, and there is no longer a gap, the position information of the key points to be tracked in the current frame image and the upper Linear interpolation is performed on the position information of the key points to be tracked in a frame of images to generate auxiliary position information of the key points to be tracked.

As a possible implementation, as shown in FIG. 7 , on the basis of the above embodiment, in the above step S103, the position information of the key point to be tracked in the current frame image and the position of the key point to be tracked in the previous frame image are compared. The process of performing linear interpolation processing on the information to generate auxiliary position information of the key points to be tracked includes the following steps:

In step 201, according to the position information of the key points to be tracked in the current frame image and the position information of the key points to be tracked in the previous frame image, obtain the key points to be tracked in the current frame image and the key points to be tracked in the previous frame image pixel distance between.

The pixel distance may be a straight-line distance between the key point to be tracked in the current frame image and the key point to be tracked in the previous frame image.

In the embodiment of the present disclosure, based on the size information of the current frame image, according to the position information of the key points to be tracked in the current frame image and the position information of the key points to be tracked in the previous frame image, the pixel distance can be calculated by the following formula:

Among them, (x ₀ , y ₀ ) is the position information of the key point to be tracked in the current frame image, and (x ₁ , y ₁ ) is the position information of the key point to be tracked in the previous frame image.

_{For example, obtain the position information (x 0} , y ₀ )=(2, 2) of the key point to be tracked in the current frame image, and _{the position information (x 1} , y ₁ ) of the key point to be tracked in the previous frame image = (8, 10), then the pixel distance between the two key points can be obtained as 10.

In step 202, according to the pixel distance and a preset interpolation gap, the quantity of auxiliary position information of the key points to be tracked to be interpolated is determined.

Wherein, the interpolation gap may be the minimum interpolation distance for linear interpolation processing in the direction of the connection between the key point to be tracked in the current frame image and the key point to be tracked in the previous frame image.

In the embodiment of the present disclosure, after the pixel distance is obtained, the quantity of auxiliary position information of the key points to be tracked to be interpolated may be determined based on the interpolation gap. Among them, the interpolation gap can be set according to the actual situation. For example, the interpolation gap can be set to 5, 7, etc.

For example, if the obtained pixel distance is 10 and the interpolation gap is 5, it can be determined that the number of auxiliary position information of the key point to be tracked to be interpolated is 1.

In step 203, the auxiliary position of the key point to be tracked is determined according to the position information of the key point to be tracked in the current frame image, the position information of the key point to be tracked in the previous frame image and the auxiliary position information of the key point to be tracked information.

The auxiliary position information includes first auxiliary position information in the horizontal direction and second auxiliary position information in the vertical direction.

In the embodiment of the present disclosure, the component of the interpolation gap in the horizontal direction can be obtained according to the interpolation gap, and then according to the obtained component, the position information of the key points to be tracked in the current frame image and the to-be-tracked in the previous frame image The position information of the key point is obtained, and the first auxiliary position information is obtained. Further, according to the first auxiliary position information, the position information of the key point to be tracked in the current frame image and the position information of the key point to be tracked in the previous frame image, the second auxiliary position information can be calculated by the following formula:

Wherein, x is the first auxiliary position information, and y is the second auxiliary position information.

For example, the position information of the key point to be tracked in the current frame image is (2, 2), the position information of the key point to be tracked in the previous frame image is (8, 10), and the auxiliary position of the key point to be tracked is obtained. If the number of information is 1 and the interpolation gap is 5, the component of the interpolation gap in the horizontal direction can be obtained as 3, and then according to the obtained components, the position information of the key points to be tracked in the current frame image and the previous frame image In the position information of the key point to be tracked, the first auxiliary position information x=5 is obtained. Further, the second auxiliary position information y=6 can be calculated according to the first auxiliary position information, the position information of the key points to be tracked in the current frame image, and the position information of the key points to be tracked in the previous frame image. That is, the auxiliary position information of the key point to be tracked is (5, 6).

It should be noted that, in the process of generating the auxiliary position information of the key point to be tracked, if a decimal appears, it can be obtained by rounding down to ensure that the auxiliary position information is an integer.

The method for displaying video special effects provided by the embodiments of the present disclosure can obtain the pixel distance between the key point to be tracked in the current frame image and the key point to be tracked in the previous frame image, and calculate the pixel distance according to the pixel distance and the preset Interpolate the gap to determine the amount of auxiliary position information of the key points to be tracked to be interpolated, and then according to the position information of the key points to be tracked in the current frame image, the position information of the key points to be tracked in the previous frame image, and the key to be tracked. The number of auxiliary position information of the point to determine the auxiliary position information of the key point to be tracked. Therefore, the present disclosure can generate auxiliary position information by performing linear interpolation processing on the position information of the key points to be tracked in the current frame image and the position information of the key points to be tracked in the previous frame image, so as to ensure that the obtained auxiliary position information can be The trajectories of key points in the current frame image are complemented, so that the displayed trajectories of key points in the rendered current frame image are smooth and continuous, and there is no gap.

It should be noted that, in the present disclosure, in order to achieve fast rendering of key point trajectories, avoid rendering freezes, and shorten the time-consuming process of video special effects display, the key points obtained in each frame of image can be directly processed into key point trajectories. Render to get the rendered current frame image.

As a possible implementation, as shown in FIG. 8 , on the basis of the above embodiment, in the above step S104 , according to the position information of the key points to be tracked in the current frame and each frame image before the current frame and the The process of tracking the auxiliary position information of the key points and rendering the key point trajectory for the current frame image includes the following steps:

In step 301, the stored texture is rendered according to the position information of the key point to be tracked in the current frame image and the auxiliary position information of the key point to be tracked generated this time, wherein the stored texture is based on each frame before the current frame. Generated by rendering the position information of the key points to be tracked in the image and the auxiliary position information of the key points to be tracked generated each time before this time.

In this embodiment of the present disclosure, the track of the key points can be updated to the key points to be tracked in each image frame before the current frame according to the position information of the key points to be tracked in the current frame image and the auxiliary position information of the key points to be tracked generated this time. The position information of the point and the auxiliary position information of the key points to be tracked generated each time before this time are rendered on the texture generated, and the rendered texture is saved.

It should be noted that, if the current frame image is the first frame image, the stored texture is a blank texture, and the size of the texture is consistent with the size of the video.

For example, by recording the process of rendering the stored texture according to the position information of the key points to be tracked in the current frame image and the auxiliary position information of the key points to be tracked generated this time, it can be obtained as shown in FIG. 9 . Schematic diagram of the keypoint trajectory change process.

In step 302, the rendered texture is rendered onto the current frame image.

In this embodiment of the present disclosure, the saved rendered texture may be rendered onto the current frame image to obtain the rendered current frame image.

It should be noted that the specific process of rendering is in the prior art, which will not be repeated here.

The method for displaying video special effects provided by the embodiments of the present disclosure can render the stored texture according to the position information of the key points to be tracked in the current frame image and the auxiliary position information of the key points to be tracked generated this time, and render the stored textures. The rendered texture is rendered onto the current frame image to obtain the rendered current frame image. As a result, the present disclosure can directly render the key point trajectory through the key points obtained in each frame of image, which can realize the rapid rendering of the key point trajectory, avoid the technical problem of rendering stuck, and greatly shorten the display of video special effects. time-consuming process.

It should be noted that, in practical applications, users may be dissatisfied with the current trajectory of key point movement, want to try other key point movement methods, or select other objects in the current frame image as target objects to experience key point trajectory special effects, etc. For this reason, the special effect clearing instruction is sent, and thus, in the present disclosure, the rendering effect of the key point trajectory of the current frame can be cleared based on the special effect clearing instruction, and the re-acquired current frame image can be displayed.

As a possible implementation, as shown in FIG. 10 , on the basis of the above embodiment, based on the special effect removal instruction, the process of displaying the re-acquired rendered current frame image specifically includes the following steps:

In step 401, based on the special effect clearing instruction, the re-acquired current frame image is displayed.

In the embodiment of the present disclosure, the stored position information of the key points to be tracked and the auxiliary position information of the key points to be tracked can be cleared based on the special effect clearing instruction, or the stored texture can be replaced with a blank texture, and the re-acquired current frame can be displayed. image.

It should be noted that, when trying to clear the special effects of the key points in the current frame image, the user can issue a command to clear the special effects by clicking in various ways, such as double-clicking the screen, inputting the voice message of "Please clear the current special effects", etc. Correspondingly, after detecting the special effect clearing instruction issued by the user, the current special effect can be cleared based on the special effect clearing instruction, and the re-acquired current frame image is displayed.

In step 402, the current frame image and the key points to be tracked in the current frame image are acquired based on the special effect restoration instruction.

In this embodiment of the present disclosure, the step of acquiring the current frame image and the key points to be tracked in the current frame image may be continued based on the special effect restoration instruction.

It should be noted that, when trying to restore the special effect of the key point track in the current frame image, the user can issue the restore special effect instruction by clicking in various ways, such as double-clicking the screen, inputting the voice information of "please restore the current special effect", etc. Correspondingly, after detecting the restoration special effect instruction issued by the user, the step of acquiring the current frame image and the key points to be tracked in the current frame image may be continued based on the special effect restoration instruction.

In step 403, the position information of the key points to be tracked in the current frame image is acquired.

In step 404, auxiliary position information of the key point to be tracked is generated according to the position information of the key point to be tracked in the current frame image and the position information of the key point to be tracked in the previous frame image.

In step 405, according to the position information of the key points to be tracked in the current frame and each frame image before the current frame and the auxiliary position information of the key points to be tracked generated each time, key point trajectory rendering is performed on the image of the current frame.

In step 406, the rendered current frame image is displayed.

The above steps 403 to 406 are the same as the steps 102 to 105 in the embodiment shown in FIG. 1 , and are not repeated here.

The method for displaying video special effects provided by the embodiments of the present disclosure can clear the stored position information of the key points to be tracked and the auxiliary position information of the key points to be tracked, or replace the stored textures with blank textures, based on the special effect clearing instruction, And display the re-acquired current frame image, and then continue to perform the steps of obtaining the current frame image and the key points to be tracked in the current frame image, providing users with the rendering effect of clearing the current frame key point trajectory, and re-previewing the key point trajectory special effects display gameplay, The interaction process is simplified and the operation is convenient, which further improves the user experience.

Therefore, the present disclosure completes the trajectory of key points based on the linear interpolation processing method, and directly renders the trajectory of the key points obtained in each frame of image, so that the user selects the trajectory of the key points on the special effects preview interface. After the special effect, you can quickly see the smooth and continuous key point trajectory special effect display effect of the trajectory.

Further, after the user completes the preview of the display effect of the key point trajectory special effect, a shooting instruction can be issued to shoot, so as to obtain a video with the key point trajectory special effect added.

As a possible implementation, as shown in FIG. 11 , on the basis of the above embodiment, based on the video recording start instruction, the process of adding key point trajectory special effects to the recorded video specifically includes the following steps:

In step 501, based on the video recording start instruction, the stored location information of the key points to be tracked and the auxiliary location information of the key points to be tracked are cleared.

It should be noted that, when trying to record a video, the user can issue a command to clear the special effect by clicking in various ways, such as clicking the recording control on the special effect preview interface, and entering the voice message of "Please start recording video", etc. Correspondingly, after the video recording start instruction issued by the user is detected, the stored location information of the key points to be tracked and auxiliary location information of the key points to be tracked may be cleared based on the video recording start instruction.

As a possible implementation, when trying to record a video, the position of the target operation performed by the user on the special effect preview interface can be obtained, and when the position of the target operation is detected to be the recording control operation area, it can be determined that the target operation has been detected. Video recording start command.

In step 502, the current frame image and the key points to be tracked in the current frame image are acquired.

In step 503, the position information of the key points to be tracked in the current frame image is acquired.

In step 504, auxiliary position information of the key point to be tracked is generated according to the position information of the key point to be tracked in the current frame image and the position information of the key point to be tracked in the previous frame image.

In step 505, according to the position information of the key points to be tracked in the current frame and each frame image before the current frame and the auxiliary position information of the key points to be tracked generated each time, key point trajectory rendering is performed on the current frame image.

In step 506, the rendered current frame image is displayed.

The foregoing steps 502 to 506 are the same as the steps 101 to 105 in the embodiment shown in FIG. 1 , and are not repeated here.

The method for displaying video special effects provided by the embodiments of the present disclosure can clear the stored position information of the key points to be tracked and the auxiliary position information of the key points to be tracked, or replace the stored textures with blank textures based on the video recording start instruction , and continue to perform the steps of obtaining the current frame image and the key points to be tracked in the current frame image, so that the user can add smooth and continuous key point trajectory effects to the video through simple interaction, which further improves the user experience.

12 to 13 are block diagrams of an apparatus for displaying video special effects according to an exemplary embodiment.

As shown in FIG. 12 , the apparatus 1000 includes a first acquiring unit 121 , a second acquiring unit 122 , a generating unit 123 , a rendering unit 124 and a displaying unit 125 .

The first obtaining unit 121 is configured to execute an instruction based on the keypoint track special effect opening, to obtain the current frame image and the to-be-tracked keypoint in the current frame image;

The second obtaining unit 122 is configured to obtain the position information of the key points to be tracked in the current frame image;

The generating unit 123 is configured to generate auxiliary position information of the key point to be tracked according to the position information of the key point to be tracked in the current frame image and the position information of the key point to be tracked in the previous frame image;

The rendering unit 124 is configured to perform key point trajectory rendering on the current frame image according to the position information of the key point to be tracked in the current frame and each frame image before the current frame and the auxiliary position information of the key point to be tracked generated each time;

The presentation unit 125 is configured to perform presentation of the rendered current frame image.

In the embodiment of the present disclosure, as shown in FIG. 13 , the generating unit 123 in FIG. 12 includes: a generating subunit 1231 , which is configured to perform the analysis of the position information of the key points to be tracked in the current frame image and the position information of the key points to be tracked in the previous frame image. The position information of the key points to be tracked is subjected to linear interpolation processing to generate auxiliary position information of the key points to be tracked.

In the embodiment of the present disclosure, as shown in FIG. 13 , the generating subunit 1231 includes: an acquiring module 12311, which is configured to execute the key points to be tracked according to the position information of the key points to be tracked in the current frame image and the key points to be tracked in the previous frame image. to obtain the pixel distance between the key point to be tracked in the current frame image and the key point to be tracked in the previous frame image; the first determination module 12312 is configured to perform a determination according to the pixel distance and the preset interpolation gap, determine The number of auxiliary position information of the key points to be tracked that needs to be interpolated; the second determination module 12313 is configured to execute the position information of the key points to be tracked in the current frame image, the position information of the key points to be tracked in the previous frame image, and The number of auxiliary position information of the key point to be tracked determines the auxiliary position information of the key point to be tracked.

In an embodiment of the present disclosure, as shown in FIG. 13 , the rendering unit 124 in FIG. 12 includes: a first rendering sub-unit 1241 configured to The generated auxiliary position information of the key points to be tracked is rendered to the stored texture, where the stored texture is based on the position information of the key points to be tracked in each frame of the image before the current frame and the key to be tracked generated each time before this time. The auxiliary position information of the point is generated by rendering; the second rendering subunit 1242 is configured to perform rendering of the rendered texture onto the current frame image.

In this embodiment of the present disclosure, the first rendering subunit 1241 is configured to execute a texture in which the stored texture is blank when the current frame image is the first frame image.

In an embodiment of the present disclosure, the first rendering sub-unit 1241 is configured to perform the storage of the size of the texture consistent with the size of the video.

In the embodiment of the present disclosure, the first obtaining unit 121 is further configured to execute a video recording start instruction, clear the stored position information of the key points to be tracked and the auxiliary position information of the key points to be tracked, or store the stored texture Replace it with a blank texture, and continue to perform the steps of obtaining the current frame image and the key points to be tracked in the current frame image.

In the embodiment of the present disclosure, the first obtaining unit 121 is further configured to execute a special effect-based clearing instruction, clear the stored position information of the key points to be tracked and the auxiliary position information of the key points to be tracked, or replace the stored texture It is a blank texture and displays the re-acquired current frame image.

In the embodiment of the present disclosure, the first obtaining unit 121 is further configured to execute the special effect-based restoration instruction, and continue to execute the step of obtaining the current frame image and the key points to be tracked in the current frame image.

Regarding the apparatus in the above-mentioned embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be described in detail here.

The display device for video special effects provided by the embodiments of the present disclosure can obtain the current frame image and the key points to be tracked in the current frame image based on the key point trajectory special effect opening instruction, and obtain the position information of the key points to be tracked in the current frame image and the position information of the key points to be tracked in the previous frame image to generate auxiliary position information of the key points to be tracked, and then according to the position information of the key points to be tracked in the current frame and each frame image before the current frame and each generation The auxiliary position information of the key points to be tracked, the key point trajectory rendering is performed on the current frame image, and the rendered current frame image is displayed to realize the display of video special effects. Therefore, the present disclosure can generate auxiliary position information by performing linear interpolation processing on the position information of the key point to be tracked in the current frame image and the position information of the key point to be tracked in the previous frame image, and then according to the position of the key point to be tracked information and auxiliary position information, perform key point trajectory rendering on the current frame image, so that the displayed trajectory of the key points in the rendered current frame image is smooth and continuous, and there is no gap. Further, the present disclosure can directly render the key point trajectory through the key points obtained in each frame of image, which can realize the rapid rendering of the key point trajectory, avoid the technical problem of rendering stuck, and greatly shorten the display of video special effects. time-consuming process.

In order to implement the above embodiments, the present disclosure further provides an electronic device. As shown in FIG. 14 , the electronic device 8000 includes: a processor 801 ; and one or more memories for storing executable instructions of the processor 801 802; wherein, the processor 801 is configured to execute the method for displaying video special effects described in the foregoing embodiments. The processor 801 and the memory 802 are connected by a communication bus.

In order to implement the above-mentioned embodiments, the present disclosure also provides a storage medium including instructions, such as a memory 802 including instructions, and the above-mentioned instructions can be executed by the processor 801 of the apparatus 1000 to complete the above-mentioned method. Alternatively, the storage medium may be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage equipment, etc.

Other embodiments of the present disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or techniques in the technical field not disclosed by the present disclosure . The specification and examples are to be regarded as exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

It is to be understood that the present disclosure is not limited to the precise structures described above and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

A method for displaying video special effects, comprising:

Obtain the current frame image and the to-be-tracked key point in the current frame image based on the key point track special effect opening instruction;

obtaining the position information of the key points to be tracked in the current frame image;

generating auxiliary position information of the key point to be tracked according to the position information of the key point to be tracked in the current frame image and the position information of the key point to be tracked in the previous frame image;

According to the position information of the key point to be tracked in the current frame and each frame image before the current frame and the auxiliary position information of the key point to be tracked generated each time, perform key point trajectory rendering on the image of the current frame; and

Displays the rendered image of the current frame.
The display method according to claim 1, wherein the generating the Describe the auxiliary location information of the key points to be tracked, including:

Linear interpolation processing is performed on the position information of the key point to be tracked in the current frame image and the position information of the key point to be tracked in the previous frame image to generate auxiliary position information of the key point to be tracked.
The display method according to claim 2, characterized in that, the step is to perform position information on the position information of the key point to be tracked in the current frame image and the position information of the key point to be tracked in the previous frame image. Linear interpolation processing to generate auxiliary position information of the key points to be tracked, including:

According to the position information of the key point to be tracked in the current frame image and the position information of the key point to be tracked in the previous frame image, obtain the key point to be tracked in the current frame image and the key point to be tracked The pixel distance between the key points to be tracked in the previous frame of image;

According to the pixel distance and a preset interpolation gap, determine the amount of auxiliary position information of the key points to be tracked to be interpolated;

Determine said Auxiliary position information of key points to be tracked.
The display method according to claim 1, wherein the position information of the key point to be tracked in the current frame and each frame of images before the current frame and the auxiliary position of the key point to be tracked generated each time information, perform keypoint trajectory rendering on the current frame image, including:

According to the position information of the key point to be tracked in the current frame image and the auxiliary position information of the key point to be tracked generated this time, the stored texture is rendered, wherein the stored texture is based on the Generated by rendering the position information of the key point to be tracked in each frame of the image before the current frame and the auxiliary position information of the key point to be tracked generated each time before this time; and

Render the rendered texture onto the current frame image.
The display method according to claim 4, wherein when the current frame image is the first frame image, the stored texture is a blank texture.
The display method according to claim 4, wherein the size of the stored texture is consistent with the size of the video.
The display method according to claim 1 or 4, further comprising:

Based on the video recording start instruction, clear the stored position information of the key point to be tracked and the auxiliary position information of the key point to be tracked, or replace the stored texture with a blank texture, and continue to execute the acquisition of the current Steps of key points to be tracked in the frame image and the current frame image.
The display method according to claim 1 or 4, further comprising:

Based on the special effect clear instruction, clear the stored position information of the key point to be tracked and the auxiliary position information of the key point to be tracked, or replace the stored texture with a blank texture, and display the re-acquired current frame image .
The display method according to claim 8, further comprising:

Based on the special effect restoration instruction, the step of obtaining the current frame image and the key points to be tracked in the current frame image is continued.
A display device for video special effects, comprising:

a first acquisition unit, configured to execute an instruction based on the keypoint track special effect opening, to acquire the current frame image and the to-be-tracked keypoint in the current frame image;

a second acquiring unit, configured to perform acquiring the position information of the key points to be tracked in the current frame image;

A generating unit configured to generate auxiliary position information of the key point to be tracked according to the position information of the key point to be tracked in the current frame image and the position information of the key point to be tracked in the previous frame image ;

The rendering unit is configured to perform the current frame image according to the position information of the key point to be tracked in the current frame and each frame image before the current frame and the auxiliary position information of the key point to be tracked generated each time. key trajectory rendering; and

The presentation unit is configured to perform presentation of the rendered current frame image.
The display device according to claim 10, wherein the generating unit comprises:

A generating subunit, configured to perform linear interpolation processing on the position information of the key point to be tracked in the current frame image and the position information of the key point to be tracked in the previous frame image, to generate the to-be-tracked key point Auxiliary position information for tracking key points.
The display device according to claim 11, wherein the generating subunit comprises:

The acquisition module is configured to perform, according to the position information of the key point to be tracked in the current frame image and the position information of the key point to be tracked in the previous frame image, to obtain the information in the current frame image. the pixel distance between the key point to be tracked and the key point to be tracked in the previous frame of image;

a first determination module, configured to determine the number of auxiliary position information of the to-be-tracked key point to be interpolated according to the pixel distance and a preset interpolation gap;

The second determination module is configured to perform a calculation according to the position information of the key point to be tracked in the current frame image, the position information of the key point to be tracked in the previous frame image, and the key point to be tracked. The quantity of auxiliary position information determines the auxiliary position information of the key point to be tracked.
The display device according to claim 10, wherein the rendering unit comprises:

The first rendering subunit is configured to perform rendering to the stored texture according to the position information of the key point to be tracked in the current frame image and the auxiliary position information of the key point to be tracked generated this time, wherein , the stored texture is generated by rendering according to the position information of the key points to be tracked in each frame image before the current frame and the auxiliary position information of the key points to be tracked generated each time before this time; and

The second rendering subunit is configured to perform rendering of the rendered texture onto the current frame image.
The display device according to claim 13, wherein when the current frame image is the first frame image, the stored texture is a blank texture.
The display device according to claim 13, wherein the size of the stored texture is consistent with the size of the video.
The display device according to claim 10 or 13, wherein the first obtaining unit is further configured to execute:

Based on the video recording start instruction, clear the stored position information of the key point to be tracked and the auxiliary position information of the key point to be tracked, or replace the stored texture with a blank texture, and continue to execute the acquisition of the current Steps of key points to be tracked in the frame image and the current frame image.
The display device according to claim 10 or 13, wherein the first obtaining unit is further configured to execute:

Based on the special effect clear instruction, clear the stored position information of the key point to be tracked and the auxiliary position information of the key point to be tracked, or replace the stored texture with a blank texture, and display the re-acquired current frame image .
The display device according to claim 17, wherein the first obtaining unit is further configured to execute:

Based on the special effect restoration instruction, the step of obtaining the current frame image and the key points to be tracked in the current frame image is continued.
An electronic device, comprising:

processor;

a memory for storing the processor-executable instructions;

Wherein, the processor is configured to execute the instructions to implement the method for displaying video special effects according to any one of claims 1 to 9.
A storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the method for displaying video special effects according to any one of claims 1 to 9.