CN116095250B

CN116095250B - Method and device for video cropping

Info

Publication number: CN116095250B
Application number: CN202211166928.8A
Authority: CN
Inventors: 吴孟函; 吴比
Original assignee: Honor Device Co Ltd
Current assignee: Honor Device Co Ltd
Priority date: 2022-05-30
Filing date: 2022-09-23
Publication date: 2023-10-31
Anticipated expiration: 2042-09-23
Also published as: CN117915020A; CN116095250A

Abstract

A method and a device for video clipping relate to the technical field of multimedia, and the method comprises the following steps: detecting a first user operation, wherein the first user operation is used for triggering a cropping operation of a first video frame in a cropping interface of a video editing application; responding to a first user operation, and recording a clipping parameter of a first display area corresponding to a first video frame, wherein the clipping parameter corresponds to the first user operation; invoking the GPU to draw the video frame according to the clipping parameters so as to obtain first picture data of a clipped first video frame; and displaying the updated clipping interface according to the first picture data, wherein the updated clipping interface comprises the clipped first video frame. The method can reduce delay and clamping situations of video pictures in the video clipping process, gives a user smoother operation experience, and improves the processing efficiency of video clipping.

Description

Method and device for video cropping

The present application claims priority from the chinese patent application filed 5-30-2022, filed 5-30, filed 202210601437.5, entitled "method and apparatus for video cropping", the entire contents of which are incorporated herein by reference.

Technical Field

The present application relates to the field of multimedia technologies, and in particular, to a method and apparatus for video cropping.

Background

With the popularization and development of the internet, video is becoming a mainstream multimedia carrier, and various video editing tools are also evolving and upgrading continuously, so as to provide video editing functions such as video special effects and video filters. Wherein video cropping is the basic function of video editing. Video cropping may refer to editing operations such as moving, scaling, rotating, cropping, etc., of a video picture and presenting the cropped video picture effect through a video play window.

However, in the process of performing video cropping by the video editing tool, various function calls are involved, for example, a video encoding and decoding module, logic calculation of a rendering tool, etc., which causes a phenomenon of delayed blocking in the video cropping process, and affects user experience.

Therefore, the industry is favored to study video cropping methods with higher processing efficiency and better user experience.

Disclosure of Invention

The embodiment of the application provides a method and a device for video clipping, which can reduce delay and blocking situations of video pictures in the video clipping process, provide a smoother operation experience for users, and improve the processing efficiency of video clipping.

In a first aspect, an embodiment of the present application provides a method for video cropping, applied to a terminal device, where the method includes: detecting a first user operation for triggering a cropping operation of a first video frame in a cropping interface of a video editing application; responding to the first user operation, and calling an application frame to record clipping parameters of a first display area corresponding to the first video frame, wherein the clipping parameters correspond to the first user operation; invoking an image processing unit GPU to draw the video frame according to the clipping parameters so as to obtain first picture data of a clipped first video frame; and displaying an updated clipping interface according to the first picture data, wherein the updated clipping interface comprises the clipped first video frame.

In this way, the clipping interface does not use the rendering module to render the video frame in the process of responding to the clipping operation of the user, namely does not trigger the rendering thread, but records and transmits clipping parameters when the application frame and the GPU are called to draw the video frame so as to update the picture of the clipped video frame. The method reduces multithread time slice scheduling when the clipping interface is displayed, so that the operation experience of the clipping interface is smoother, and the phenomenon of delay jamming in the clipping interface is improved.

In addition, rendering tools such as OpenGl and EGL are not introduced into the clipping interface, so that the information interacted between the editing interface and the clipping interface is reduced, the coupling degree of the editing interface and the clipping interface is reduced and the editing interface is independent, and the background management efficiency of video editing application is improved.

In one possible implementation, the first video frame is rendered without using a rendering module in response to the first user operation.

In one possible implementation, the clipping operation includes at least one of: rotation, movement, scaling, mirroring, picture cropping.

In one possible implementation, the clipping parameters include at least one of: rotation angle, mirror image parameters, clipping coordinates.

In one possible implementation manner, the recording, in response to the first user operation, a cropping parameter of a first display area corresponding to the first video frame includes: the video editing application sends a first instruction to the application framework, wherein the first instruction is used for instructing the application framework to initiate a video drawing process, the first instruction comprises a fifth instruction, and the fifth instruction is used for instructing the application framework to record the clipping parameters; the application framework records the clipping parameters.

In one possible implementation manner, the invoking the GPU to draw the video frame according to the clipping parameters includes: the GPU receives a second instruction sent by the application framework, the second instruction is used for indicating the GPU to draw video frames, and the second instruction comprises the clipping parameters; the GPU performs video frame drawing according to the clipping parameters to obtain the first picture data; and the GPU stores the first picture data into a first shared memory corresponding to the first display area.

In one possible implementation manner, the displaying a clipping interface according to the first picture data includes: the video editing application sends a third instruction to the application framework, wherein the third instruction is used for instructing the application framework to output the picture data in the shared memory corresponding to the first window view to the screen display, the first window view is the window view corresponding to the clipping interface, and the shared memory corresponding to the first window view comprises the first shared memory; the application framework sends a fourth instruction to the surfeFlinger, wherein the fourth instruction is used for notifying the surfeFlinger to output the picture data in the shared memory; the SurfaceFlinger superimposes and mixes the picture data in the shared memory corresponding to a plurality of window views into total picture data, wherein the plurality of window views comprise the first window view; and the SurfaceFlinger loads the total picture data into a cache area of a screen so as to display the clipping interface.

In one possible implementation, before the detecting the first user operation, the method further includes: detecting a second user operation, wherein the second user operation is used for triggering the operation of entering the cropping interface from the editing interface of the video editing application; and responding to the second user operation, displaying the cropping interface, wherein the cropping interface comprises the first video frame, and in the process of responding to the second user operation, a rendering module is not used for rendering the first video frame.

Therefore, in the process of entering the clipping interface, the rendering module is not initialized and the process of rendering the video frames by using the rendering module is not triggered, namely, the multi-thread time slice scheduling is reduced in the clipping interface, so that the operation experience of the clipping interface is smoother, and the phenomenon of delay blocking in the clipping interface is improved. In addition, rendering tools such as OpenGL and EGL are not introduced into the clipping interface, so that the information interacted between the editing interface and the clipping interface is reduced, the coupling degree of the editing interface and the clipping interface is reduced and the editing interface is independent, and the background management efficiency of video editing application is improved.

In one possible implementation, the displaying the clipping interface in response to the second user operation includes: calling a MediaCodec to create a codec; analyzing the video data corresponding to the first video frame according to the coder-decoder to obtain a first video frame in YUV format; drawing according to the first video frame in the YUV format to obtain second picture data corresponding to the first video frame; and displaying the clipping interface according to the second picture data.

In one possible implementation, the calling MediaCodec creates an encoder and a decoder, including: calling the MediaCodec to acquire a video file path corresponding to the first video frame, wherein the video file format indicates a compression storage mode of video data corresponding to the first video frame; and calling the MediaCodec to create the coder and decoder according to the video file path.

In one possible implementation manner, the calling the MediaCodec to obtain the video file path corresponding to the first video frame includes: the video editing application sends a video file path corresponding to the first video frame to the MediaCodec; the MediaCodec parses the video file path to obtain the video file format; the MediaCodec sends the video file format to the video editing application.

In one possible implementation, the calling the MediaCodec to create the codec from the video file path includes: the video editing application sends a seventh instruction for creating a codec to the MediaCodec, wherein the seventh instruction comprises the video file format; the MediaCodec creates a codec according to the video file format; the MediaCodec sends information to the video editing application indicating that codec creation was successful.

In one possible implementation, the method further includes: the video editing application applies for a first shared memory corresponding to the first display area to an internal memory, wherein the first shared memory is used for storing input data or output data of the coder-decoder; the memory sends information of successful creation of the shared memory to the video editing application, wherein the information of successful creation of the shared memory comprises an address of the first shared memory; the video editing application sends the address of the first shared memory to the MediaCodec.

In one possible implementation, the address of the first shared memory is carried in the first instruction.

In one possible implementation manner, parsing, according to the codec, the video data corresponding to the first video frame to obtain a first video frame in YUV format includes: the video editing application sends a command for analyzing video to the MediaCodec; the MediaCodec analyzes the video data corresponding to the first video frame through the coder-decoder to obtain a first video frame in YUV format; the MediaCodec stores the first video frame in the YUV format into a first shared memory corresponding to the first display area; the MediaCodec sends information of successful video parsing to the video editing application.

In a possible implementation manner, the drawing according to the first video frame in the YUV format to obtain second picture data corresponding to the first video frame includes: the video editing application sends a first instruction to the application framework, wherein the first instruction is used for indicating the application framework to initiate a video drawing process; the application framework sends a sixth instruction to the GPU, wherein the sixth instruction is used for instructing the GPU to draw video frames; the GPU performs drawing of video frames to obtain second picture data; and the GPU stores the second picture data into a first shared memory corresponding to the first display area.

In one possible implementation manner, displaying the clipping interface according to the second screen data includes: the video editing application sends a third instruction to the application framework, wherein the third instruction is used for instructing the application framework to output the picture data in the shared memory corresponding to the first window view to the screen display, the first window view is the window view corresponding to the clipping interface, and the shared memory corresponding to the first window view comprises the first shared memory; the application framework sends a fourth instruction to the surfeFlinger, wherein the fourth instruction is used for notifying the surfeFlinger to output the picture data in the shared memory; the SurfaceFlinger superimposes and mixes the picture data in the shared memory corresponding to a plurality of window views into total picture data, wherein the plurality of window views comprise the first window view; and the SurfaceFlinger loads the total picture data into a cache area of a screen so as to display the clipping interface.

In one possible implementation, after the detecting the first user operation, the method further includes: detecting a third user operation, wherein the third user operation is used for indicating that the cutting operation of the cutting interface is completed; and responding to the third user operation, displaying an editing interface, wherein the editing interface comprises the first video frame after clipping.

In a second aspect, a method for video cropping is provided, applied to a terminal device, the method comprising: detecting a first user operation for triggering a cropping operation of a first video frame in a cropping interface of a video editing application; and responding to the first user operation, displaying an updated clipping interface, wherein the updated clipping interface comprises the clipped first video frame, and a rendering module is not used for rendering the first video frame in the process of responding to the first user operation.

In one possible implementation, the displaying, in response to the first user operation, an updated clipping interface includes: calling an application framework to record a clipping parameter of a first display area corresponding to the first video frame, wherein the clipping parameter is used for indicating a parameter of the first display area which is changed due to the first user operation; invoking an image processing unit GPU to draw the video frame according to the clipping parameters so as to obtain first picture data of a clipped first video frame; and displaying the updated clipping interface according to the first picture data.

In a third aspect, embodiments of the present application provide an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, when executing the computer program, causing the electronic device to perform the method as described in the first aspect or any implementation of the first aspect or the method as described in the second aspect or any implementation of the second aspect.

In a fourth aspect, embodiments of the present application provide a computer-readable storage medium storing instructions that, when executed, cause a computer to perform a method as described in the first aspect or any implementation of the first aspect, or to perform a method as described in the second aspect or any implementation of the second aspect.

A fifth aspect, a computer program product comprising a computer program which, when run, causes a computer to perform the method as described in the first aspect or any implementation of the first aspect or to perform the method as described in the second aspect or any implementation of the second aspect.

It should be understood that, in the second aspect to the fifth aspect of the present application, corresponding to the technical solutions of the first aspect of the present application, the beneficial effects obtained by each aspect and the corresponding possible embodiments are similar, and are not repeated.

Drawings

FIGS. 1 (a) -1 (g) are diagrams of an interactive interface of a video cropping application according to an embodiment of the application;

FIG. 2 is a schematic diagram of a clip interface 400 for mirroring operations according to one embodiment of the application;

FIG. 3 is a schematic diagram of a cropping interface 400 for performing an enlargement operation according to an embodiment of the present application;

FIG. 4 is a schematic diagram of a cutting interface 400 for performing a rotation operation according to an embodiment of the present application;

FIG. 5 is a schematic diagram showing a video frame without a clip when performing a clipping operation according to an embodiment of the present application;

FIG. 6 is a schematic diagram of a video frame showing a clip when a clipping operation is performed according to an embodiment of the present application;

Fig. 7 is a software configuration block diagram of a terminal device according to an embodiment of the present application;

FIG. 8 is a schematic diagram of an internal implementation flow of a method for entering an edit page of a video editing application in accordance with one embodiment of the application;

FIG. 9 is a schematic diagram of the internal flow of creating a codec according to an embodiment of the present application;

FIG. 10 is a schematic diagram of an internal flow of initialization of a rendering module according to an embodiment of the application;

FIG. 11 is a schematic diagram illustrating an internal flow of a video parsing method according to an embodiment of the application;

FIG. 12 is a schematic diagram of an internal flow of a video frame rendering method according to an embodiment of the present application;

FIG. 13 is a schematic diagram of the internal flow of the process of rendering and delivering video frames according to an embodiment of the present application;

FIG. 14 is a flow chart of a method for video cropping of a video frame according to an embodiment of the application;

FIG. 15 is a flow chart of a method of entering a clipping interface according to an embodiment of the application;

FIG. 16 is a flow chart of a method for video cropping of a cropping interface according to an embodiment of the application;

FIG. 17 is a schematic diagram of the internal flow of the process of drawing and sending video frames at a cropping interface according to an embodiment of the application;

FIG. 18 is a flow chart of a method for clipping an interface back to an editing interface according to an embodiment of the application;

FIG. 19 is a schematic view of a clipping interface according to an embodiment of the application;

FIG. 20 is a flow chart of a method for video cropping of a video frame according to an embodiment of the application;

fig. 21 is a schematic hardware structure of a terminal device according to an embodiment of the present application;

FIG. 22 is a schematic structural diagram of an apparatus for video cropping according to an embodiment of the present application;

fig. 23 is a schematic hardware structure of another terminal device according to an embodiment of the present application.

Detailed Description

In order to clearly describe the technical solution of the embodiments of the present application, in the embodiments of the present application, the words "first", "second", etc. are used to distinguish the same item or similar items having substantially the same function and effect. For example, the first chip and the second chip are merely for distinguishing different chips, and the order of the different chips is not limited. It will be appreciated by those of skill in the art that the words "first," "second," and the like do not limit the amount and order of execution, and that the words "first," "second," and the like do not necessarily differ.

It should be noted that, in the embodiments of the present application, words such as "exemplary" or "such as" are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "for example" should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete fashion.

In the embodiments of the present application, "at least one" means one or more, and "a plurality" means two or more. "and/or", describes an association relationship of an association object, and indicates that there may be three relationships, for example, a and/or B, and may indicate: a alone, a and B together, and B alone, wherein a, B may be singular or plural. The character "/" generally indicates that the context-dependent object is an "or" relationship. "at least one of" or the like means any combination of these items, including any combination of single item(s) or plural items(s). For example, at least one (one) of a, b, or c may represent: a, b, c, a-b, a-c, b-c, or a-b-c, wherein a, b, c may be single or plural.

The method for video clipping provided by the embodiment of the application can be applied to terminal equipment with a display function. The terminal device may also be referred to as a terminal (terminal), a User Equipment (UE), a Mobile Station (MS), a Mobile Terminal (MT), etc. The terminal device may be a mobile phone, a smart television, a wearable device, a tablet (Pad), a computer with wireless transceiving function, a Virtual Reality (VR) terminal device, an augmented reality (augmented reality, AR) terminal device, a wireless terminal in industrial control (industrial control), a wireless terminal in unmanned driving (self-driving), a wireless terminal in teleoperation (remote medical surgery), a wireless terminal in smart grid (smart grid), a wireless terminal in transportation safety (transportation safety), a wireless terminal in smart city (smart city), a wireless terminal in smart home (smart home), or the like. The embodiment of the application does not limit the specific technology and the specific equipment form adopted by the terminal equipment.

An example will be described below taking a terminal device as a mobile phone, which example does not constitute a limitation of the embodiments of the present application.

As described above, in the process of performing video cropping by the video editing tool, various function calls, such as video encoding and decoding, rendering, etc., are involved, which may cause a phenomenon of delayed blocking during video cropping, and affect user experience. The process of video cropping will be described next with reference to the accompanying drawings. FIG. 1 is a schematic diagram of an interactive interface for video cropping according to an embodiment of the present application. As shown in fig. 1, video cropping includes the following procedure.

First, fig. 1 (a) shows an interface of a video library in a terminal device, which may be part of a gallery application, as an example, and may include one or more icons of videos therein, and a user may select a video to be edited and click into the display interface 200 of the video. Accordingly, after receiving the operation information of clicking the video by the user, the terminal device displays the display interface 200 of the video. Fig. 1 (b) is a display interface 200 of a video of a terminal device. As shown in fig. 1 (b), the display interface 200 includes buttons such as a video play button 211 and an edit button 213. The user may enter the editing interface 300 of the video editing application by clicking the edit button 213. Accordingly, after receiving the operation information that the user clicks the edit button 213, the terminal apparatus can present the edit interface 300.

It should be noted that, in the embodiment of the present application, the video editing application may be a function control or an add-in configured in the video library, or may be an independent application program, and the user may enter the editing interface 300 of the video application by clicking the editing button 213 of the video, or may enter the editing interface 300 of the video editing application by clicking an icon of the video editing application.

Fig. 1 (c) is a schematic diagram of an editing interface 300 of a video editing interface. As shown in fig. 1 (c), the editing interface 300 includes buttons corresponding to various functions for editing video, for example, buttons for cutting, a filter, text, a special effect, a sticker, and the like. In the event that a video needs to be cropped, the user may click on the crop button 310 in the editing interface 300 to enter the crop interface 400. The cropping interface 400 may be used to perform editing operations such as moving, zooming, mirroring, rotating, cropping, etc. on a video frame.

It should be noted that, in the embodiment of the present application, the video clipping may refer to a clipping operation performed on a video frame by a user in the clipping interface 400, where the clipping operation changes coordinates of four vertices of the video frame, so as to obtain a clipped video frame. Such clipping operations include, but are not limited to: moving, scaling, rotating, mirroring, picture cropping.

Clipping interface 400 referring to fig. 1 (d), clipping interface 400 may include a display area 308 for playing video and a display area 311 for clipping scale buttons. The cropping ratio refers to the aspect ratio of the cropped video frame. For example, a cut-to-scale button may include, but is not limited to, the following options: free, raw ratio, 9:16, 16:9, 3:4. Wherein, the freedom means that the width-to-length ratio of the cut video picture is not limited; the original ratio is that the ratio of the video picture after cutting is the same as that of the video picture before cutting. Optionally, the cropping interface 400 further includes a rotation button 301, a rotation degree button 302, and a mirror button 303. The rotation button 301 is used to rotate the video picture by a fixed angle, for example, by 90 ° counterclockwise or clockwise. The rotation number button 302 can be used to adjust a specific rotation number of the video picture, and the mirror button 303 is used to implement a mirror operation of the video picture.

Also included in the cropping interface 400 is a cropping frame 306, which may be resized by a user to determine the size of the cropped video frame by adjusting the size of the cropping frame 306. The operation of adjusting the crop box 306 to adjust the size of the video frame is the frame cropping described above. Illustratively, the user may resize the crop box 306 by a touch operation. With continued reference to fig. 1 (d), the user may click the cut-to-scale button "free" and perform a screen cut. "free" means that the screen is cut without limiting the scale of the crop frame. As shown in fig. 1 (e), the user may adjust the size of the cropping frame 306 through a touch operation, and accordingly, after receiving the touch operation information of the user, the terminal device may adjust the size of the cropping frame 306 according to the touch operation information and display the adjusted size in the cropping interface 400, and the display area 308 for playing the video changes accordingly. As shown in fig. 1 (f), the user may click on the cut complete button 304 (i.e., v in the figure) with the size of the cut frame 306 adjusted to be satisfactory. Accordingly, the terminal device may end the clipping after receiving the information that the user has transmitted the clipping completion.

After the user clicks the cut-out button 304, the terminal device may return to the editing interface 300, as shown in fig. 1 (g), at which time a cut-out video screen is presented in the editing interface 300. The user may continue clicking on the other edited function buttons for editing or may end the video editing operation.

Optionally, in the cropping interface 400, the user may perform one or more of the following operations on the video screen to implement the cropping function: moving, scaling, rotating, mirroring, picture cropping. For example, the user may perform a rotation operation on the video screen through the rotation button 301 or the rotation degree button 302, perform a mirror operation on the video through the mirror button 303, and move, zoom in, or zoom out the video screen through a touch gesture.

By way of example, fig. 2-4 are schematic diagrams of different operational states of a clipping interface 400 according to an embodiment of the application. Wherein fig. 2 is a schematic diagram of the operation of mirroring a video picture. As shown in fig. 2 (a), the user may click the mirror button 303 in the editing interface, and the terminal device may present a mirror image of the video image shown in fig. 2 (b) after receiving the operation information of clicking the mirror button 303 transmitted by the user.

Fig. 3 is a schematic diagram of an operation of enlarging a video picture. As shown in fig. 3 (a), the user may perform an operation of zooming in on the screen through a gesture touching the display screen, and the terminal device may present an zoomed-in screen of the video shown in fig. 3 (b) after receiving operation information of zooming in on the video screen transmitted by the user.

Fig. 4 is a schematic diagram of an operation of rotating a video picture. As shown in fig. 4 (a), the user may click on the rotation degree button 302 to adjust the rotation degree of the video screen so as to implement an operation of rotating the screen, and the terminal device may present the rotation screen of the video shown in fig. 4 (b) after receiving the operation information of rotating the video screen sent by the user. Alternatively, the user may perform a rotation operation of the video screen by clicking the button 302 of the degree of rotation, or may perform a rotation operation of the video screen by moving a finger on the display screen.

In the cropping interface 400, a user may delay or clip a video frame as the video frame is moved, scaled, rotated, or mirrored.

Fig. 5 and fig. 6 are schematic diagrams of video frames without and with a clip when clipping operation is performed according to an embodiment of the present application. It should be appreciated that in order to embody the distinction between displayed video picture frames, the extent of variation of successive video picture frames in fig. 5 and 6 may be greater than in the case of actual use.

In fig. 5, (a) - (c) show changes in video pictures of successive frames in order when the video pictures are rotated without being stuck. Under the condition of no clamping, in the rotating process, video frames can be smoothly played, and the clamping phenomenon can not occur. Fig. 6 (a) - (c) sequentially show changes in video pictures of successive frames when the video picture is subjected to a rotation operation in the stuck state. In the case of a stuck, during rotation, a frame loss phenomenon will occur between video frames, thereby causing a stuck or delayed.

One of the reasons for this is that in the cropping interface 400, when a user performs a moving, scaling, rotating or mirroring operation on a video frame, the video editing application needs to call the rendering module to perform a rendering operation on the video frame all the time, and then present the rendered frame on the cropping interface 400. Because performing the rendering operation increases the time required for data computation and rendering, it causes a delay or a stuck phenomenon in the video frame.

In order to improve the problems, the embodiment of the application provides a method and terminal equipment for video cropping. The method can reduce delay and clamping situations of video pictures in the video clipping process, gives a user smoother operation experience, and improves the processing efficiency of video clipping.

The software architecture of the terminal device according to the embodiment of the present application is first described below. Fig. 7 is a software configuration block diagram of a terminal device according to an embodiment of the present application.

The layered architecture divides the software system of the terminal device into a plurality of layers, each layer having a distinct role and division of labor. The layers communicate with each other through a software interface. In some embodiments, the Android system may be divided into four layers, an application layer (applications), an application framework layer (application framework), a system library, and a hardware layer, respectively. It should be understood that only the functional modules related to the method for video editing according to the embodiment of the present application are shown in fig. 7, and in practice, more or fewer functional modules may be included in the terminal device, or some of the functional modules may be replaced by other functional modules.

The application layer may include a series of application packages. As shown in fig. 7, the application layer includes a video editing application therein. The video editing application has video data processing capability, and can provide video editing functions for users, including video data processing functions such as clipping and rendering. As an example, the video editing application may include a VideoEditor in an Android system. The embodiment of the application can utilize the video editing application to realize the method for clipping the video.

Alternatively, the application layer may also include other applications, such as, for example, camera, gallery, calendar, talk, bluetooth, music, video, short message, etc. applications.

The application framework layer provides an application programming interface (application programming interface, API) and programming framework for application programs of the application layer. The application framework layer includes a number of predefined functions.

As shown in fig. 7, the application framework layer may include a media framework and an application framework.

A plurality of tools for editing video and audio are provided in the media frame. Wherein the tool comprises MediaCodec. MediaCodec is an Android-supplied module for encoding and decoding audio and video. It includes an encoder, a decoder, and a surface type cache.

The encoder provided by MeidaCodec may convert video or audio of one format input to the encoder to another format by compression techniques, while the decoder is used to perform the inverse of encoding to convert video or audio of one format input to the decoder to another format by decompression techniques.

MediaCodec may also apply for a shared memory, i.e., a surface memory in the hardware layer (hereinafter referred to as surface). Surface may be used to buffer video data. For example, after the terminal device performs an editing operation to obtain a rendered video image frame, the terminal device may input the image frame into a surface buffer. The application may then obtain the rendered video image frames from the surface for storage or display, etc.

The media framework also comprises a shared memory management module (Surface Flinger), wherein the Surface eFlink is a service function module provided by the media framework, is started in a system process, and manages Surface data of a plurality of system windows on a screen of the terminal equipment.

The shared memory management module is used for superposing and mixing surface data of views of application windows of a plurality of application programs into one view surface data according to the hierarchical sequence of screen display, and placing the view surface data into a cache area. The screen may display a view of the buffer area onto the screen while refreshing the picture according to a certain frequency.

An application FrameWork (FrameWork) is used as a FrameWork layer of Android, provides a plurality of callable function modules and API interfaces for an application layer App, and provides a window management module in the application FrameWork, which can manage window processing events and actions of each application App at present and perform data communication with a surface eFlinger; the method can also process the working event or the method call of the application App, and make corresponding response according to the requirement of the App.

The system library may include a plurality of functional modules. As shown in fig. 7, a rendering module may be included in the system library, and may be used to implement three-dimensional graphics drawing, image rendering, composition, layer processing, and the like. By way of example and not limitation, the rendering module includes, but is not limited to, at least one of: open graphics library (open graphics library, openGL), open source computer vision library (open source computer vision library, openCV), open operation language library (open computing language library, openCL).

The rendering module is provided with a plurality of image rendering functions that can be used to draw a scene from simple graphics to complex three dimensions. The rendering module provided by the system library in the embodiment of the application can be used for supporting the video editing application to execute image editing operation, such as video clipping operation, filter addition and the like.

As an example, other functional modules may also be included in the system library, such as, for example: status monitoring services, surface manager (surface manager), media library (Media Libraries), etc.

The hardware layer comprises a memory and a Graphics Processor (GPU). The memory is used for temporarily storing operation data in a central processing unit (central processing unit, CPU) and data exchanged with an external memory such as a hard disk. The memory also includes a shared memory (surface) and a video memory. The storage space required by the terminal device to run the video editing application may be provided by a shared memory.

A GPU is a processor that performs image and graphics-related operations. In the embodiment of the application, the process of video clipping by the terminal equipment by using the rendering module can be completed by the GPU.

The frame buffer (also called frame buffer) is a block of memory space that is reserved in memory by the GPU. The video memory is used for storing the rendering picture data after each GPU calculation.

The following describes the technical scheme of the present application and how the technical scheme of the present application solves the above technical problems in detail with specific embodiments. The following embodiments may be implemented independently or combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

The following first describes the internal implementation logic of the method for video cropping according to an embodiment of the present application with reference to the accompanying drawings.

Fig. 8 is a schematic diagram illustrating an internal implementation flow of a method for entering an edit page of a video editing application according to an embodiment of the present application. For example, the implementation of editing interface 300 of fig. 1 (c) is entered from display interface 200 of the video in fig. 1 (b). In fig. 8, the rendering module is OpenGL, and the shared memory management module is surfeflinger. As shown in fig. 8, the method includes the following flow.

S501, receiving operation information for starting a video editing application, which is sent by a user.

For example, referring to fig. 1 (b), the user may click a video editing button 213 in the video, and the terminal device opens the video editing application after acquiring the operation information of the user. Alternatively, the user may click on an icon of the video editing application in the display interface of the terminal device to enter the video editing application.

In the embodiment of the present application, the mode that the user clicks the screen to perform interaction is taken as an example to describe, in practice, the user may also perform interaction with the terminal device through other modes, for example, through modes of voice control or key input, etc., and the embodiment of the present application does not limit the interaction mode between the user and the terminal device.

It should be noted that the video editing application may include a plurality of functional modules therein. For example, an editing interface (or, alternatively, a main interface) module and a cropping interface module may be included. The editing interface module is responsible for the functions related to display, operation and the like of the editing interface, and the cutting interface module is responsible for the functions related to display, operation and the like of the cutting interface. Communication interaction can be carried out between the editing interface module and the clipping interface module.

S502. the video editing application invokes MediaCodec creation codec.

In particular, the video editing application may create an encoder and decoder for editing pages through MediaCodec and memory.

Illustratively, the internal flow of creating a codec of an embodiment of the present application is shown in FIG. 9. The operations performed by the video editing application in fig. 9 may be performed by an editing interface module. As shown in fig. 9, the flow includes the following.

S5021. the video editing application sends a video file path to MediaCodec.

For example, when a user clicks a video editing button in a video at a gallery application, the gallery application sends video material to be edited and a video file path to the video editing application. The video editing application, after receiving the video file path, may send the video file path to the MediaCodec. As an example, the video file path described above may be represented as: the ratio of sdcard/dcim/camera/123.mp4.

S5022.MediaCodec parses the video file path to obtain the video file format

The video file format indicates a compressed storage mode of video data, or indicates how video frame data is encoded and compressed into a video. By way of example, the video file formats described above may include, but are not limited to, the following: MPEG4, AVI, MKV, RMVB, etc. As a specific example, the parsed video file format may be MPEG4 format.

S5023.mediacodec returns the video file format to the video editing application.

Illustratively, the MediaCodec may send the parsed MPEG4 format to a video editing application.

S5024, the video editing application sends a request for applying the first shared memory to the memory.

The first shared memory may be used to store input data and output data of the codec. A shared memory may be used to store data corresponding to a display area. For example, as shown in fig. 1 (d), the clipping interface includes a plurality of display areas, and different display areas correspond to different functional modules in the clipping interface. For example, the display area includes: a display area 308 in the cropping interface for playing the video; a display area 311 where the cut-out scale button is located; a rotation button 301, a rotation number button 302, a mirror image button 303, and the like. The first shared memory may correspond to the display area 308 for playing video. In the present application, the display area 308 is also referred to as a first display area.

S5025, the memory sends information of successful application to the video editing application and carries the address of the first shared memory.

S5026, the video editing application sends a first instruction for indicating the creation of the codec to the MediaCodec, wherein the first instruction carries a video file format and an address of a first shared memory.

Optionally, the address of the first shared memory may be carried in other signaling, which is not limited in the embodiment of the present application.

S5027.mediacodec creates a codec from a video file format.

Among other things, a codec may be created from a video file format, which by way of example, is capable of identifying video encapsulation and compression formats and parsing video into video streams. MediaCodec can create a decoder from a video file format.

After creating the codec, the codec may store the associated input data and output data in the first shared memory, e.g., the editor may store the decoded data in the first shared memory.

S5028.mediacodec creation codec succeeds and returns information of creation success to the video editing application.

S503, initializing a rendering module by the video editing application.

Specifically, the video editing application may implement initialization of the rendering module by the rendering module and the GPU.

Illustratively, the internal flow of initialization of the rendering module of an embodiment of the present application is shown in FIG. 10. The operations performed by the video editing application in fig. 10 may be performed by an editing interface module. As shown in fig. 10, the flow includes the following.

S5031, the video editing application sends a command for initializing the environment of the rendering module to the rendering module.

S5032, the rendering module sends a request for applying a video memory to the GPU, wherein the video memory is used for receiving video pictures.

For example, the video editing application may send a request for applying for a video memory to the GPU through the rendering module, where the request may include the size of the video memory to be applied for, the color format of the video frame, and so on.

S5033, the GPU allocates the video memory and sends information of successfully applying for the video memory to the rendering module.

Optionally, after the memory application is successful, the rendering module may receive an index identification of the allocated memory from the GPU, which may be mapped to the address of the applied memory.

S5034, the rendering module sends information that the initialization of the rendering module is successful to the video editing application.

S504, the video editing application analyzes the video to obtain a video frame in a YUV format, and the video frame is stored in a first shared memory.

The YUV format refers to a storage format of a color space, which is simply called a color format. In practice, the video stream output by MediaCodec is typically in YUV format. Alternatively, in other fields, the color space may be stored in other formats, such as RGB format, HSV format, LUV format, and the like, which may be used to store color data. The embodiment of the application is illustrated by taking YUV format as an example.

During video parsing, the decoder may parse video data into a video stream in YUV format according to a video file format (for example, MPEG4 format or MKV format), and sample and intercept the video stream in YUV format according to a time point or a time range, so as to obtain a YUV video frame at a certain time point.

Specifically, the video editing application may implement video parsing through MediaCodec and memory.

Illustratively, an internal flow of a video parsing method of an embodiment of the present application is shown in fig. 11. The operations performed by the video editing application in fig. 11 may be performed by an editing interface module. As shown in fig. 11, the flow includes the following.

S5041, the video editing application sends a command for triggering video analysis to the MediaCodec.

S5042.MediaCodec analyzes the video to obtain a video frame in YUV format.

Specifically, mediaCodec parses video through a decoder, thereby obtaining video frames in YUV format.

S5043.mediacodec stores video frames in YUV format into the first shared memory.

S5044, the memory returns storage success information.

S5045.mediacodec sends information to the video editing application that the parsing of YUV video frames was successful.

S505, the video editing application performs rendering operation on the video frames.

In a specific example, the video editing application may implement rendering of video frames through MediaCodec, memory, application framework, rendering module, and GPU.

Illustratively, an internal flow of a video frame rendering method of an embodiment of the present application is shown in fig. 12. The operations performed by the video editing application in fig. 12 may be performed by an editing interface module. As shown in fig. 12, the flow includes the following.

S5051, the video editing application sends an instruction for rendering the video frame to the rendering module, wherein the instruction comprises clipping coordinates of a complete picture of the current video frame.

The clipping coordinates may refer to screen coordinates in a clipping interface. Specifically, the clipping coordinates may refer to coordinates of four vertices of a picture of the video frame in the clipping interface. Since the video frame is not cut at this time, the cut coordinates refer to coordinates of four vertices of the complete frame. For example, taking normalized coordinates as an example, the clipping coordinates described above may be expressed as: upper left (0.0, 1.0), upper right (1.0 ), lower left (0.0 ), lower right (1.0,0.0).

S5052, the rendering module sends a request for reading a video frame to the memory, wherein the color format of the video frame is YUV format.

S5053, the memory sends a video frame to the rendering module, wherein the color format of the video frame is YUV.

S5054, the rendering module converts the video frames in the YUV format into video frames in the RGB format.

Illustratively, after a video frame in YUV format is converted to a video frame in RGB format, the pixels on the video frame are normalized RGB floating point (float) data.

S5055, preparing drawing logic by the rendering module according to the clipping coordinates.

The rendering module modifies the drawing coordinates of the video frame on the video editing interface according to the clipping coordinates sent by the video editing interface.

S5056, the rendering module sends instructions for rendering calculation of video pictures to the GPU.

And S5057, performing rendering calculation of the video frame picture by the GPU.

As an example, the rendering calculations described above include, but are not limited to, at least one of: filters, stickers, special effects, text, etc.

S5058. the gpu sends information indicating that rendering is complete to the rendering module.

S5059. the rendering module instructs the GPU to convert the RGB video frames to YUV video frames.

S50510. the gpu performs conversion of RGB video frames into YUV video frames.

And S50511, the GPU sends information of successful video format conversion to the rendering module.

S50512. the rendering module sends a notification of completion of video frame rendering to the video editing application.

S506, the video editing application sends the video frames to a display screen for display (simply called sending display).

Optionally, after the video frame is rendered, the video editing application may call the GPU to draw the video frame and then send it to the display screen for display.

Specifically, the video editing application may implement a drawing process and a rendering process by calling an application framework, a memory, and SufaceFlinger, GPU.

Illustratively, the internal flow of the drawing and rendering process of video frames of an embodiment of the present application is shown in FIG. 13. The operations performed by the video editing application in fig. 13 may be performed by an editing interface module. As shown in fig. 13, the flow includes the following.

S5061, the video editing application sends a first instruction to the application framework, wherein the first instruction is used for indicating the application framework to initiate a video drawing process.

In other words, the video editing application invokes the application framework to initiate the drawing process of the video frames.

S5062, the application framework sends a sixth instruction to the GPU, wherein the sixth instruction is used for indicating the GPU to draw video frames.

In other words, the application framework invokes the GPU to initiate the drawing process of the video frame.

And S5063, the GPU performs drawing of the video frame to obtain second picture data corresponding to the video frame.

S5064, after the video frame is drawn, the GPU stores drawn second picture data into the first shared memory.

For example, referring to fig. 1 (d), the first shared memory corresponds to the display area 308 for playing video, and the first shared memory may be used to store picture data of the changed video preview area.

S5065, the GPU sends second picture data storage success information to the application framework to indicate the second picture data to store in the first shared memory.

S5066, the application framework sends second picture data storage success information to the video editing application.

S5067, the video editing application sends a third instruction to the application framework, wherein the third instruction is used for instructing the application framework to output the picture data in the shared memory corresponding to the first window view to the screen display.

The first window view is the whole window view corresponding to the clipping interface, and the shared memory corresponding to the first window view is used for storing the whole picture data in the editing interface. The shared memory corresponding to the first window view comprises the first shared memory.

S5068, the application framework sends a fourth instruction to the SurfaceFlinger, wherein the fourth instruction is used for notifying the SurfaceFlinger to output the picture data in the shared memory.

S5069, overlaying and mixing picture data in a shared memory corresponding to a plurality of window views into total picture data by SurfaceFlinger.

Wherein the plurality of window views may refer to a hierarchy of a plurality of applications currently displayed by the terminal device. Each window view corresponds to an application. The plurality of window views includes a first window view. For example, taking fig. 1 (d) as an example, the plurality of application programs displayed by the current terminal device include a desktop application, a gallery application, an editing interface in a video gallery application, and a cropping interface in a video gallery application. The windows corresponding to the clipping interface are located at the uppermost layer, the window views corresponding to the application programs are required to be overlapped and mixed, and finally a total picture is displayed.

S50610.Surface Flinger loads the superimposed total picture data into the buffer area of the screen.

Thus, the screen of the terminal device can present a video picture to the user. Illustratively, the screen driver may refresh the screen at a certain frequency and read the frames from the cached area of the screen, the user visually seeing the completed video frames.

Wherein, in S506 part, after receiving the operation information of the user clicking the button for entering the video editing, the screen of the terminal device may display an editing interface (i.e., a main interface) to the user. The editing interface may be the editing interface 300 shown in fig. 1 (c).

Upon receiving the operation information of the user clicking into the editing interface, a clip button, such as clip button 310 shown in fig. 1 (c), may be presented in the editing interface. The user may enter the crop interface 400 by clicking the crop button 310.

The above describes the process of the terminal device entering the editing interface, and in the process of entering the editing interface, the rendering module is required to perform rendering operation on the displayed picture, so that more computing resources are occupied. The embodiment of the application provides a scheme for video clipping, which can render video frames without using a rendering module in the process of presenting pictures of a clipping interface by terminal equipment, namely, does not trigger a rendering thread, reduces multithreaded time slice scheduling in the clipping interface, so that the operation experience of the clipping interface is smoother, and the phenomenon of delay blocking in the clipping interface is improved.

Next, a method for video cropping according to an embodiment of the present application will be described with reference to fig. 14 to 18. Wherein fig. 14 depicts a method for video cropping in accordance with an embodiment of the present application. Fig. 15 depicts a procedure in which the terminal device enters the clipping interface, and fig. 16 and 17 depict a procedure in which the terminal device performs a clipping operation in the clipping interface. Fig. 18 depicts a procedure in which the terminal device returns to the editing interface from the cropping interface.

Fig. 14 is a schematic diagram of a method for video cropping in accordance with an embodiment of the application. As shown in fig. 14, the method includes the following.

S601, detecting a first user operation, wherein the first user operation is used for triggering a cropping operation of a first video frame in a cropping interface of a video editing application.

Optionally, the cropping operation includes at least one of: rotation, movement, scaling, mirroring, picture cropping.

Alternatively, the first user operation in S601 may correspond to S801 part of fig. 16.

For example, the first user operation may include the clipping operation in fig. 1 (d) and may also include the clipping operations shown in fig. 2-4.

S602, responding to a first user operation, recording clipping parameters of a first display area corresponding to a first video frame, wherein the clipping parameters correspond to the first user operation.

It is understood that the clipping parameter is a parameter indicating a change in the first display area caused by the first user operation.

Optionally, the clipping parameters include at least one of: rotation angle, mirror image parameters, clipping coordinates.

Alternatively, the specific implementation of the S602 section may refer to the S8021 and S8022 sections in fig. 17.

S603, invoking the CPU to draw the video frame according to the clipping parameters so as to obtain first picture data of the clipped first video frame.

Alternatively, the specific implementation of the S603 section may refer to the S8023 to S8027 sections in fig. 17.

S604, displaying an updated clipping interface according to the first picture data, wherein the updated clipping interface comprises a clipped first video frame.

Alternatively, the specific implementation of the S604 section may refer to the S8028-S80211 sections in fig. 17.

In the embodiment of the application, the clipping interface does not use a rendering module to render the video frame in the process of responding to clipping operation of a user, namely does not trigger a rendering thread, but records and transmits clipping parameters when the application frame and the GPU are called to draw the video frame so as to update the picture of the clipped video frame. The method reduces multithread time slice scheduling when the clipping interface is displayed, so that the operation experience of the clipping interface is smoother, and the phenomenon of delay jamming in the clipping interface is improved.

Optionally, the method of fig. 14 further includes a process of executing an operation of editing an interface into the clipping interface, including: detecting a second user operation, wherein the second user operation is used for triggering the operation of entering the cropping interface from the editing interface of the video editing application; and responding to the second user operation, displaying the cropping interface, wherein the cropping interface comprises the first video frame, and in the process of responding to the second user operation, a rendering module is not used for rendering the first video frame.

Alternatively, the second user operation may refer to operation information in which the user clicks a button to enter the cropping interface in S701 of fig. 15.

As an example, the second user operation may refer to entering the crop interface 400 by clicking the crop button 310 shown in fig. 1 (c).

In the embodiment of the application, in the process of responding to the entering of the user to the clipping interface, the rendering module is not used for rendering the video frames, namely, the rendering thread is not triggered. The method reduces multithread time slice scheduling when the clipping interface is displayed, so that the operation experience of the clipping interface is smoother, and the phenomenon of delay jamming in the clipping interface is improved.

The specific process of entering the clipping interface from the editing interface can be described with reference to fig. 15.

Optionally, the method of fig. 14 further includes a process of returning to the editing interface from the cropping interface. For example, the terminal device may perform the steps of: detecting a third user operation, wherein the third user operation is used for indicating that the cutting operation of the cutting interface is completed; and responding to the third user operation, displaying an editing interface, wherein the editing interface comprises the first video frame after clipping.

The procedure in which the terminal device operates in the clipping interface may be described with reference to the related description in fig. 18.

FIG. 15 is a flow chart of a method for entering a clipping interface according to an embodiment of the application. As shown in fig. 15, the method includes the following.

S701, receiving operation information of clicking a button entering a cutting interface by a user.

For example, the above button for entering the clipping interface may refer to the clipping button 310 shown in fig. 1 (c), and the clipping interface 400 shown in fig. 1 (d) may be entered by clicking the clipping button 310.

Alternatively, the operation information of receiving the user clicking the button to enter the clipping interface in S701 corresponds to the second user operation above.

S702, an editing interface module of the video editing application sends clipping coordinates to a clipping interface module.

It should be understood that since the video frame does not start clipping at this time, clipping coordinates refer to coordinates of four vertices of the complete frame. For example, taking normalized coordinates as an example, the clipping coordinates described above may be expressed as: upper left (0.0, 1.0), upper right (1.0 ), lower left (0.0 ), lower right (1.0,0.0).

S703, a clipping interface module creates a codec.

The process of creating the codec by the clipping interface module is similar to S502 in fig. 7, and the specific process may refer to the method of creating the codec in fig. 9, where the difference is that the execution body is replaced by the editing interface module by the clipping interface module, which is not repeated herein for brevity.

S704, the clipping interface module performs video analysis.

The clipping interface module performs video analysis to obtain a video frame in a YUV format and stores the video frame in the first shared memory.

The first shared memory is a shared memory corresponding to the display area 308 (or the first display area) in fig. 1 (d).

The process of the clipping interface module for video parsing is similar to S504 in fig. 8, and the execution body is only required to be replaced by the editing interface module, and the specific process can refer to the method for video parsing in fig. 11, so that details are not repeated here for brevity.

S705, the clipping interface module sends the video frames to a display screen for display.

Alternatively, the video editing application may invoke the GPU to render the video frames and then send them to the display screen for display.

The process of drawing and displaying by the clipping interface module is similar to S506 in fig. 8, and the execution body is only required to be replaced by the editing interface module, and the specific process can refer to the method of drawing and displaying the video frame in fig. 13, which is not repeated here for brevity.

After the partial completion of S705, the terminal device may present the clipping interface to the user. The clipping interface may be, for example, the clipping interface 400 shown in fig. 1 (d).

Compared with the process of opening the editing interface in fig. 8, in the process of entering the clipping interface in fig. 15, the process of initializing the rendering module and using the rendering module to render the video frame is not performed, i.e. the rendering thread is not triggered, and the multi-thread time slice scheduling is reduced in the clipping interface, so that the operation experience of the clipping interface is smoother, and the phenomenon of delay blocking in the clipping interface is improved. In addition, rendering tools such as OpenGL and EGL are not introduced into the clipping interface, so that the information interacted between the editing interface and the clipping interface is reduced, the coupling degree of the editing interface and the clipping interface is reduced and the editing interface is independent, and the background management efficiency of video editing application is improved.

It should be noted that, in consideration of the clipping operation performed at the clipping interface, functions related to the rendering thread are not generally used, for example, functions of adding a filter, special effects, characters, a sticker, and the like. Therefore, omitting the rendering thread in the process of displaying the video frames on the clipping interface does not affect the correct display of the video frames on the clipping interface, and the delay blocking phenomenon in the clipping interface can be improved and the operation experience of a user can be improved due to the fact that the rendering thread is reduced.

FIG. 16 is a flow chart of a method for video cropping of a cropping interface according to an embodiment of the application. As shown in fig. 16, the method includes the following.

S801, operation information of cutting operation performed by a user on a cutting interface is received.

For example, the user may perform movements, zoom, rotation, mirroring, cropping, etc. For a specific example of the clipping operation, reference may be made to the content related to fig. 1 to 4, and for brevity, the description will not be repeated here.

Alternatively, the operation information of the user performing the cropping operation at the cropping interface in S801 may correspond to the first user operation in fig. 14.

S802, in the process of executing cutting operation by a user, the video frames are sent to a display screen for display.

In the process of executing the clipping operation by the user, the rendering module is not required to be called for rendering the video frames. The clipping interface module can present the video picture to the user only by completing the drawing and display process of the video frame.

In the cropping interface, a user performs cropping operations such as rotation, mirroring, scaling, movement, and the like, which may cause a change in cropping parameters associated with the video frame. The cropping parameter may refer to a parameter that varies from a display area in which the video frame is located as a result of a cropping operation by a user. For example, the clipping parameters described above include: rotation angle, mirror image parameters, clipping coordinates. In a specific example, when the video frame is rotated, mirrored, scaled or cut, the clipping coordinates of the video frame in the clipping frame will be changed, so that the change condition when the video frame is rotated, mirrored, scaled or cut can be reflected by the parameter of the clipping coordinates.

Thus, the cropping interface module may invoke the application framework, recording cropping parameters corresponding to the current video frame that vary during cropping, e.g., recording rotation angle, mirror parameters, cropping coordinates, etc. And in the subsequent process, drawing the video frame by using the recorded cutting parameters, and sending the video frame to a display screen for display.

For example, taking the record cropping coordinates as an example, assume that the user performs the top-half cropping of the video frame by dragging the cropping frame. The application framework may record clipping coordinates, which may be expressed as upper left (0.0, 1.0), upper right (1.0 ), lower left (0.0, 0.5), lower right (1.0, 0.5).

Illustratively, an internal flow of the drawing and sending process of video frames at a clipping interface in accordance with one embodiment of the application is shown in FIG. 17. Among them, fig. 17 differs from fig. 13 in that: in the sections S8021 to S8024, when the cropping interface module invokes the application frame, the cropping parameters of the first display area corresponding to the current video frame, which are changed in the cropping process, are recorded, and the recorded changed cropping parameters can be used for performing the subsequent drawing process of the video frame. It should be noted that, because the terminal device omits operation and rendering operation of the video frame in the process of opening the clipping interface, video frame drawing needs to be performed according to the varied clipping parameters in the process of drawing and sending and displaying, so that the video frame sent and displayed is the clipped video frame.

The operations performed by the video editing application in fig. 17 may be performed by a cropping interface module. As shown in fig. 17, the flow includes the following.

S8021, the video editing application sends a first instruction to an application framework, wherein the first instruction is used for indicating the application framework to initiate a video drawing process. The first instruction further includes a fifth instruction, where the fifth instruction is used to instruct the application framework to record the clipping parameters.

In other words, the cropping interface module invokes the application framework to draw to initiate the drawing process of the video frame.

S8022 the application framework records the changed cropping parameters of the current video frame.

The cropping parameter refers to a parameter of a first display area where a video frame is changed due to cropping operation of a user on a cropping interface. Such as rotation angle, mirror parameters, clipping coordinates, etc.

S8023, the application framework sends the second instruction to the GPU, the second instruction is used for indicating the GPU to draw video frames, and the second instruction comprises the clipping parameters.

In other words, the application framework may invoke the GPU to initiate drawing of the video frame and send clipping parameters to the GPU, the clipping parameters including one or more of rotation angle, mirror parameters, clipping coordinate parameters. The GPU can draw the video frames according to the clipping parameters.

In some examples, the clipping parameters may also be carried in other types of information, as long as the conditions for sending the clipping parameters to the GPU are met.

And S8024, the GPU draws the video frame according to the clipping parameters to obtain first picture data corresponding to the video frame.

And S8025, the GPU finishes drawing of the video frame and stores drawn first picture data into a first shared memory.

Referring to fig. 1 (e), the first shared memory corresponds to a display area 308 for playing video, and the first shared memory may be used to store the picture data of the changed video preview area.

S8026, the GPU sends first picture data storage success information to the application framework to indicate that the first picture data is successfully stored in the first shared memory.

S8027. the application framework sends the first picture data storage success information to the video editing application.

S8028, the video editing application sends a third instruction to the application frame, wherein the third instruction is used for instructing the application frame to output the picture data in the shared memory corresponding to the first window view to the screen display.

S8029, the application framework sends a fourth instruction to the SurfaceFlinger, wherein the fourth instruction is used for notifying the SurfaceFlinger to output the picture data in the shared memory.

S80210.SurfaceFlinger superimposes and mixes the picture data in the shared memory corresponding to the window views into a total picture data.

Wherein the plurality of window views may refer to a hierarchy of a plurality of applications currently displayed by the terminal device. Each window view corresponds to an application. The plurality of window views includes a first window view. For example, taking fig. 1 (e) as an example, the plurality of application programs displayed by the current terminal device include a desktop application, a gallery application, an editing interface in a video gallery application, and a cropping interface in a video gallery application. The windows corresponding to the clipping interface are located at the uppermost layer, the window views corresponding to the application programs are required to be overlapped and mixed, and finally a total picture is displayed.

S80211.Surface Flinger loads the total picture data after superposition into the buffer area of the screen.

S803, receiving operation information indicating that cutting is completed by a user.

For example, the cut-out button may refer to button 304 in FIG. 1 (d). After the user clicks the crop complete button 304, the video editing application may return from the crop interface to the editing interface. The internal flow in returning to the editing interface may refer to the relevant content in fig. 18.

Compared with the process of opening the editing interface in fig. 8, in the process of clipping the interface, the method in fig. 16 and 17 does not use the rendering module to render the video frame, i.e. does not trigger the rendering thread, but records and transmits clipping parameters when the application frame and the GPU are invoked to draw the video frame, so as to update the picture of the clipped video frame. The multi-thread time slice scheduling is reduced at the clipping interface, so that the operation experience of the clipping interface is smoother, and the phenomenon of delay blocking in the clipping interface is improved. In addition, rendering tools such as OpenGL and EGL are not introduced into the clipping interface, so that the information interacted between the editing interface and the clipping interface is reduced, the coupling degree of the editing interface and the clipping interface is reduced and the editing interface is independent, and the background management efficiency of video editing application is improved.

FIG. 18 is a flow chart of a method for clipping an interface back to an editing interface according to an embodiment of the application. As shown in fig. 18, the method includes the following.

And S901, receiving operation information of a button for indicating cutting completion by a user.

For example, the cut-out button described above may refer to button 304 in fig. 1 (f). At this point, the video editing application is in the cropping interface.

Alternatively, the operation information of the button, which the user in S901 indicates that clipping is completed, may correspond to the third user operation above.

S902, a clipping interface module in the video editing application sends clipping coordinates to an editing interface module.

For example, assume that the user performs picture cropping of the upper half of the video picture by dragging the crop box. The clipping coordinates at this time can be expressed as: upper left (0.0, 1.0), upper right (1.0 ), lower left (0.0, 0.5), lower right (1.0, 0.5).

By way of example, FIG. 19 is a schematic diagram of a clipping interface in accordance with an embodiment of the application. Fig. 19 (a) shows a clipping interface before clipping, and fig. 19 (b) shows a clipping interface after clipping. Fig. 19 (c) shows normalized coordinates of the clipping interface before clipping, and fig. 19 (d) shows normalized coordinates of the clipping interface return editing interface after clipping.

As shown in (a) of fig. 19, the outer frame represents specific coordinates of the mobile phone screen. At this time, as an example, the pixel coordinates of the four corners of the mobile phone screen may be sequentially expressed as an upper left corner coordinate (0, 2000), an upper right corner coordinate (2000 ), a lower left corner coordinate (0, 0), and a lower right corner coordinate (2000,0). The inner frame represents the video picture in the cropping interface. Since the cropping operation has not yet been started, the cropping frame (bold line) and the video frame responsible for cropping are coincident. At this time, the pixel coordinates of the crop frame or the video picture can be expressed as: upper left (100, 1600), upper right (1900, 1600), lower left (100, 400), lower right (1900, 400). Referring to fig. 19 (c), the normalized coordinates of the crop box at this time are expressed as: upper left (0.0, 1.0), upper right (1.0 ), lower left (0.0 ), lower right (1.0,0.0).

With continued reference to fig. 19 (b), the user modifies the range of the crop frame to perform the upper half of the picture cropping, at which time the pixel coordinates of the crop frame may be expressed as: upper left (100, 1600), upper right (1900, 1600), lower left (100, 1000), lower right (1900, 1000). Referring to fig. 19 (d), the cropping interface may return normalized coordinates of the cropping frame to the editing interface, which may be expressed as: upper left (0.0, 1.0), upper right (1.0 ), lower left (0.0, 0.5), lower right (1.0, 0.5). The editing interface may generate a video frame from the returned crop frame coordinates.

S903, the editing interface module analyzes the video.

The process of video parsing by the editing interface module is similar to S504 in fig. 8, and the specific process may refer to the method of video parsing in fig. 11, which is not repeated here for brevity.

S904, the editing interface module performs video frame rendering.

The video frame rendering process performed by the editing interface module is similar to S505 in fig. 8, and the specific process may refer to the video frame rendering method in fig. 12, and for brevity, will not be described herein.

In the process of video frame rendering, video rendering is required according to clipping coordinates. For example, in section S5051 in fig. 12, the clipping coordinates may be changed to upper left corner coordinates (0.0, 1.0), upper right corner coordinates (1.0 ), lower left corner coordinates (0.0, 0.5), lower right corner coordinates (1.0, 0.5). And in section S5055 in fig. 12, the rendering module needs to prepare drawing logic according to the changed clipping coordinates.

S905, sending the video frames to a display screen for display by the editing interface module.

Optionally, after the video frame is rendered, the editing interface module may call the GPU to draw the video frame and then send the video frame to the display screen for display.

The process of drawing and displaying by the editing interface module is similar to S506 in fig. 8, and the specific process may refer to the method of drawing and displaying the video frame in fig. 13, which is not repeated herein for brevity.

After the completion of the S905 section, the terminal device presents an editing interface to the user. The editing interface displays the cut video picture. For example, the editing interface may be the editing interface 300 in fig. 1 (c).

As shown in fig. 18, during the return from the cropping interface to the editing interface, the video frames may be rendered using a rendering module to obtain the cropped video.

Fig. 20 is a schematic diagram of a method for video cropping in accordance with yet another embodiment of the application. As shown in fig. 20, the method includes the following.

S1101. detecting a first user operation, where the first user operation is used to trigger a cropping operation of a first video frame in a cropping interface of a video editing application.

Alternatively, the first user operation in S1101 may correspond to S801 of fig. 16.

S1102, responding to a first user operation, displaying an updated clipping interface, wherein the updated clipping interface comprises a clipped first video frame, and a rendering module is not used for rendering the first video frame in the process of responding to the first user operation.

Therefore, the clipping interface does not use the rendering module to render the video frame in the process of responding to the clipping operation of the user, namely does not trigger the rendering thread, so that the multi-thread time slice scheduling is reduced when the clipping interface is displayed, the operation experience of the clipping interface is smoother, and the phenomenon of delay blocking in the clipping interface is improved.

In some examples, in response to a first user operation, displaying an updated clipping interface includes: calling an application frame to record a cutting parameter of a first display area corresponding to the first video frame, wherein the cutting parameter is used for indicating a parameter of the first display area which is changed due to the first user operation; invoking an image processing unit GPU to draw the video frame according to the clipping parameters so as to obtain first picture data of a clipped first video frame; and displaying the updated clipping interface according to the first picture data.

Further, the specific implementation of displaying the updated clipping interface according to the first user operation may refer to the relevant content in fig. 16 and 17, and for brevity, will not be described herein.

Optionally, before detecting the first user operation, the method of fig. 20 further includes: detecting a second user operation, wherein the second user operation is used for triggering the operation of entering a cropping interface from an editing interface of the video editing application; and responding to the second user operation, displaying a clipping interface, wherein the clipping interface comprises a first video frame, and in the process of responding to the second user operation, the rendering module is not used for rendering the first video frame.

Further, the specific implementation of entering the clipping interface according to the second user operation may refer to the relevant content in fig. 15, and for brevity, will not be described herein.

Therefore, in the process of entering the clipping interface, the rendering module is not initialized and the process of rendering the video frames by using the rendering module is not triggered, namely, the multi-thread time slice scheduling is reduced in the clipping interface, so that the operation experience of the clipping interface is smoother, and the phenomenon of delay blocking in the clipping interface is improved.

The method provided by the embodiment of the present application is described above with reference to fig. 1 to 20, and the device for performing the method provided by the embodiment of the present application is described below.

In order to better understand the embodiments of the present application, the following describes the structure of the terminal device according to the embodiments of the present application:

fig. 21 shows a hardware configuration diagram of the terminal device 100. The terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a sensor module 180, keys 190, a motor 151, an indicator 192, a camera 193, and a display 194. Wherein the sensor module 180 may include a pressure sensor 180A, a touch sensor 180K, etc.

It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the terminal device 100. In other embodiments of the application, terminal device 100 may include more or less components than illustrated, or certain components may be combined, or certain components may be split, or different arrangements of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units, such as: the processor 110 may include an application processor (application processor, AP), a graphics processor (graphics processingunit, GPU), an image signal processor (image signal processor, ISP), a video codec, etc. Wherein the different processing units may be separate devices or may be integrated in one or more processors.

It should be understood that the interfacing relationship between the modules illustrated in the embodiment of the present application is illustrated schematically, and does not constitute a structural limitation of the terminal device 100. In other embodiments of the present application, the terminal device 100 may also use different interfacing manners, or a combination of multiple interfacing manners in the foregoing embodiments.

The terminal device 100 implements display functions through a GPU, a display screen 194, an application processor, and the like. The GPU is a microprocessor for image processing, and is connected to the display 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

The display screen 194 is used for displaying images, displaying videos, receiving sliding operations, and the like. The display 194 includes a display panel. The display panel may employ a liquid crystal display (liquid crystal display, LCD), an organic light-emitting diode (OLED), an active-matrixorganic light emitting diod (AMOLED), a flexible light-emitting diode (flex), a mini, a Micro-OLED, a quantum dot light-emitting diode (quantum dot lightemitting diodes, QLED), or the like. In some embodiments, the terminal device 100 may include 1 or N display screens 194, N being a positive integer greater than 1.

The terminal device 100 may implement a photographing function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, an application processor, and the like.

The camera 193 is used to capture still images or video. The object generates an optical image through the lens and projects the optical image onto the photosensitive element. The photosensitive element may be a charge coupled device (charge coupled device, CCD) or a Complementary Metal Oxide Semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then transferred to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, or the like format. In some embodiments, the terminal device 100 may include 1 or N cameras 193, N being a positive integer greater than 1.

Video codecs are used to compress or decompress digital video. The terminal device 100 may support one or more video codecs. In this way, the terminal device 100 can play or record video in various encoding formats, for example: dynamic picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

The external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to realize expansion of the memory capability of the terminal device 100. The external memory card communicates with the processor 110 via an external memory interface 116 to implement data storage functions. For example, files such as music, video, etc. are stored in an external memory card.

The internal memory 121 may be used to store computer-executable program code that includes instructions. The internal memory 121 may include a storage program area and a storage data area. The storage program area may store an application program (such as a sound playing function, an image playing function, etc.) required for at least one function of the operating system, etc. The storage data area may store data (such as audio data, phonebook, etc.) created during use of the terminal device 100, and the like.

The pressure sensor 180A is used to sense a pressure signal, and may convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A may be disposed on the display screen 194.

The touch sensor 180K, also referred to as a "touch device". The touch sensor 180K may be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, which is also called a "touch screen". The touch sensor 180K is for detecting a touch operation acting thereon or thereabout. The touch sensor may communicate the detected touch operation to the application processor to determine the touch event type. Visual output related to touch operations may be provided through the display 194. In other embodiments, the touch sensor 180K may also be disposed on the surface of the terminal device 100 at a different location than the display 194.

The keys 190 include a power-on key, a volume key, etc. The keys 190 may be mechanical keys. Or may be a touch key. The terminal device 100 may receive key inputs, generating key signal inputs related to user settings and function controls of the terminal device 100.

The motor 191 may generate a vibration cue. The motor 191 may be used for incoming call vibration alerting as well as for touch vibration feedback. For example, touch operations acting on different applications (e.g., photographing, audio playing, etc.) may correspond to different vibration feedback effects. Touch operations on different areas of the display screen 194,

fig. 22 is a schematic structural diagram of an apparatus for video cropping according to an embodiment of the present application, where the apparatus for video cropping may be a terminal device in an embodiment of the present application, or may be a chip or a chip system in the terminal device.

As shown in fig. 22, an apparatus 1500 for video cropping may be used in a communication device, circuit, hardware component, or chip, the apparatus for video cropping comprising: a display unit 1501, and a processing unit 1502. Wherein the display unit 1501 is used for supporting the steps of display performed by the apparatus 1500 for video cropping; the processing unit 1502 is used for supporting the steps of performing information processing by the apparatus 1500 for video cropping.

In a possible embodiment, the apparatus for video cropping may further include: and a memory cell 1503. The processing unit 1502 and the storage unit 1503 are connected by a line. The memory unit 1503 may include one or more memories, which may be one or more devices, circuits, or means for storing programs or data. The storage unit 1503 may exist independently and is connected to the processing unit 1502 provided in the apparatus for video cropping through a communication line. The memory unit 1503 may also be integrated with the processing unit 1502.

The storage unit 1503 may store computer-executed instructions of the method in the terminal device to cause the processing unit 1502 to execute the method in the above-described embodiment. The storage unit 1503 may be a register, a cache, a RAM, or the like, and the storage unit 1503 may be integrated with the processing unit 1502. The memory unit 1503 may be a read-only memory (ROM) or other type of static storage device that may store static information and instructions, and the memory unit 1503 may be independent of the processing unit 1502.

Fig. 23 is a schematic hardware structure of another terminal device according to an embodiment of the present application, as shown in fig. 23, where the terminal device includes a processor 1601, a communication line 1604 and at least one communication interface (illustrated in fig. 23 by taking a communication interface 1603 as an example).

The processor 1601 may be a general purpose central processing unit (central processing unit, CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present application.

Communication line 1604 may include circuitry for communicating information between the components described above.

Communication interface 1603, using any transceiver-like device, is used to communicate with other devices or communication networks, such as ethernet, wireless local area network (wireless local area networks, WLAN), etc.

Possibly, the terminal device may also comprise a memory 1602.

Memory 1602 may be, but is not limited to, read-only memory (ROM) or other type of static storage device that can store static information and instructions, random access memory (random access memory, RAM) or other type of dynamic storage device that can store information and instructions, as well as electrically erasable programmable read-only memory (EEPROM), compact disc-read only memory (compact disc read-only memory) or other optical disk storage, optical disk storage (including compact disc, laser disc, optical disc, digital versatile disc, blu-ray disc, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. The memory may be self-contained and coupled to the processor via communication line 1604. The memory may also be integrated with the processor.

The memory 1602 is used for storing computer-executable instructions for executing aspects of the present application, and is controlled by the processor 1601. The processor 1601 is configured to execute computer-executable instructions stored in the memory 1602 to implement a method for video cropping provided by an embodiment of the present application.

Possibly, the computer-executable instructions in the embodiments of the present application may also be referred to as application program codes, which are not limited in particular.

In a particular implementation, as one embodiment, the processor 1601 may include one or more CPUs, such as CPU0 and CPU1 in fig. 23.

In a specific implementation, as an embodiment, the terminal device may include a plurality of processors, such as the processor 1601 and the processor 1605 in fig. 23. Each of these processors may be a single-core (single-CPU) processor or may be a multi-core (multi-CPU) processor. A processor herein may refer to one or more devices, circuits, and/or processing cores for processing data (e.g., computer program instructions).

In the above embodiments, the instructions stored by the memory for execution by the processor may be implemented in the form of a computer program product. The computer program product may be written in the memory in advance, or may be downloaded in the form of software and installed in the memory.

The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions in accordance with embodiments of the present application are produced in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wired (e.g., coaxial cable, fiber optic, digital subscriber line (digital subscriber line, DSL), or wireless (e.g., infrared, wireless, microwave, etc.), or semiconductor medium (e.g., solid state disk, SSD)) or the like.

The embodiment of the application also provides a computer readable storage medium. The methods described in the above embodiments may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. Computer readable media can include computer storage media and communication media and can include any medium that can transfer a computer program from one place to another. The storage media may be any target media that is accessible by a computer.

As one possible design, the computer-readable medium may include compact disk read-only memory (CD-ROM), RAM, ROM, EEPROM, or other optical disk memory; the computer readable medium may include disk storage or other disk storage devices. Moreover, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, includes Compact Disc (CD), laser disc, optical disc, digital versatile disc (digital versatile disc, DVD), floppy disk and blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers.

Combinations of the above should also be included within the scope of computer-readable media. The foregoing is merely illustrative embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily think about variations or substitutions within the technical scope of the present invention, and the invention should be covered. Therefore, the protection scope of the invention is subject to the protection scope of the claims.

Claims

1. A method for video cropping, characterized in that it is applied to a terminal device, said method comprising:

detecting a first user operation for triggering a cropping operation of a first video frame in a cropping interface of a video editing application;

responding to the first user operation, rendering the first video frame without using a rendering module, and recording a clipping parameter of a first display area corresponding to the first video frame, wherein the clipping parameter corresponds to the first user operation;

invoking an image processing unit GPU to draw the video frame according to the clipping parameters so as to obtain first picture data of a clipped first video frame;

and displaying an updated clipping interface according to the first picture data, wherein the updated clipping interface comprises the clipped first video frame.

2. The method of claim 1, wherein the clipping operation comprises at least one of: rotation, movement, scaling, mirroring, picture cropping.

3. The method of claim 1 or 2, wherein the clipping parameters include at least one of: rotation angle, mirror image parameters, clipping coordinates.

4. The method of claim 3, wherein the recording clipping parameters of the first display area corresponding to the first video frame in response to the first user operation comprises:

the video editing application sends a first instruction to the application framework, wherein the first instruction is used for instructing the application framework to initiate a video drawing process, the first instruction comprises a fifth instruction, and the fifth instruction is used for instructing the application framework to record the clipping parameters;

the application framework records the clipping parameters.

5. The method of claim 4, wherein the invoking the GPU to render the video frame according to the clipping parameters comprises:

the GPU receives a second instruction sent by the application framework, the second instruction is used for indicating the GPU to draw video frames, and the second instruction comprises the clipping parameters;

The GPU performs video frame drawing according to the clipping parameters to obtain the first picture data;

and the GPU stores the first picture data into a first shared memory corresponding to the first display area.

6. The method of claim 5, wherein displaying a cropping interface from the first screen data comprises:

the video editing application sends a third instruction to the application framework, wherein the third instruction is used for instructing the application framework to output the picture data in the shared memory corresponding to the first window view to the screen display, the first window view is the window view corresponding to the clipping interface, and the shared memory corresponding to the first window view comprises the first shared memory;

the application framework sends a fourth instruction to the surfeFlinger, wherein the fourth instruction is used for notifying the surfeFlinger to output the picture data in the shared memory;

the SurfaceFlinger superimposes and mixes the picture data in the shared memory corresponding to a plurality of window views into total picture data, wherein the plurality of window views comprise the first window view;

and the SurfaceFlinger loads the total picture data into a cache area of a screen so as to display the clipping interface.

7. The method of any of claims 4 to 6, wherein prior to the detecting the first user operation, the method further comprises:

detecting a second user operation, wherein the second user operation is used for triggering the operation of entering the cropping interface from the editing interface of the video editing application;

and responding to the second user operation, displaying the cropping interface, wherein the cropping interface comprises the first video frame, and in the process of responding to the second user operation, a rendering module is not used for rendering the first video frame.

8. The method of claim 7, wherein the displaying the clipping interface in response to the second user operation comprises:

calling a MediaCodec to create a codec;

analyzing the video data corresponding to the first video frame according to the coder-decoder to obtain a first video frame in YUV format;

drawing according to the first video frame in the YUV format to obtain second picture data corresponding to the first video frame;

and displaying the clipping interface according to the second picture data.

9. The method of claim 8, wherein the invoking MediaCodec creates an encoder and a decoder, comprising:

The video editing application sends a video file path corresponding to the first video frame to the MediaCodec;

the MediaCodec parses the video file path to obtain the video file format;

the MediaCodec transmitting the video file format to the video editing application;

the video editing application sends a seventh instruction for creating a codec to the MediaCodec, wherein the seventh instruction comprises the video file format;

the MediaCodec creates a codec from the video file format.

10. The method of claim 9, wherein the method further comprises:

the video editing application applies for a first shared memory corresponding to the first display area to an internal memory, wherein the first shared memory is used for storing input data or output data of the coder-decoder;

the memory sends information of successful creation of the shared memory to the video editing application, wherein the information of successful creation of the shared memory comprises an address of the first shared memory;

the video editing application sends the address of the first shared memory to the MediaCodec.

11. The method according to any one of claims 8 to 10, wherein parsing the video data corresponding to the first video frame according to the codec to obtain a first video frame in YUV format comprises:

The video editing application sends a command for analyzing video to the MediaCodec;

the MediaCodec analyzes the video data corresponding to the first video frame through the coder-decoder to obtain a first video frame in YUV format;

the MediaCodec stores the first video frame in the YUV format into a first shared memory corresponding to the first display area;

the MediaCodec sends information of successful video parsing to the video editing application.

12. The method of claim 11, wherein the drawing according to the first video frame in YUV format to obtain the second picture data corresponding to the first video frame comprises:

the video editing application sends a first instruction to the application framework, wherein the first instruction is used for indicating the application framework to initiate a video drawing process;

the application framework sends a sixth instruction to the GPU, wherein the sixth instruction is used for instructing the GPU to draw video frames;

the GPU performs drawing of video frames to obtain second picture data;

and the GPU stores the second picture data into a first shared memory corresponding to the first display area.

13. The method of claim 12, wherein displaying the cropping interface according to the second screen data comprises:

14. The method of claim 12 or 13, wherein after the detecting the first user operation, the method further comprises:

detecting a third user operation, wherein the third user operation is used for indicating that the cutting operation of the cutting interface is completed;

and responding to the third user operation, displaying an editing interface, wherein the editing interface comprises the first video frame after clipping.

15. A method for video cropping, characterized in that it is applied to a terminal device, said method comprising:

and responding to the first user operation, displaying an updated clipping interface, wherein the updated clipping interface comprises the clipped first video frame, and a rendering module is not used for rendering the first video frame in the process of responding to the first user operation.

16. The method of claim 15, wherein the displaying an updated clipping interface in response to the first user operation comprises:

calling an application framework to record a clipping parameter of a first display area corresponding to the first video frame, wherein the clipping parameter is used for indicating a parameter of the first display area which is changed due to the first user operation;

and displaying the updated clipping interface according to the first picture data.

17. The method of claim 15 or 16, wherein prior to the detecting the first user operation, the method further comprises:

18. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, causes the electronic device to perform the method of any one of claims 1-14 or to perform the method of any one of claims 15-17.

19. A computer readable storage medium storing a computer program, which when executed by a processor causes the computer to perform the method of any one of claims 1-14 or to perform the method of any one of claims 15-17.