CN114598685A

CN114598685A - Multimedia data processing method and device, electronic equipment and storage medium

Info

Publication number: CN114598685A
Application number: CN202210145149.3A
Authority: CN
Inventors: 黄龙飞; 郭健锋
Original assignee: Alibaba China Co Ltd
Current assignee: Alibaba China Co Ltd
Priority date: 2022-02-17
Filing date: 2022-02-17
Publication date: 2022-06-07

Abstract

The embodiment of the application provides a multimedia data processing method, a multimedia data processing device, electronic equipment and a storage medium, wherein the method comprises the following steps: receiving a data editing request of terminal equipment; creating an instruction channel service and a data channel service according to the data editing request; receiving a data instruction through the instruction channel service, wherein the data instruction is generated according to the operation received by an editing page on the terminal equipment; editing the multimedia data according to the data instruction, and generating a data stream by adopting the processed multimedia data; transmitting the data stream to a terminal device through a data channel service to output the data stream in the terminal device to view processed multimedia data; the method of the embodiment of the application can be used for more conveniently clipping the multimedia data.

Description

Multimedia data processing method and device, electronic equipment and storage medium

Technical Field

The present application relates to the field of computer technologies, and in particular, to a method for processing multimedia data, an apparatus for processing multimedia data, an electronic device, and a storage medium.

Background

Multimedia (Multimedia) is a combination of various media, generally including text, sound, and image, and various media forms, for example, Multimedia data may be video data, and video data may be composed of image data and audio data.

In the process of editing multimedia data, the multimedia data is generally edited in the local client, for example, the multimedia data to be processed may be transmitted to the local client, and according to the editing operation of user, such as cutting, fusing, etc., the video data is cut into segments, and different segments are fused (or concatenated).

However, when the data editing process is performed at the local client, the computational force requirement on the local client is high, the multimedia data editing operation is difficult, and the multimedia data editing is very inconvenient.

Disclosure of Invention

The embodiment of the application provides a multimedia data processing method, so that the clipping processing of the multimedia data is facilitated.

Correspondingly, the embodiment of the application also provides a multimedia data processing device, an electronic device and a storage medium, which are used for ensuring the realization and application of the system.

In order to solve the above problem, an embodiment of the present application discloses a method for processing multimedia data, where the method includes: receiving a data editing request of terminal equipment; creating an instruction channel service and a data channel service according to the data editing request; receiving a data instruction through the instruction channel service, wherein the data instruction is generated according to the operation received by an editing page on the terminal equipment; editing the multimedia data according to the data instruction, and generating a data stream by adopting the processed multimedia data; and transmitting the data stream to the terminal equipment through a data channel service so as to output the data stream in the terminal equipment to view the processed multimedia data.

Optionally, the data instruction comprises a view instruction; the receiving data instructions over the instruction channel service includes: receiving a viewing instruction sent by terminal equipment through an instruction channel service, wherein the viewing instruction is used for viewing multimedia data for editing; the editing processing is performed on the multimedia data according to the data instruction, and a data stream is generated by using the processed multimedia data, including: and determining preview key data of the multimedia data according to the viewing instruction, and sending the preview key data to the terminal equipment through a data channel service so as to preview the multimedia data at the terminal equipment.

Optionally, the data instruction includes an editing instruction; the receiving data instructions over the instruction channel service includes: receiving an editing instruction sent by terminal equipment through an instruction channel service; the editing processing is performed on the multimedia data according to the data instruction, and a data stream is generated by using the processed multimedia data, including: and editing the multimedia data according to the editing instruction, and generating a target data stream by adopting the edited multimedia data.

Optionally, the method further includes: providing page data of an edit page to a terminal device through a data channel service to display the edit page on the terminal device, the edit page including: a first display area, a second display area and a clipping operation area; sending preview information and time information of multimedia data to terminal equipment through a data channel service so as to display the preview information in a first display area of the terminal equipment and display a data time axis corresponding to the time information in a clipping operation area; the receiving of the viewing instruction sent by the terminal device through the instruction channel service includes: and receiving a viewing instruction of the multimedia data sent by the terminal equipment through an instruction channel service, wherein the second display area is used for displaying preview key data corresponding to the viewing instruction, and the viewing instruction is determined based on the operation of the data time axis.

Optionally, the determining the preview key data of the multimedia data according to the viewing instruction includes: determining the time axis starting time of the target time axis according to the target time axis information in the viewing instruction; determining a plurality of groups of preview time periods corresponding to different time axis scaling ratios according to the time axis starting time; and determining display key data corresponding to the preview time period in the multimedia data to determine preview key data.

Optionally, the editing the multimedia data according to the editing instruction includes at least one of the following steps: determining corresponding cutting start-stop time according to the segment cutting instruction, and cutting a corresponding target segment from the multimedia data; carrying out zooming processing on a target segment of the multimedia data according to the segment zooming instruction; adjusting the segment attribute of a target segment of multimedia data according to a segment attribute setting instruction, wherein the multimedia data comprises video data, and the segment attribute comprises video size, video position, video definition and video time; and fusing the first segment and the second segment of the multimedia data according to the segment fusion instruction.

In order to solve the above problem, an embodiment of the present application discloses a method for processing multimedia data, where the method includes: sending a data editing request to a server so that the server can establish instruction channel service and data channel service according to the data editing request; responding to an operation on an editing page, and generating a corresponding data instruction, wherein the editing page is used for receiving the operation aiming at the multimedia data; sending the data instruction to a server through an instruction channel service so that the server processes multimedia data according to the data instruction and generates a data stream according to the processed multimedia data; receiving a data stream fed back by the server through a data channel service; and analyzing the data stream, and outputting the multimedia data on the editing page.

Optionally, the generating the data instruction in response to the editing operation on the editing page includes at least one of the following steps: generating a clip clipping instruction based on the triggering of the data time axis and the clip clipping control; generating a fragment zooming instruction based on the triggering of the data time axis and the fragment zooming control; generating a fragment attribute setting instruction based on triggering of a data time axis and a fragment attribute configuration control; and generating a fragment fusion instruction based on the triggering of the fragment fusion control.

In order to solve the above problem, an embodiment of the present application discloses an electronic device, including: a processor; and a memory having stored thereon executable code which, when executed by the processor, performs the method as in any one of the above embodiments.

To solve the above problem, the embodiments of the present application disclose one or more machine-readable media having executable code stored thereon, which when executed by a processor, performs the method as described in any of the above embodiments.

Compared with the prior art, the embodiment of the application has the following advantages:

the embodiment of the application can be applied to a server, the server can receive a data editing request of terminal equipment and respond to the data editing request to create instruction channel service and data channel service, the terminal equipment can transmit a data instruction to the server through the instruction channel service, and the server can transmit a data stream to the terminal equipment in real time through the data channel service. In the embodiment of the application, the server can receive the data instruction sent by the terminal device through the instruction channel service and edit the multimedia data in the server to form a real-time data stream, so that the data stream is sent to the terminal device through the data channel service to view the processed multimedia data in the terminal device. According to the embodiment of the application, the data instruction can be transmitted through the instruction channel service, and the data stream can be transmitted through the data channel service, so that the data editing process of the multimedia data can be completed at the server side, the calculation requirement on the terminal equipment is reduced, and the multimedia data can be conveniently edited. In addition, compared with the scheme of downloading the multimedia data to the terminal for editing and fusing, the embodiment of the application can reduce the downloading and sending processes of the multimedia data, can reduce the consumed flow, and does not need to occupy a large amount of storage space of the terminal equipment.

Drawings

Fig. 1 is a flowchart illustrating a method for processing multimedia data according to an embodiment of the present application;

fig. 2 is a flowchart illustrating a multimedia data processing method according to another embodiment of the present application;

fig. 3 is a flowchart illustrating a multimedia data processing method according to still another embodiment of the present application;

fig. 4 is a flowchart illustrating a multimedia data processing method according to still another embodiment of the present application;

fig. 5 is a schematic structural diagram of a multimedia data processing apparatus according to an embodiment of the present application;

fig. 6 is a schematic structural diagram of a multimedia data processing apparatus according to another embodiment of the present application;

fig. 7 is a schematic structural diagram of an exemplary apparatus provided in an embodiment of the present application.

Detailed Description

In order to make the aforementioned objects, features and advantages of the present application more comprehensible, the present application is described in further detail with reference to the accompanying drawings and the detailed description.

The embodiment of the application can be applied to the field of processing of Multimedia data, Multimedia (Multimedia) is the integration of multiple media, generally including multiple media forms such as text, sound and image, for example, the Multimedia data can be video data, and the video data can be composed of image data and audio data.

As shown in fig. 1, in the embodiment of the present application, a data processing process of multimedia data may be completed at a server, the server may establish a connection with a terminal device to receive an instruction of the terminal device, transmit data to be processed and a rendering result obtained after rendering according to the instruction to the terminal device, a user may perform an operation on the terminal device on the multimedia data and send the data instruction to the server, the server performs processing such as clipping and merging on the multimedia data according to the data instruction, and generates a target data stream by using edited multimedia data (rendering result), so as to send the target data stream to the terminal device. The embodiment of the application can finish the data editing process of the multimedia data at the server side, reduces the calculation requirement on the terminal equipment, and can facilitate the editing processing of the multimedia data. And, some schemes are to download the multimedia data to the terminal equipment for editing, and transmit the edited data to the cloud server, compared with the scheme, the embodiment of the application can reduce the downloading and sending processes of the multimedia data, reduce the consumed flow, and do not need to occupy a large amount of storage space of the terminal equipment.

Specifically, the embodiment of the application can be applied to a server, the server can interact with a terminal device, and the server can establish two connection services with the terminal device: the method includes receiving a data instruction sent by a terminal device through the instruction channel service, and feeding back a data stream to the terminal device in real time through the data channel service, where it should be noted that the data stream fed back to the terminal device by a server can be understood as a data stream of cache data in a cache of the server. The data instruction processing method and the data instruction processing device can configure instruction time for the data instruction, so that the processing sequence of a plurality of data instructions is determined through the instruction time, and the server can conveniently execute processing corresponding to the data instruction according to the processing sequence.

The server can transmit data to the terminal device in real time through the data channel service, and before data transmission, the server can acquire device information of the terminal device, such as information of decoding playing information, coding rate information and the like of the terminal device, so that data coding can be performed according to the device information of the terminal device, and the data coding method and device are suitable for terminal devices with different performances.

The data transmitted by the server through the data channel service may include page data of an edit page, and the edit page may include a first display area, a second display area, and a clipping operation area. The first display area is used for displaying preview information of the multimedia data to be processed, so that a user can select corresponding multimedia data to process according to the preview information. The preview information may be understood as description information of multimedia data, such as picture information, text information, etc., and as in the example shown in fig. 1, a landscape video and a play video may be displayed in the first display area of an edit page of the terminal device to facilitate the user to select a video to be processed. According to the embodiment of the application, the multimedia data to be processed can be stored in the cloud, and in order to facilitate editing of a user, the account information of the user can be obtained, the relevant multimedia data can be determined, the relevant description information of the relevant multimedia data can be displayed in the first display area, the relevant description information can be at least one of character information and picture information, and the user can operate in the first display area to select the multimedia data to be processed.

The clipping operation area is used for displaying a data timeline of the multimedia data to be processed, in an example shown in fig. 1, a user of the terminal device can select the video to be processed in the first display area to display the data timeline of the video clip in the second display area so that the user can perform the clipping operation, and the clipping operation area can also show the time scale of the timeline so that the clipping operation can be performed. The second display area is used for displaying the preview key data of the multimedia data, and it should be noted that displaying the preview key data of the multimedia data in the second display area is only one embodiment of the present application, and the embodiment of the present application may also display the preview key data in the clipping operation area so as to process the multimedia data.

The preview key data refers to representative partial data captured from the multimedia data, the preview key data includes data corresponding to different times on a data time axis, and a user can perform an operation on the data time axis to determine corresponding data content. For example, when the multimedia data is video data, the preview key data may be a Group Of Pictures (GOP) that is transmitted from the server to the terminal device, where the Group Of GOP includes a plurality Of Pictures corresponding to different time information; when the multimedia data is audio data, the preview key data of the audio data may also be a preview image group, and the preview image group of the audio data may be composed of a waveform diagram of the audio data. And a scaling adjustment control of the data time axis can be configured in the editing page, and a user can adjust the length of the data time axis by adjusting the scaling adjustment control. Correspondingly, the preview key data issued by the server to the terminal device can be provided with a plurality of groups, and each group of preview key data corresponds to the data time axis with different proportions, so that the user can edit the multimedia data conveniently.

The clip operation area can include a clip cutting control, a clip zooming control, a clip attribute configuration control, a clip fusion control and the like, and a user can trigger the data time axis and the corresponding control to send an editing instruction to the server. After receiving the editing instruction, the server may perform editing processing such as editing, fusion, and attribute adjustment on the multimedia data according to the editing instruction, so as to generate a target data stream by using the edited multimedia data, and feed back the target data stream to the terminal device. The terminal device may display the edited multimedia data in the second display area so as to view a rendering effect of the multimedia data, and as in the example shown in fig. 1, a video obtained by fusing a landscape video and a play video may be displayed in the second display area.

The embodiment of the application can be applied to a scene of editing multimedia data, the editing processing process of the multimedia data can be completed through the server, the calculation requirement on the terminal equipment is reduced, and the processing processes of editing, fusing, attribute adjustment and the like of the multimedia data can be performed more conveniently. The embodiment of the application optimizes the data editing process of the multimedia data, so that the embodiment of the application can be applied to data processing scenes of various multimedia data, for example, division is performed according to data types, and the embodiment of the application can be applied to editing, fusing and attribute adjusting scenes of multimedia data such as video data and audio data. For another example, the data processing method and device can be divided according to data application scenes, and the data processing method and device can be applied to scenes for processing video and audio in scenes such as e-commerce, logistics, transportation, live broadcast, medical treatment, short video platforms and personal shooting.

On the basis of the foregoing embodiments, an embodiment of the present application provides a method for processing multimedia data, which can be applied to a server, where the server can interact with a terminal device, so as to complete an editing process of the multimedia data at the server and feed back a corresponding processing result to the terminal device in real time, and specifically, as shown in fig. 2, the method includes:

step 202, receiving a data editing request of the terminal device.

The embodiment of the application can be applied to a server, the server can interact with a terminal device, a user can operate in the terminal device to make a data editing request so as to send the data editing request to the server, and after the server receives the data editing request, in step 204, an instruction channel service and a data channel service can be created according to the data editing request. The instruction channel service is used for sending a data instruction of the terminal device to the server, and the data channel service is used for transmitting a data stream of the server to the terminal device.

Specifically, the server may receive a data instruction through the instruction channel service in step 206, where the data instruction is generated according to an operation received by an editing page on the terminal device. In step 208, the multimedia data is edited according to the data instruction, and a data stream is generated by using the processed multimedia data. In step 210, the data stream is transmitted to the terminal device through a data channel service to output the data stream in the terminal device to view the processed multimedia data.

In this embodiment of the application, the server may transmit page data of the edit page to the terminal device through the data channel service, so as to display the edit page on the terminal device, and the user may perform an operation in the edit page to determine the data instruction, and specifically, as an optional embodiment, the method further includes: providing page data of an edit page to a terminal device through a data channel service to display the edit page on the terminal device, the edit page including: a first display area, a second display area and a clipping operation area; sending preview information and time information of multimedia data to terminal equipment through a data channel service so as to display the preview information in a first display area of the terminal equipment and display a data time axis corresponding to the time information in a clipping operation area; the receiving of the viewing instruction sent by the terminal device through the instruction channel service includes: and receiving a viewing instruction of the multimedia data sent by the terminal equipment through an instruction channel service, wherein the second display area is used for displaying preview key data corresponding to the viewing instruction, and the viewing instruction is determined based on the operation of the data time axis.

The user can make a data editing request on the terminal device and send the data editing request to the server through the instruction channel service, and the server responds to the data editing request and feeds back page data of an editing page to the terminal device. The editing page comprises a first display area, a second display area and an editing operation area. The first display area is used for displaying multimedia data to be processed, and the multimedia data may include continuous image data (video data), audio data, and the like. The first display area may display related description information of related multimedia data having an association with the user, where the related description information may be information such as an identifier, a name, and an image of the related multimedia data. Specifically, as an optional embodiment, the method further includes: sending the related description information of the related multimedia data to the terminal equipment through a data channel service so as to display the related description information of the related multimedia data in a first display area; and receiving a multimedia selection instruction based on the multimedia in the first display area from the terminal equipment through the instruction channel service so as to determine the multimedia data to be processed.

Specifically, as an optional embodiment, the sending of the related description information of the related multimedia data to the terminal device through the data channel service includes: acquiring account information sent by terminal equipment, and determining related multimedia data corresponding to the account information; and issuing the related description information of the related multimedia data to the terminal equipment through the data channel service so as to be displayed in the first display area. According to the method and the device, the incidence relation can be established between the related multimedia data associated with the user and the account information of the user, so that the related multimedia data associated with the user can be determined through the incidence relation, and then the related description information is displayed in the first display area, so that the user can conveniently select the multimedia data to be processed.

Specifically, as an optional embodiment, the data instruction includes a viewing instruction; the receiving data instructions over the instruction channel service includes: receiving a viewing instruction sent by terminal equipment through an instruction channel service, wherein the viewing instruction is used for viewing multimedia data for editing; the editing processing is performed on the multimedia data according to the data instruction, and a data stream is generated by using the processed multimedia data, including: and determining preview key data of the multimedia data according to the viewing instruction, and sending the preview key data to the terminal equipment through a data channel service so as to preview the multimedia data at the terminal equipment.

According to the method and the device, the viewing instruction of the terminal equipment can be sent to the server through the instruction channel service, the server responds to the viewing instruction, the preview key data is determined, and the preview key data is transmitted to the terminal equipment as the data stream through the data channel service so as to be displayed in the terminal equipment. In a clipping operation area of an editing page of a terminal device, a data timeline of multimedia data may be displayed, and in order to facilitate a user to operate the data timeline in the clipping operation area, according to a time positioning operation of the user on the data timeline, preview data (data in preview key data) corresponding to the time positioning operation may be displayed in a second display area, where the time positioning operation may be understood as a position adjustment operation of the user on a timeline progress identifier on the data timeline in the clipping operation area. The preview key data may be understood as a group of images or the like, for example, when the multimedia data is video data, the preview key data may be a group of preview images; when the multimedia data is audio data, the preview key data may be a preview image group of a waveform image of the audio.

Taking the preview key data as the preview image group as an example, the server can send the preview image group to the terminal device, and the preview image group is composed of a plurality of images corresponding to different times. The terminal device can determine time information corresponding to the time positioning operation, and further determine a corresponding target image in the preview image group according to the time information for displaying. For example, the target time period may correspond to a time period of one minute, and accordingly (60) display images corresponding to each second may be acquired to form a preview image, and each second may correspond to multiple frames (e.g., 24 frames) of images, so that compared with a scheme of acquiring a video of one minute, the data amount transmitted to the terminal device may be reduced in the embodiment of the present application.

In addition, in another optional embodiment, a scaling control of the data timeline may be further configured in the editing page, where the scaling control is configured to adjust a length scale of the data timeline, for example, the data timeline in the clip operation region may be adjusted to be 0.1-10 times by the scaling control, so as to facilitate a user to operate the data timeline. Correspondingly, the preview key data issued by the server to the terminal device can be provided with a plurality of groups, and each group of preview key data corresponds to the data time axis with different proportions, so that the user can edit the multimedia data conveniently. Specifically, the determining the preview key data of the multimedia data according to the viewing instruction includes: determining the time axis starting time of the target time axis according to the target time axis information in the viewing instruction; determining a plurality of groups of preview time periods corresponding to different time axis scaling ratios according to the time axis starting time; and determining display key data corresponding to the preview time period in the multimedia data to determine preview key data. Each group of preview data corresponds to different scaling ratios, correspondingly, the time periods corresponding to different scaling ratios are different in length, and the corresponding preview data acquisition intervals can be set according to requirements, for example, the total data amount of each group of preview data can be preset, the intervals can be determined according to the total data amount, preset time intervals (the time intervals of each group can be the same or different) can also be preset, so that the preview data can be acquired according to the preset time intervals, and then the preview data is combined into preview key data and sent to the terminal equipment, so as to be displayed in the terminal equipment.

The data time axis of the multimedia data to be processed can be displayed in a clipping operation area of an editing page, a user can operate the data time axis in the clipping operation area in terminal equipment and send an editing instruction to a server, and the server can receive the editing instruction sent by the terminal equipment through an instruction channel service; and editing the multimedia data according to the editing instruction, and generating a target data stream by adopting the edited multimedia data. Specifically, as an optional embodiment, the data instruction includes an editing instruction; the receiving data instructions over the instruction channel service includes: receiving an editing instruction sent by terminal equipment through an instruction channel service; the editing processing is performed on the multimedia data according to the data instruction, and a data stream is generated by using the processed multimedia data, including: and editing the multimedia data according to the editing instruction, and generating a target data stream by adopting the edited multimedia data.

Specifically, as an optional embodiment, the editing processing of the multimedia data according to the editing instruction includes at least one of the following steps: determining corresponding cutting start-stop time according to the segment cutting instruction, and cutting a corresponding target segment from the multimedia data; carrying out zooming processing on a target segment of the multimedia data according to the segment zooming instruction; adjusting the segment attribute of a target segment of multimedia data according to a segment attribute setting instruction, wherein the multimedia data comprises video data, and the segment attribute comprises video size, video position, video definition and video time; and fusing the first segment and the second segment of the multimedia data according to the segment fusion instruction. In the embodiment of the application, at least one data time axis of the multimedia data can be displayed in the clipping operation area, and a user can operate the at least one data time axis in the clipping operation area to process the multimedia data. It should be noted that, in addition to the above-described data processing operation of multimedia data, the embodiment of the present application may also complete other operations, which may be configured specifically according to requirements.

In this embodiment of the application, a terminal device may send an edit instruction to a server many times, in order to prevent an execution sequence of multiple edit instructions from being incorrect and causing an abnormal edit of multimedia data, the terminal device may add instruction time to the edit instruction, and the server may determine a processing sequence according to the instruction time of the edit instruction, so as to execute processing corresponding to the edit instruction in sequence, specifically, as an optional embodiment, the editing processing of the multimedia data according to the edit instruction includes: determining instruction time corresponding to the editing instruction; and determining a processing sequence corresponding to the editing instruction according to the instruction time so as to execute processing corresponding to the editing instruction according to the processing sequence. The terminal equipment can send one editing instruction to the server at one time, and also can send at least two editing instructions to the server at one time, and when the terminal equipment sends one editing instruction each time through the instruction channel service, the server can determine the execution sequence of the editing instructions according to the receiving time of the editing instructions; when the terminal device sends at least two editing instructions through the instruction channel service, the terminal device can configure instruction time for the editing instructions to determine the processing sequence of the editing instructions through the instruction time, so that the server executes the processing corresponding to the editing instructions according to the processing sequence, and the probability of data processing errors is reduced.

After the server edits the multimedia data to form a target data stream, the target data stream can be sent to the terminal equipment in real time through the data channel service so as to output the processed multimedia data in the terminal equipment, and a user can check the processing effect of the multimedia data in the second display area. The device performance of different terminal devices is different, and the encoding and decoding effects are different, so that before data is transmitted to the terminal device, the embodiment of the present application may collect device information of the terminal device, so as to feed back data corresponding to the device performance of the terminal device according to the device information of the terminal device, and specifically, as an optional embodiment, the method further includes: acquiring equipment information of terminal equipment, wherein the equipment information comprises decoding playing information and coding code rate information; and coding the data stream to be transmitted according to the equipment information so as to output the coded data stream to the terminal equipment. The data stream encoded by the server may include page data of an edited page, multimedia data, data-processed multimedia data, and the like.

On the basis of the foregoing embodiments, an embodiment of the present application further provides a method for processing multimedia data, which can be applied to a server, as shown in fig. 3, where the method includes:

step 302, receiving a data editing request of the terminal device.

Step 304, creating an instruction channel service and a data channel service according to the data editing request.

Step 306, providing page data of the editing page to the terminal device through the data channel service, so as to display the editing page on the terminal device, where the editing page includes: a first display area, a second display area and a clipping operation area.

And 308, sending the preview information and the time information of the multimedia data to the terminal equipment through a data channel service so as to display the preview information in a first display area of the terminal equipment and display a data time axis corresponding to the time information in a clipping operation area.

And 310, receiving a viewing instruction of the multimedia data sent by the terminal equipment through an instruction channel service, wherein the second display area is used for displaying preview key data corresponding to the viewing instruction, and the viewing instruction is determined based on the operation of the data time axis. The viewing instructions are for viewing the multimedia data for editing.

Step 312, determining the preview key data of the multimedia data according to the viewing instruction, and sending the preview key data to the terminal device through the data channel service, so as to preview the multimedia data at the terminal device.

And step 314, receiving the editing instruction sent by the terminal equipment through the instruction channel service.

And step 316, editing the multimedia data according to the editing instruction, and generating a target data stream by using the edited multimedia data.

Step 318, sending the data stream to the terminal device through the data channel service, so as to output the data stream in the terminal device to view the processed multimedia data.

The method and the device can be applied to a server, the server can create an instruction channel service and a data channel service to interact with terminal equipment, a user can make a data editing request on the terminal equipment and send the data editing request to the server through the instruction channel service, the server feeds an editing page back to the terminal equipment through the data channel service, the server can output preview information and time information of multimedia data to be processed to the terminal equipment, the terminal equipment displays the preview information in a first display area of the editing page, and a data time axis corresponding to the time information is displayed in a clipping operation area of the editing page; in addition, in order to facilitate a user to operate a data timeline in the editing operation area, in the embodiment of the application, the preview key data corresponding to the data timeline can be issued to the terminal device, so that the user can determine preview images corresponding to different times on the data timeline, the user can operate the data timeline in the editing operation area, and send an editing instruction to the server through the instruction channel service, the server processes the multimedia data according to the editing instruction, determines the processed multimedia data, forms a target data stream, and feeds back the target data stream to the terminal device through the data channel service, so that the edited multimedia data is displayed in a second display area in an editing page of the terminal device.

On the basis of the foregoing embodiment, the embodiment of the present application further provides a method for processing multimedia data, which can be applied to a terminal device, where the terminal device can interact with a server, a user can operate on the terminal device and send an editing instruction to the server, and the server performs operations such as editing, fusing, and attribute adjustment of the multimedia data according to the editing instruction. Specifically, as shown in fig. 4, the method includes:

step 402, sending a data editing request to the server, so that the server creates an instruction channel service and a data channel service according to the data editing request. The server can also provide an editing page to the terminal equipment, and the editing page comprises a first display area, a second display area and a clipping operation area.

The embodiment of the application can be applied to terminal equipment, a user can make a data editing request in a terminal page and send the data editing request to a server, the server responds to the data editing request, establishes an instruction channel service and a data channel service, and feeds back page data of the editing page to the terminal equipment through the data channel service so as to display the editing page in the terminal equipment, and further can respond to the operation on the editing page in step 404 to generate a corresponding data instruction, wherein the editing page is used for receiving the operation aiming at multimedia data; in step 406, the data instruction is sent to the server through an instruction channel service, so that the server processes the multimedia data according to the data instruction and generates a data stream according to the processed multimedia data. In step 408, receiving a data stream fed back by the server through a data channel service; in step 410, the data stream is parsed and the multimedia data is output on the edit page.

Specifically, as an optional embodiment, the generating a corresponding data instruction in response to the operation on the edit page includes at least one of the following steps: responding to the viewing operation of the editing page, and generating a corresponding viewing instruction, wherein the viewing instruction is used for determining preview key data of the multimedia data for editing; and responding to the editing operation of the editing page, and generating an editing instruction, wherein the editing instruction is used for editing and processing the multimedia data. Correspondingly, in an optional embodiment, the receiving, by the data channel service, the data stream fed back by the server includes at least one of the following steps: receiving preview key data corresponding to the viewing instruction through a data channel service; and receiving the processed target data stream corresponding to the editing instruction through the data channel service.

On one hand, the terminal device of the embodiment of the application can acquire the preview key data for editing, and specifically, the terminal device of the embodiment of the application can send a viewing instruction to the server through the instruction channel service, the server responds to the viewing instruction, determines the preview key data, and feeds the preview key data back to the terminal device through the data channel service, so that the preview data in the preview key data is displayed in the second display area of the terminal device. On the other hand, the embodiment of the present application may send an edit instruction for the multimedia data to edit the multimedia data, and specifically, the embodiment of the present application may receive an edit operation of a user in the editing operation area to form an edit instruction, and send the edit instruction to the server through the instruction channel service, where the server processes the multimedia data in response to the edit instruction to form a target data stream, and then feeds the data stream back to the terminal device through the data channel service to be displayed in the second display area of the edit page. The terminal device can analyze the multimedia data from the target data stream, and can configure a play control of the multimedia data in the editing page, and the user can trigger the play control to control actions such as playing, pausing, fast forwarding, fast rewinding and the like of the multimedia data in the second display area.

The user can adjust the data time axis of the multimedia data in the clipping operation area of the terminal device, for example, adjust the proportion of the data time axis, and adjust the information of the start time, the end time and the like of the data time axis, so as to send a viewing instruction to the server, where the viewing instruction may include target time axis information of a target time axis in the clipping operation area, so that the server determines corresponding preview key data according to the target time axis information, and feeds the preview key data back to the terminal device. The user can adjust the time axis progress mark on the data time axis in the clipping operation area at the terminal equipment to determine corresponding time information, and further determine the preview data displayed in the second display area, so that the user can conveniently operate.

The user can operate in a clipping operation area of the editing page, and the following controls can be set in the clipping operation area: the system comprises a clip cutting control, a clip zooming control, a clip attribute configuration control, a clip fusion control and the like, wherein a user can trigger a data time axis and the corresponding control in the terminal equipment so as to send an editing instruction to a server. Specifically, as an optional embodiment, the generating the data instruction in response to the editing operation on the editing page includes at least one of the following steps: generating a clip clipping instruction based on the triggering of the data time axis and the clip clipping control; generating a fragment zooming instruction based on the triggering of the data time axis and the fragment zooming control; generating a fragment attribute setting instruction based on triggering of a data time axis and a fragment attribute configuration control; and generating a fragment fusion instruction based on the triggering of the fragment fusion control.

In this embodiment of the present application, the terminal device may send one editing instruction at a time, or may send a plurality of editing instructions at a time, and when sending a plurality of editing instructions at a time, the server may determine a processing sequence of the editing instructions according to instruction time of the editing instructions, and specifically, as an optional embodiment, sending the data instruction to the server through an instruction channel service includes: acquiring instruction time corresponding to an editing instruction; and adding the instruction time into the editing instruction, and sending the editing instruction to a server through an instruction channel service. The method and the device for processing the editing instructions can configure instruction time for the editing instructions, and can determine the processing sequence of a plurality of editing instructions through the instruction time under the condition that at least two editing instructions are sent at one time, so that the server can conveniently execute processing corresponding to the editing instructions according to the processing sequence.

The implementation manner of this embodiment is similar to that of the above embodiment, and the detailed implementation manner of the above embodiment may be referred to, and is not described herein again.

The method and the device can be applied to terminal equipment, the terminal equipment can interact with a server, the server can establish instruction channel service and data channel service, the server can provide page data of an editing page for the terminal equipment through the data channel service and output preview information and time axis information of multimedia data to be processed to the terminal equipment, the terminal equipment displays the preview information in a first display area of the editing page and displays a data time axis in a clipping operation area of the editing page; the user can operate the data timeline in the editing operation area of the editing page of the terminal device, and sends an editing instruction to the server through the instruction channel service, the server processes the multimedia data according to the editing instruction, determines the edited multimedia data, and feeds back a target data stream to the terminal device through the data channel service so as to display the data timeline in the second display area of the terminal device.

It should be noted that, for simplicity of description, the method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the embodiments are not limited by the order of acts described, as some steps may occur in other orders or concurrently depending on the embodiments. Further, those skilled in the art will also appreciate that the embodiments described in the specification are presently preferred and that no particular act is required of the embodiments of the application.

On the basis of the foregoing embodiment, this embodiment further provides a multimedia data processing apparatus, and with reference to fig. 5, the apparatus may specifically include the following modules:

an editing request receiving module 502, configured to receive a data editing request of a terminal device.

A channel service creating module 504, configured to create an instruction channel service and a data channel service according to the data editing request.

A data instruction receiving module 506, configured to receive a data instruction through the instruction channel service, where the data instruction is generated according to an operation received by an edit page on a terminal device.

And a data stream generating module 508, configured to edit the multimedia data according to the data instruction, and generate a data stream using the processed multimedia data.

A data stream feedback module 510, configured to send the data stream to the terminal device through a data channel service, so as to output the data stream in the terminal device to view the processed multimedia data.

The embodiment of the application can be applied to a server, the server can receive a data editing request of terminal equipment and respond to the data editing request to create instruction channel service and data channel service, the terminal equipment can transmit a data instruction to the server through the instruction channel service, and the server can transmit a data stream to the terminal equipment through the data channel service. In the embodiment of the application, the server can receive the data instruction sent by the terminal device through the instruction channel service and edit the multimedia data in the server to form the data stream, so that the data stream is sent to the terminal device through the data channel service to view the processed multimedia data in the terminal device. According to the embodiment of the application, the data instruction can be transmitted through the instruction channel service, and the data stream can be transmitted through the data channel service, so that the data editing process of the multimedia data can be completed at the server side, the calculation requirement on the terminal equipment is reduced, and the multimedia data can be conveniently edited.

On the basis of the foregoing embodiment, this embodiment further provides a processing apparatus for multimedia data, which may specifically include the following modules:

and the editing request receiving and processing module is used for receiving a data editing request of the terminal equipment.

The channel service creating and processing module is used for creating instruction channel service and data channel service according to the data editing request;

the page data providing and processing module is used for providing page data of an editing page to the terminal equipment through a data channel service so as to display the editing page on the terminal equipment, and the editing page comprises: a first display area, a second display area and a clipping operation area.

And the preview information issuing processing module is used for sending the preview information and the time information of the multimedia data to the terminal equipment through the data channel service so as to display the preview information in a first display area of the terminal equipment and display a data time axis corresponding to the time information in a clipping operation area.

The viewing instruction receiving and processing module is used for receiving a viewing instruction of the multimedia data sent by the terminal equipment through an instruction channel service, the second display area is used for displaying preview key data corresponding to the viewing instruction, and the viewing instruction is determined based on the operation of the data time axis. The viewing instructions are for viewing the multimedia data for editing.

And the key data issuing processing module is used for determining preview key data of the multimedia data according to the viewing instruction, and sending the preview key data to the terminal equipment through a data channel service so as to preview the multimedia data at the terminal equipment.

And the editing instruction receiving and processing module is used for receiving the editing instruction sent by the terminal equipment through the instruction channel service.

And the data stream generation processing module is used for editing the multimedia data according to the editing instruction and generating a target data stream by adopting the edited multimedia data.

And the data stream issuing processing module is used for sending the data stream to the terminal equipment through a data channel service so as to output the data stream in the terminal equipment to check the processed multimedia data.

On the basis of the foregoing embodiment, this embodiment further provides a multimedia data processing apparatus, and with reference to fig. 6, the apparatus may specifically include the following modules:

an edit request sending module 602, configured to send a data edit request to the server, so that the server creates an instruction channel service and a data channel service according to the data edit request.

The data instruction generating module 604 is configured to generate a corresponding data instruction in response to an operation on an edit page, where the edit page is used to receive an operation on multimedia data.

A data instruction sending module 606, configured to send the data instruction to a server through an instruction channel service, so that the server processes multimedia data according to the data instruction, and generates a data stream according to the processed multimedia data.

A data stream receiving module 608, configured to receive a data stream fed back by the server through a data channel service.

And a data stream parsing module 610, configured to parse the data stream and output the multimedia data on the edit page.

The present application further provides a non-transitory, readable storage medium, where one or more modules (programs) are stored, and when the one or more modules are applied to a device, the device may execute instructions (instructions) of method steps in this application.

Embodiments of the present application provide one or more machine-readable media having instructions stored thereon, which when executed by one or more processors, cause an electronic device to perform the methods as described in one or more of the above embodiments. In the embodiment of the application, the electronic device includes a server, a terminal device and other devices.

Embodiments of the present disclosure may be implemented as an apparatus, which may include a server (cluster), a terminal, etc. electronic device, using any suitable hardware, firmware, software, or any combination thereof, to perform a desired configuration. Fig. 7 schematically illustrates an example apparatus 700 that may be used to implement various embodiments described herein.

For one embodiment, fig. 7 illustrates an exemplary apparatus 700 having one or more processors 702, a control module (chipset) 704 coupled to at least one of the processor(s) 702, a memory 706 coupled to the control module 704, a non-volatile memory (NVM)/storage 708 coupled to the control module 704, one or more input/output devices 710 coupled to the control module 704, and a network interface 712 coupled to the control module 704.

The processor 702 may include one or more single-core or multi-core processors, and the processor 702 may include any combination of general-purpose or special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, the apparatus 700 can be used as a server, a terminal, or the like in the embodiments of the present application.

In some embodiments, the apparatus 700 may include one or more computer-readable media (e.g., the memory 706 or the NVM/storage 708) having instructions 714 and one or more processors 702 in combination with the one or more computer-readable media configured to execute the instructions 714 to implement modules to perform the actions described in this disclosure.

For one embodiment, control module 704 may include any suitable interface controllers to provide any suitable interface to at least one of processor(s) 702 and/or any suitable device or component in communication with control module 704.

The control module 704 may include a memory controller module to provide an interface to the memory 706. The memory controller module may be a hardware module, a software module, and/or a firmware module.

The memory 706 may be used, for example, to load and store data and/or instructions 714 for the apparatus 700. For one embodiment, memory 706 may comprise any suitable volatile memory, such as suitable DRAM. In some embodiments, the memory 706 may comprise a double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

For one embodiment, control module 704 may include one or more input/output controllers to provide an interface to NVM/storage 708 and input/output device(s) 710.

For example, NVM/storage 708 may be used to store data and/or instructions 714. NVM/storage 708 may include any suitable non-volatile memory (e.g., flash memory) and/or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drive(s) (HDD (s)), one or more Compact Disc (CD) drive(s), and/or one or more Digital Versatile Disc (DVD) drive (s)).

NVM/storage 708 may include storage resources that are part of the device on which apparatus 700 is installed, or it may be accessible by the device and may not necessarily be part of the device. For example, NVM/storage 708 may be accessible over a network via input/output device(s) 710.

Input/output device(s) 710 may provide an interface for apparatus 700 to communicate with any other suitable device, input/output device(s) 710 may include communication components, audio components, sensor components, and so forth. Network interface 712 may provide an interface for device 700 to communicate over one or more networks, and device 700 may communicate wirelessly with one or more components of a wireless network according to any of one or more wireless network standards and/or protocols, such as access to a communication standard-based wireless network, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof.

For one embodiment, at least one of the processor(s) 702 may be packaged together with logic for one or more controller(s) (e.g., memory controller module) of control module 704. For one embodiment, at least one of the processor(s) 702 may be packaged together with logic for one or more controllers of control module 704 to form a System In Package (SiP). For one embodiment, at least one of the processor(s) 702 may be integrated on the same die with logic for one or more controller(s) of control module 704. For one embodiment, at least one of the processor(s) 702 may be integrated on the same die with logic for one or more controllers of control module 704 to form a system on a chip (SoC).

In various embodiments, the apparatus 700 may be, but is not limited to being: a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet, a netbook, etc.), among other terminal devices. In various embodiments, apparatus 700 may have more or fewer components and/or different architectures. For example, in some embodiments, device 700 includes one or more cameras, a keyboard, a Liquid Crystal Display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an Application Specific Integrated Circuit (ASIC), and speakers.

The detection device can adopt a main control chip as a processor or a control module, sensor data, position information and the like are stored in a memory or an NVM/storage device, a sensor group can be used as an input/output device, and a communication interface can comprise a network interface.

An embodiment of the present application further provides an electronic device, including: a processor; and a memory having executable code stored thereon that, when executed, causes the processor to perform a method as described in one or more of the embodiments of the application.

Embodiments of the present application also provide one or more machine-readable media having executable code stored thereon that, when executed, cause a processor to perform a method as described in one or more of the embodiments of the present application.

For the device embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, refer to the partial description of the method embodiment.

The embodiments in the present specification are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other.

Embodiments of the present application are described with reference to flowchart illustrations and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing terminal to cause a series of operational steps to be performed on the computer or other programmable terminal to produce a computer implemented process such that the instructions which execute on the computer or other programmable terminal provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

While preferred embodiments of the present application have been described, additional variations and modifications of these embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including the preferred embodiment and all such alterations and modifications as fall within the true scope of the embodiments of the application.

Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or terminal. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or terminal that comprises the element.

The foregoing has introduced in detail a multimedia data processing method, a multimedia data processing apparatus, an electronic device and a storage medium provided in the present application, and specific examples are applied herein to explain the principles and embodiments of the present application, and the descriptions of the foregoing embodiments are only used to help understand the method and core ideas of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.

Claims

1. A method for processing multimedia data, the method comprising:

receiving a data editing request of terminal equipment;

creating an instruction channel service and a data channel service according to the data editing request;

receiving a data instruction through the instruction channel service, wherein the data instruction is generated according to the operation received by an editing page on the terminal equipment;

editing the multimedia data according to the data instruction, and generating a data stream by adopting the processed multimedia data;

and transmitting the data stream to the terminal equipment through a data channel service so as to output the data stream in the terminal equipment to view the processed multimedia data.

2. The method of claim 1, wherein the data instruction comprises a view instruction;

the receiving data instructions over the instruction channel service includes:

receiving a viewing instruction sent by terminal equipment through an instruction channel service, wherein the viewing instruction is used for viewing multimedia data for editing;

the editing processing is performed on the multimedia data according to the data instruction, and a data stream is generated by using the processed multimedia data, including:

and determining preview key data of the multimedia data according to the viewing instruction, and sending the preview key data to terminal equipment through a data channel service so as to preview the multimedia data on the terminal equipment.

3. The method of claim 1 or 2, wherein the data instructions comprise editing instructions;

the receiving data instructions over the instruction channel service includes:

receiving an editing instruction sent by terminal equipment through an instruction channel service;

the editing processing is performed on the multimedia data according to the data instruction, and a data stream is generated by adopting the processed multimedia data, and the method comprises the following steps:

and editing the multimedia data according to the editing instruction, and generating a target data stream by adopting the edited multimedia data.

4. The method of claim 2, further comprising:

providing page data of an edit page to a terminal device through a data channel service to display the edit page on the terminal device, the edit page including: a first display area, a second display area and a clipping operation area;

sending preview information and time information of multimedia data to terminal equipment through a data channel service so as to display the preview information in a first display area of the terminal equipment and display a data time axis corresponding to the time information in a clipping operation area;

the receiving of the viewing instruction sent by the terminal device through the instruction channel service includes:

and receiving a viewing instruction of the multimedia data sent by the terminal equipment through an instruction channel service, wherein the second display area is used for displaying preview key data corresponding to the viewing instruction, and the viewing instruction is determined based on the operation of the data time axis.

5. The method of claim 4, wherein determining preview key data for multimedia data according to the viewing instruction comprises:

determining the time axis starting time of the target time axis according to the target time axis information in the viewing instruction;

determining a plurality of groups of preview time periods corresponding to different time axis scaling ratios according to the time axis starting time;

and determining display key data corresponding to the preview time period in the multimedia data to determine preview key data.

6. The method according to claim 1, wherein the editing the multimedia data according to the editing instruction comprises at least one of the following steps:

determining corresponding cutting start-stop time according to the segment cutting instruction, and cutting a corresponding target segment from the multimedia data;

carrying out zooming processing on a target segment of the multimedia data according to the segment zooming instruction;

setting an instruction according to the segment attributes, and adjusting the segment attributes of a target segment of multimedia data, wherein the multimedia data comprises video data, and the segment attributes comprise video size, video position, video definition and video time;

and fusing the first segment and the second segment of the multimedia data according to the segment fusion instruction.

7. A method for processing multimedia data, the method comprising:

sending a data editing request to a server so that the server can establish instruction channel service and data channel service according to the data editing request;

responding to an operation on an editing page, and generating a corresponding data instruction, wherein the editing page is used for receiving the operation aiming at the multimedia data;

sending the data instruction to a server through an instruction channel service so that the server processes multimedia data according to the data instruction and generates a data stream according to the processed multimedia data;

receiving a data stream fed back by the server through a data channel service;

and analyzing the data stream, and outputting the multimedia data on the editing page.

8. The method of claim 7, wherein generating the data instructions in response to the editing operation on the editing page comprises at least one of:

generating a clip clipping instruction based on the triggering of the data time axis and the clip clipping control;

generating a fragment zooming instruction based on the triggering of the data time axis and the fragment zooming control;

generating a fragment attribute setting instruction based on triggering of a data time axis and a fragment attribute configuration control;

and generating a fragment fusion instruction based on the triggering of the fragment fusion control.

9. An electronic device, comprising: a processor; and

memory having stored thereon executable code which, when executed by a processor, performs the method of any one of claims 1-8.

10. One or more machine-readable media having executable code stored thereon that, when executed by a processor, performs the method of any of claims 1-8.