WO2020063850A1

WO2020063850A1 - Method for processing media data and terminal and server

Info

Publication number: WO2020063850A1
Application number: PCT/CN2019/108514
Authority: WO
Inventors: 宋翼; 范宇群; 邸佩云; 王业奎
Original assignee: 华为技术有限公司
Priority date: 2018-09-27
Filing date: 2019-09-27
Publication date: 2020-04-02

Abstract

Disclosed are a method for processing media data, a terminal and a server, relating to the technical field of streaming media transmission and used for reducing the operation complexity of a user needing to respectively execute the same operation on overlays in order to realize a corresponding target so as to make the operation of the user on the overlay more effective and improve the user subjective experience. The method comprises: a terminal acquiring at least two overlays corresponding to media data, wherein the overlay corresponds to first information or the overlay corresponds to second information and third information, the first information comprises group identification information, the second information is used for indicating an operation function of the overlay, the third information is used for indicating the group identification information, and the overlays in the same group correspond to the same operation function; and a terminal processes at least two overlays according to the first information of at least two overlays.

Description

Method, terminal and server for processing media data

This application claims priority from a U.S. provisional patent application filed on September 27, 2018 with the U.S. Patent Office, application number 62 / 737,900, and invention name "Method, terminal and server for processing media." Incorporated in this application.

Technical field

The embodiments of the present application relate to the technical field of streaming media transmission, and in particular, to a method, terminal, and server for processing media data.

Background technique

The ISO / IEC 23090-2 standard specification is also called the OMAF (Omnidirectional Media Format) standard specification. This specification defines a media application format that can implement omnidirectional media presentation in applications. Omnidirectional media mainly refers to panoramic video (360-degree video) and related audio. The OMAF specification first specifies a list of projection methods that can be used to convert spherical video into two-dimensional video, and secondly, how to use ISO base media file format (ISOBMFF) to store omnidirectional media and the associated media. Metadata, and how to encapsulate omnidirectional media data and transmit omnidirectional media data in a streaming media system, such as through Dynamic Adaptive Streaming based on HyperText Transfer Protocol (HTTP) HTTP, DASH), ISO / IEC 23009-1 standard dynamic adaptive streaming transmission.

Application scenarios are currently proposed in which certain areas in a virtual reality (VR) video can be superimposed to display additional overlays (eg, pictures or videos). In the current OMAF standard, the basic data structure of the overlay has been defined. The basic data structure of the overlay defines some basic properties of the overlay structure (for example, including the number of overlays, Id number, control symbol, control structure, etc.). Among them, the specific function of each structure is defined in the semantics of the control symbol syntax element overlay_control_flag. After the terminal resolves to the overlay, it can determine how to handle the overlay based on these syntax elements.

However, the above syntax elements are described in terms of operations and processing methods that can be performed by a single overlay. When the user needs to perform the same operation on one or more overlays, if the current method is used to implement all overlays one by one, this will increase the complexity of the operation and reduce the user's experience when watching videos.

Summary of the Invention

The embodiments of the present application provide a method, a terminal, and a server for processing media data, so as to reduce the complexity of operations that require users to perform the same operations on each overlay to achieve the corresponding purpose, make the operations on overlays more efficient, and improve the subjectivity of users. Experience.

In order to achieve the foregoing objective, the embodiments of the present application provide the following technical solutions:

In a first aspect, an embodiment of the present application provides a method for processing media data, including: a terminal receiving at least two overlays corresponding to media data; overlay corresponding to first information; or overlay corresponding to second information and third information; The first information includes the group identification information of the overlay, the second information is used to indicate the operation function corresponding to the overlay, and the third information is used to indicate the group identification information of the overlay or the information that belongs to the same group as the overlay. Identification information of other overlays; when the overlay corresponds to the first information, the terminal processes the at least two overlays according to the first information of the at least two overlays, or when the overlay corresponds to the first information In the case of the second information and the third information, the terminal processes the at least two overlays according to the second information and the third information corresponding to the at least two overlays.

An embodiment of the present application provides a method for processing media data. A terminal can process one or more overlays having the same group identification information by using first information corresponding to each overlay in at least two overlays. Compared with the prior art, in which one or more overlays are processed the same, each overlay can be processed one by one, which can reduce the complexity of operations and improve the user's subjective experience.

It should be understood that the foregoing terminal may perform the same processing on the overlays belonging to the same group according to the first information of at least two overlays. For example, to display all overlays in the same group at the granularity of the group. Or close all overlays in the same group with group granularity. For example, all overlays in the same group are scaled with group granularity.

In a possible implementation manner, the terminal processing the at least two overlays according to the first information of the at least two overlays includes: the terminal displays at least one group, and is used to indicate at least one group Information about the operation functions corresponding to each group and the overlays belonging to each group. At least one group is determined by the first information corresponding to each of the at least two overlays; the operation function corresponding to one group is determined by the overlay structure included in the overlay in each group.

In a possible implementation manner, the terminal processes the at least two overlays according to the second information and the third information corresponding to the at least two overlays, including: the terminal displaying at least one group, and indicating at least one group The information of the operation function corresponding to each group in the group and the overlay belonging to each group. At least one group is determined by third information corresponding to each of the at least two overlays, and an operation function corresponding to one group is determined by an overlay-related area control structure included in the overlay in the group.

It should be understood that the terminal may display at least one group on the display interface and information used to indicate the operation function corresponding to each group. In addition, it may also display information used to indicate the overlay in each group, which is convenient. The user knows the operation function corresponding to each group and what overlays there are in each group.

In a possible implementation manner, the terminal processes at least two overlays according to the first information of the at least two overlays, or the terminal processes second information and third information corresponding to the at least two overlays. Processing the at least two overlays includes: when any group is triggered, all overlays in the any group respond to an operation function of the any group.

In a possible implementation manner, the terminal processes the at least two overlays according to the first information of the at least two overlays, or the terminal processes the second information and the first according to the second information corresponding to the at least two overlays. Three pieces of information process the at least two overlays, including: when any overlay in any group is triggered, other overlays belonging to the same group as any overlay also respond to the operation of the any group Features.

In a possible implementation manner, the operation function is display. The method provided in the embodiment of the present application further includes: when any one of the at least one group is triggered, the overlay in any one of the groups is displayed. In this way, the terminal can display all overlays belonging to the same group with granularity based on the triggered operation. It should be understood that the operation function of the overlay is taken as an example here. In an actual process, an operation function of the overlay may also be size scaling, position change, and the like.

In a possible implementation manner, the operation function is turned off. The method provided in the embodiment of the present application further includes: when any overlay is triggered, any overlay displayed on the terminal, and any overlay that belongs to the same group as any overlay Other overlays are closed. In this way, when multiple overlays are displayed on the terminal, the terminal can close all overlays belonging to the same group with the granularity of the group based on the trigger operation. Compared with the prior art, if multiple overlays are displayed on a terminal, each overlay needs to be closed one by one, which can reduce the operation complexity.

In a possible implementation manner, the overlay also corresponds to fourth information. The fourth information is used to indicate that, when the first operation function is performed on the overlay, all the overlays in the group to which the overlay belongs respond to the first operation function. The first information of the at least two overlays processes the at least two overlays, or the terminal processes the at least two overlays according to second information and third information corresponding to the at least two overlays. And further includes: the terminal processes at least two overlays according to the first information and the fourth information of the at least two overlays; or the terminal processes at least two overlays according to the second information, the third information, and the fourth information of the at least two overlays overlay for processing. Through the fourth information, the terminal can determine the overlay for group operation processing.

In a possible implementation manner, each group in the at least one group further corresponds to indication information used to indicate a group operation.

It should be understood that when the terminal displays at least one group, the terminal may further display, according to the fourth information, instruction information indicating the group operation corresponding to each group.

In a possible implementation manner, when any overlay performs the first operation function, other overlays belonging to the same group as the overlay also respond to the first operation function.

In a possible implementation manner, the overlay also corresponds to the fifth information. The fifth information is used to indicate that in the case of performing the first operation function on the overlay, all the overlays in the group to which the overlay belongs respond to the first operation function, or the overlay is executed. In a first operation function, the terminal processes the at least two overlays according to the first information of the at least two overlays, or the terminal pairs the second information and the third information according to the at least two overlays. The processing of the at least two overlays further includes: the terminal processes the at least two overlays according to the first information and the fifth information of the at least two overlays; or the terminal processes the second information and the third information according to the at least two overlays And fifth information to process at least two overlays. Through the fifth information, the terminal can determine that the overlay can be processed in a group operation and an individual operation.

In a possible implementation manner, each group in the at least one group further corresponds to instruction information for indicating a group operation and instruction information for indicating an independent operation. This is convenient for choosing whether to handle overlays in a group operation or in a single operation.

It should be understood that when the terminal displays at least one group, the terminal may further display, according to the fifth information, the instruction information indicating the group operation and the instruction information indicating the separate operation corresponding to each group.

It should be noted that whether the terminal processes at least two overlays according to the first information, or processes at least two overlays according to the second information and the third information. The terminal may determine whether to operate alone or in groups according to the fourth information and the fifth information corresponding to the overlay.

It should be understood that when an individual operation is triggered, when any overlay in a group performs a first operation function, other overlays in the same group do not respond to the first operation function. When a group operation is triggered, when any overlay in a group performs the first operation function, other overlays in the same group also respond to the first operation function. This is convenient for selecting a group operation or an individual operation based on a user's instruction.

In a possible implementation manner, when the overlay corresponds to the first information, the overlay also corresponds to the sixth information. The sixth information is used to indicate an operation function corresponding to the overlay. The terminal performs at least two overlays according to the first information of the at least two overlays. The processing includes: the terminal processes at least two overlays according to the first information and the sixth information of the at least two overlays; or the terminal processes the at least two overlays according to the second information, the third information, and the sixth information of the at least two overlays. overlay for processing. In this way, when the terminal parses the sixth message, it can determine the operation function of each overlay based on the operation function indicated by the sixth message. When at least one group is displayed, the operation function of the group may also be determined based on the sixth message.

In a possible implementation manner, the second information and the third information are carried in an overlay file format.

It should be understood that when the overlay is encapsulated in the OMAF standard file format, the second information and the third information are carried in the OMAF file format of the overlay.

In a possible implementation manner, the file format includes an overlay structure, an overlay-related area control structure, and an overlay group box located in the overlay structure. Then the second information is located in the overlay association area control structure, and the third information is located in the overlay group box. It should be understood that in this case, the terminal can obtain the overlay association area control structure and the third information by analyzing the overlay structure, and then determine the operation function of the overlay according to the overlay association area control structure, and obtain the overlay group identification information according to the third information. To determine the group to which the overlay belongs.

As another implementation manner, the third information is located in an overlay control structure included in the overlay structure.

In a possible implementation manner, the first information is carried in an overlay file format. It should be understood that the first information is located in an overlay group box included in the overlay. At this time, the overlay structure may not carry an overlay-related area control structure. It should be understood that when the first information is carried in the file format, after receiving the overlay, the terminal may obtain the group identification information of the overlay by analyzing the overlay group box of the overlay, and then determine the group to which the overlay belongs according to the group identification information.

In a possible implementation manner, the third information is carried in supplementary enhanced information (supplementary enhancement information) of the overlay code stream corresponding to the overlay, and the second information is carried in an overlay-related area control structure of the overlay.

It should be understood that the overlay structure of the overlay at this time includes an overlay-related area control structure. At this time, the operation function corresponding to each overlay can be indicated by the overlay-related area control structure. For example, the overlay-associated area control structure may indicate that the operation function is displaying or closing.

In a possible implementation manner, the first information is carried in an SEI of an overlay code stream corresponding to the overlay. It should be understood that, at this time, the overlay structure of the overlay may not include an overlay-related area control structure. Optionally, the overlay structure may include an overlay control symbol for indicating an operation function corresponding to the overlay.

It should be understood that the SEI payload type is used to indicate that the SEI carries the group identification information of the overlay. In this way, when the terminal processes the overlay, it can determine the group identification information of the overlay carried in the SEI according to the payload type when parsing to the SEI, and further analyze the SEI to obtain the group identification information of the overlay.

It should be understood that when the overlay structure of the overlay may not include the control structure of the overlay association area, the SEI load type is also used to indicate the attribute of the group. For example, the attributes of the SEI load type group are a common display group or a common interaction group. For example, if it is a common interaction group, an interactive operation may be performed on the group. If it is a common display group, you can display or close the overlay in the group in a group operation.

In a possible implementation manner, the third information is carried in a Media Presentation Description (MPD) including a media data stream of the overlay, and the second information is carried in an overlay association area control structure of the overlay. It should be understood that the overlay structure of the overlay at this time includes an overlay-related area control structure. At this time, the operation function corresponding to each overlay can be indicated by the overlay-related area control structure. For example, the overlay-associated area control structure may indicate that the operation function is displaying or closing.

In a possible implementation manner, the first information is carried in an MPD of a media data stream including an overlay. It should be understood that the overlay's overlay structure may not include an overlay-related area control structure. Optionally, the overlay structure may include an overlay control symbol for indicating an operation function corresponding to the overlay.

In a possible implementation manner, the first information or the third information is located in an overlay description word of an adaptation set level or a representation level of the MPD.

It should be understood that when the first information or the third information is carried in the MPD, the server also needs to send the MPD corresponding to the media data including at least two overlays before sending the code stream to the terminal. The MPD includes first information or third information of each of at least two overlays. This embodiment of the present application does not limit this. The code stream here includes information of at least two overlays. The information of each of the at least two overlays is the information defined in the overlay structure corresponding to the overlay.

In a possible implementation manner, the group identification information of the overlay includes at least one group identification information. It should be understood that when the overlay corresponds to at least two group identification information, one overlay may belong to at least two groups.

In a possible implementation manner, the overlay corresponds to multiple groups. When the overlay is triggered, the overlay respectively responds to the operation function corresponding to the multiple groups, or all overlays in the multiple groups respectively respond to their respective groups. Corresponding operation function.

In a possible implementation manner, the overlay belongs to the first group and the second group, and different groups correspond to different operation functions. When the overlay in the first group is triggered, all The overlay responds to the operation function corresponding to the first group, and all overlays in the second group respond to the operation function corresponding to the second group. It should be understood that if the fifth information carried in the overlay is used to indicate that group operations and separate operations are available. When a group operation is triggered, the terminal processes at least two overlays according to the group operation. When a separate operation is triggered, the terminal processes at least two overlays according to the separate operation. The specific process can be described in the above corresponding description, which will not be repeated here.

In a possible implementation manner, when the overlay in the first group is triggered, the overlay responds to the operation function corresponding to the first group, and the overlay in the second group responds to the operation corresponding to the second group. Features.

In a possible implementation manner, the file format of the overlay further includes: an overlay group box, and the overlay group box carries name information of the overlay group; the method further includes: the terminal displays the overlay The group name indicated by the group's name information.

In a second aspect, an embodiment of the present application provides a method for processing media data, including: the server obtains the media data; the server processes the media data to obtain at least two overlays corresponding to the media data; the overlay corresponds to the first information, or the overlay Corresponds to second information and third information; wherein the first information includes group identification information of the overlay or identification information of other overlays that belong to the same group as the overlay, and the second information is used to indicate that the overlay corresponds to The third information is used to indicate the group identification information of the overlay or identification information of other overlays that belong to the same group as the overlay; the overlays in the same group correspond to the same operation function; the server sends The terminal sends the at least two overlay layers.

The processing of the media data by the server may refer to encoding the media data and encapsulating the media data stream obtained after the encoding.

In the embodiment of the present application, after the server obtains the media data, it can process the media data so that one or more overlays obtained after processing correspond to the group identification information, or at least two overlays corresponding to the group obtained after processing Group identification information, and operational functions. In this way, when the terminal obtains one or more overlays, it can process at least two overlays based on the first information, or process at least two overlays based on the second information and the third information. Since the first information and the third information both indicate group identification information, the terminal can process at least two overlays with the group granularity.

In a possible implementation manner, the second information and the third information are carried in an overlay file format. It should be understood that the server may encode the media data and use the video file format (for example, the OMAF standard file format) to encapsulate the media data stream obtained after the encoding. Therefore, the second information and the third information are carried in a file format obtained by encapsulating the overlay.

In a possible implementation manner, the file format includes an overlay structure, an overlay-related area control structure located in the overlay structure, and an overlay group box. The third information is located in the overlay group box, and the second information is located in the overlay association area control structure.

In a possible implementation manner, the first information is carried in an overlay file format. It should be understood that the server may encode the media data and use the video file format (for example, the OMAF standard file format) to encapsulate the media data stream obtained after the encoding. So that the first information is in a file format obtained after the overlay is encapsulated.

In a possible implementation manner, the file format includes an overlay group box, and the first information is located in the overlay group box. It should be understood that the file format at this time also includes an overlay structure.

In a possible implementation manner, the third information is carried in the auxiliary enhanced information SEI of the overlay code stream corresponding to the overlay, and the second information is carried in the overlay-related area control structure of the overlay. It should be understood that the server may encode the media data to obtain a media data stream, and then encode one or the overlay included in the media data to obtain an overlay code stream corresponding to each overlay. When encoding, the server carries the third information in the overlay corresponding to the overlay. The overlay bitstream includes auxiliary enhancement information SEI. Then, the encoded media data stream and the overlay code stream corresponding to each overlay are encapsulated in a video file format (for example, the OMAF standard file format). So that the second information is carried in a file format obtained by encapsulating the overlay.

In a possible implementation manner, the first information is carried in an SEI of an overlay code stream corresponding to the overlay. It should be understood that the server may encode the media data to obtain a media data stream, and then encode one or the overlay included in the media data to obtain an overlay code stream corresponding to each overlay. When encoding, the server carries the third information in the overlay corresponding to the overlay. The overlay bitstream includes auxiliary enhancement information SEI. Then, the encoded media data stream and the overlay code stream corresponding to each overlay are encapsulated. In this case, the encapsulated overlay has an overlay structure.

In a possible implementation manner, the payload type of the SEI is used to indicate that the SEI carries the group identification information of the overlay. It should be understood that the load type of the SEI may also be used to indicate the attributes of the group.

In a possible implementation manner, the third information is carried in a media presentation description MPD corresponding to the media data stream containing the overlay, and the second information is carried in an overlay-related area control structure of the overlay. It should be understood that the server may encapsulate the overlay based on the HTTP adaptive network adaptive media transmission protocol (Dynamic Adaptive Streaming Through HTTP, DASH) to obtain the MPD. The third information is then carried in the MPD. For the process of carrying the second information in the overlay-related area control structure of the overlay, refer to the foregoing description, and details are not described herein again.

In a possible implementation manner, the third information is located in an MPD's adaptation set level or representation level overlay description word.

In a possible implementation manner, the first information is carried in a media presentation description MPD corresponding to a media data stream containing an overlay. For the specific encapsulation process, refer to the process in which the third information is carried in the MPD, and details are not described herein again. In this case, the encapsulated overlay has an overlay structure.

In a possible implementation manner, the first information is located in an overlay description word of an adaptation set or a representation level of the MPD.

In a possible implementation manner, the overlay also corresponds to fourth information, and the fourth information is used to indicate that in a case where the first operation function is performed on the overlay, all overlays in a group to which the overlay belongs respond to the first Operational functions. It should be understood that the terminal can thus determine to operate in overlays in groups.

In a possible implementation manner, the overlay also corresponds to fifth information, and the fifth information is used to indicate that in a case where the first operation function is performed on the overlay, all overlays in a group to which the overlay belongs respond to the first An operation function, or the overlay responds to the first operation function. It should be understood that, in this way, the terminal can determine that the overlay can perform a group operation or an independent operation.

In a possible implementation manner, when the overlay corresponds to the first information, the overlay also corresponds to the sixth information, and the sixth information is used to indicate an operation function corresponding to the overlay.

In a possible implementation manner, the file format of the overlay further includes: an overlay group box, and the overlay group box carries name information of the overlay group.

It should be understood that the above-mentioned definitions and explanations of the corresponding content in the respective implementation manners in the first aspect are equally applicable to the respective implementation manners in the second aspect and the respective implementation manners in the third and fourth aspects described below.

In a third aspect, an embodiment of the present application provides a terminal, and the terminal includes a module for responding to the method in any one of the foregoing implementation manners of the first aspect.

It should be understood that a terminal is a device capable of presenting media data (eg, video images) and / or one or more overlays to a user.

In a fourth aspect, an embodiment of the present application provides a server, and the server includes a module for executing a method in any one of the foregoing implementation manners of the second aspect.

It should be understood that the server is a device capable of storing media data and processing one or more overlays corresponding to the media data. The server may provide video images and the processed one or more overlays to the terminal, so that the terminal can provide the media data, One or more overlays are presented to the user.

According to a fifth aspect, a terminal is provided, including: a non-volatile memory and a processor coupled to each other; wherein the processor is configured to call a program code stored in the memory to execute the method in any implementation manner of the first aspect Some or all of the steps.

According to a sixth aspect, a server is provided, including: a non-volatile memory and a processor coupled to each other; wherein the processor is configured to call program code stored in the memory to execute any one of the implementation manners of the second aspect Part or all of the steps of the method.

According to a seventh aspect, a computer-readable storage medium is provided, where the computer-readable storage medium stores program code, where the program code includes a part for performing a method in any implementation manner of the first aspect or Instructions for all steps.

According to an eighth aspect, a computer-readable storage medium is provided, where the computer-readable storage medium stores program code, where the program code includes a part for performing a method in any implementation manner of the second aspect or Instructions for all steps.

According to a ninth aspect, a computer program product is provided, and when the computer program product runs on a computer, the computer is caused to execute instructions of some or all steps of the method in any one of the implementation manners of the first aspect.

According to a tenth aspect, a computer program product is provided, and when the computer program product runs on a computer, the computer is caused to execute instructions of some or all steps of the method in any one of the implementation manners of the second aspect.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of a communication system according to an embodiment of the present application;

2 is a schematic structural diagram of a communication device according to an embodiment of the present application;

FIG. 3 is a first schematic flowchart of a media data processing method according to an embodiment of the present application; FIG.

4 is a second schematic flowchart of a method for processing media data according to an embodiment of the present application;

5 is a schematic diagram of a display interface according to an embodiment of the present application;

6 is a schematic diagram of another display interface according to an embodiment of the present application;

7 is a schematic structural diagram of a device for processing media data according to an embodiment of the present application;

FIG. 8 is a schematic structural diagram of another apparatus for processing media data according to an embodiment of the present application.

detailed description

Before introducing the embodiments of the application, the terms involved in the embodiments of the application are explained:

1) Panoramic video: Also known as 360-degree panoramic video, it is composed of a series of panoramic pictures. The content of the panoramic picture covers the entire sphere surface in three-dimensional space. It is a video shot with a full-scale 360-degree using a 3D camera. When watching a video, you can freely adjust the video to watch up, down, left and right.

2) Media presentation description (MPD): A document specified in the standard ISO / IEC 23009-1, which contains metadata for the client to construct the HTTP-URL. The MPD contains one or more period elements, each period element contains one or more adaptation sets, each adaptation set contains one or more representations, and each representation It contains one or more segments, and the client selects expressions based on the information in the MPD, and constructs the segmented HTTP-URL.

3), Track: Chinese translation "Track", the definition of Track in the standard ISO / IEC 14496-12 "timed sequence of related samples (qv) in an ISO media file. Translated into:" The relevant samples in ISO media files Time attribute sequence.

NOTE (Note): For the media data, a track, a sequence of images, or a sampled audio; for a track, a track, a stream of channels, a stream of channels. "Translates to:" For media data, a track is a sequence of images or audio samples; for a cue track, a track corresponds to a stream channel. "

Explanation: Track refers to a series of time-dependent samples in accordance with the ISOBMFF packaging method, such as video track. Video samples are code streams generated by the video encoder after encoding each frame. All video samples are encapsulated according to the ISOBMFF specification. Generate a sample.

4) Sample: Chinese translation "sample", data associated with timestamp. In ISO / IEC 14496-12, there are the following definitions and explanations:

"All the data associated with a single timestamp"

NOTE 1: No two samples within the track can share the same time-stamp.

NOTE2: In non-hint tracks, a sample sample, for example, an individual frame of video, a series of video frame frames in coding order, or a compressed section of audio section in audio coding in order; inhint tracks, a sample sample definitions of one or more streaming packets.

Corresponds to the following Chinese translation:

"All data associated with a single timeline"

Note 1. Two samples in the same trajectory cannot have the same time axis.

Note 2: In a non-cue track, the sample can be an independent video frame, a series of video frames placed in decoding order, or a compressed audio placed in the decoding order; in the cue track, the sample defines one or The shape of multiple stream packets.

5), box: Chinese translation of "box", the definition of box in the ISO / IEC 14496-12 standard: "object-oriented building block defined by unique type identifier and length. It can be translated as" object-oriented building block, Defined by a unique type identifier and length. "

NOTE: Called 'atom' in some specifics, including the first definition of MP4. "Can be translated as:" In some specifications, it is called 'atomic', including the first definition of MP4. "

ISOBMFF files are made up of multiple boxes, and boxes can contain other boxes.

6), SEI: full name supplementary enhancement, is a type of Network Abstract Unit (NALU) defined in the video codec standards (h.264, h.265).

7) Overlay: Chinese translation "overlay", that is, the media content superimposed on the background video (specifically, it can refer to an additional layer of rendered video or picture superimposed on a certain area of the background video picture), in the OMAF standard There are the following definitions and explanations:

"Piece of visual media rendering is overly omnidirectional video video image item viewport".

Chinese translation: visual media film superimposed and rendered on panoramic video, image item or perspective.

For example, the overlay can also be information such as the name and age of an element displayed on the background video.

8), background video (background visual media): video that can be superimposed by overlay. In OMAF, there are the following definitions and explanations: "piece of visual media, which is superimposed." Chinese translation: Visual media film superimposed by the overlay.

In this application, "of", corresponding "(corresponding, relevant)" and "corresponding" can sometimes be mixed. It should be noted that when the difference is not emphasized, the The meaning expressed is consistent.

It should be noted that, in the embodiments of the present application, words such as "exemplary" or "for example" are used as examples, illustrations or illustrations. Any embodiment or design described as “exemplary” or “for example” in the embodiments of the present application should not be construed as more preferred or more advantageous than other embodiments or designs. Rather, the use of the words "exemplary" or "for example" is intended to present the relevant concept in a concrete manner.

In the present application, "multiple" means two or more. "And / or" describes the association relationship of related objects, and indicates that there can be three kinds of relationships, for example, A and / or B can represent: the case where A exists alone, A and B exist simultaneously, and B alone exists, where A, B can be singular or plural. The character "/" generally indicates that the related objects are an "or" relationship. "At least one or more of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one (a), a, b, or c can be expressed as: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple . In addition, in order to facilitate a clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as “first” and “second” are used to distinguish between the same or similar items having substantially the same functions and functions. Those skilled in the art can understand that the words "first", "second" and the like do not limit the number and execution order, and the words "first" and "second" are not necessarily different.

As shown in FIG. 1, FIG. 1 shows a schematic diagram of a communication system provided by an embodiment of the present application. The communication system includes a server 100 and at least one terminal 200 that communicates with the server 100.

Wherein, the server 100 may be a media server having a function of processing panoramic video. The terminal 200 may be a device having a function of playing a panoramic video. For example, the terminal 200 may be an electronic device such as VR glasses, a mobile phone, a tablet, a television, and a computer that can be connected to a network. The terminal 200 receives the data sent by the media server, and decapsulates the code stream, and decodes and displays it.

The server 100 includes a pre-encoding processor 1001, a video encoder 1002, a code stream packaging device 1003, and a transmitting and transmitting device 1004.

Among them, the pre-encoding processor 1001 performs pre-processing on the panoramic video, such as image stitching, format conversion, etc., to convert the original panoramic video into a video that can be compression-encoded. The video encoder 1002 is used to obtain the panoramic video from the pre-encoding processor 1001. The video content is subjected to compression encoding or transcoding operation, and the encoded video bitstream is output. The bitstream encapsulation device 1003 encapsulates the encoded bitstream data into a transportable file and transmits it to the terminal or the content distribution network through the network. In addition, the server 100 may select the content to be transmitted for signal transmission according to the information (such as a user perspective) fed back by the terminal 200. Terminal 200: includes: receiving device 2001, stream de-encapsulation device 2002, video decoder 2003, and display device 2004

The receiving device 2001 is configured to receive media data sent by the server 100. The code stream decapsulating device 2002 is used for decapsulating the media data received by the receiving device 2001 to obtain a video code stream and code stream information corresponding to the code stream. The video decoder 2003 is used to decode a video code stream and output a video image frame for display and playback.

As shown in FIG. 2, FIG. 2 is a schematic diagram of a hardware structure of an apparatus for processing media data according to an embodiment of the present application. The apparatus for processing media data shown in FIG. 2 may be regarded as a computer device, and the apparatus for processing media data may be used as an implementation manner of the server 100 or the terminal 200 in the embodiment of the present application, or may be used as an embodiment of the embodiment of the present application. An implementation of the method for processing media data. The apparatus for processing media data includes a processor 110, a memory 120, an input / output interface 130, and a bus 150. Optionally, the apparatus for processing media data may further include a communication interface 140. It should be noted that when the apparatus for processing media data is used as the terminal 200 in the embodiment of the present application, the apparatus for processing media data may further include a display 160 for displaying video data to be played. For example, background video and one or more overlays.

Among them, the processor 110, the memory 120, the input / output interface 130, the communication interface 140, and the display 160 implement a communication connection with each other through the bus 150.

The processor 110 may use a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for executing related programs to To implement the functions required by the modules in the server in the embodiment of the present application, or to execute the method for processing media data in the method embodiment of the present application. The processor 110 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method may be completed by an integrated logic circuit of hardware in the processor 110 or an instruction in the form of software. The aforementioned processor 110 may be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a ready-made programmable gate array (Field Programmable Gate Array, FPGA), or other programmable logic device, Discrete gate or transistor logic devices, discrete hardware components. Various methods, steps, and logical block diagrams disclosed in the embodiments of the present application can be implemented or executed. A general-purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in combination with the embodiments of the present application may be directly implemented by a hardware decoding processor, or may be performed by using a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, and the like. The storage medium is located in the memory 120, and the processor 110 reads the information in the memory 120 and, in conjunction with its hardware, completes the functions required by the modules included in the server in the embodiment of the present application, or performs processing of media data in the embodiments of the method of the present application method.

The memory 120 may be a read-only memory (Read Only Memory, ROM), a static storage device, a dynamic storage device, or a random access memory (Random Access Memory, RAM). The memory 120 may store an operating system and other application programs. When software or firmware is used to implement the functions required by the modules included in the server in the embodiment of the present application, or the method for processing media data in the method embodiment of the present application, the method for implementing the technical solution provided in the embodiment of the present application is implemented. The program code is stored in the memory 120, and the processor 110 performs operations required by the modules included in the server 100, or executes the method for processing media data provided by the method embodiment of the present application.

The input / output interface 130 is used to receive input data and information, and output data such as operation results.

The communication interface 140 uses a transceiving device such as, but not limited to, a transceiver to implement communication between a device that processes media data and other devices or a communication network. It can be used as an obtaining module or a sending module in a device for processing media data.

The bus 150 may include a path for transmitting information between various components of a device that processes media data, such as the processor 110, the memory 120, the input / output interface 130, and the communication interface 140.

It should be noted that although the apparatus for processing media data shown in FIG. 2 only shows the processor 110, the memory 120, the input / output interface 130, the communication interface 140, and the bus 150, in the specific implementation process, those skilled in the art It should be understood that the apparatus 100 also includes other devices necessary for achieving normal operation. At the same time, according to specific needs, those skilled in the art should understand that the apparatus for processing media data may further include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the apparatus for processing media data may also include only the components necessary to implement the embodiments of the present application, and not necessarily all the components shown in FIG. 2.

For example, the apparatus for processing media data may further include one or more network cards for forming a session channel between the server 100 and the terminal 200 to transmit media services.

The overlay in the embodiments of the present application refers to the overlay media content superimposed on the background layer media content, and the overlay may be separately encoded as the media content or may be a part of the background layer media content. If the overlay is part of the media content of the background layer, the overlay may not be separately encoded, and the media data code stream obtained by the server after the media data is encapsulated will include the overlay information. If the overlay can be separately encoded as media content, the overlay codestream corresponding to each overlay will be obtained.

The media content is content displayed by playing media data.

Before introducing the embodiments of the present application, first introduce the overlay structure of each overlay in the embodiments of the present application:

In the current OMAF standard document, the basic data structure of the overlay (abbreviated as overlay structure) and the carrying method have been defined, as shown in Table 1 below:

Table 1

The overlay structure shown in Table 1 defines some basic attributes of the overlay structure, including the number of overlays (number, abbreviation: num), identification information (for example, Id number), overlay control symbols, and overlay control structure. Wait. The value of the overlay control symbol syntax element overlay_control_flag can be used to indicate the function of the overlay control structure. For example, the semantics of overlay_control_flag include the overlay's associated source, hierarchical order, transparency, user operation information, flags, and priorities, as shown in Table 2:

Table 2 Overlay control symbol semantics defined in OMAF

When the i value of overlay_control_flag is 7, and the value of the i-th bit is 1, an interactive control structure (OverlayInteraction control structure) is defined. It can be understood that the OverlayInteraction control structure is one of the overlay control structures. Among them, the OverlayInteraction control structure contains the types of interaction that the overlay may be operated by the user. The structure is shown in Table 3:

table 3

For example, the semantics of the syntax elements involved in Table 3 are shown in Table 4 below:

Table 4

It should be understood that Table 4 is only a list of some operation functions. In the actual process, there may be other operation functions for overlay, and of course, there may be other operation functions.

As shown in FIG. 3, FIG. 3 shows a schematic flowchart of a method for processing media data according to an embodiment of the present application. The method includes:

Step 101: The server obtains media data.

Optionally, the foregoing media data may be a video image, for example, a panoramic video. The one or more overlays corresponding to the media data may be one or more overlays displayed on the media data. For example, the overlay layer may be a video or a picture displayed on the media data. For example, if the content of the media data is the player as an example, the picture overlaid on it may be a name or an age.

Step 102: The server processes the media data to obtain at least two overlay layers corresponding to the media data.

The overlay layer is a video, image, or text that is used to be superimposed on a background video or a background image for display.

It should be understood that the processing of media data includes operations such as preprocessing, encoding, and encapsulation of the media data.

An example is that the overlay corresponds to the first information.

Exemplarily, the first information includes group identification information of the overlay or identification information of other overlays that belong to the same group as the overlay.

For example, the identification information of other overlays is used to determine other overlays that belong to the same group as the overlay. For example, the identification information of other overlays corresponding to overlay1 is overlay1 and overlay2. This means that overlay1, overlay2, and overlay3 belong to the same group.

In another example, the overlay corresponds to the second information and the third information.

The first information and the third information are used to determine a group of the overlay, respectively.

Exemplarily, the second information is used to indicate an operation function corresponding to the overlay. The third information is used to indicate group identification information of the overlay or identification information of other overlays that belong to the same group as the overlay.

Exemplarily, the group identification information of the overlay in the same group is the same.

It should be understood that all overlays in the same group have at least one same operation function.

It should be understood that the operation functions corresponding to overlays in the same group have the same meaning: all operation functions corresponding to all overlays included in the same group are all the same. For example, overlay1 and overlay2 belong to group 1, and the operation functions corresponding to overlay1 and overlay2 both include rotation and the size of the window can be changed.

The same operation function corresponding to overlays in the same group means that at least one operation function corresponding to all overlays included in the same group is the same. For example, the operation functions corresponding to overlay1 include rotation and the size of the window can be changed. The corresponding operation functions of overlay2 include rotation. When the server processes media data, overlay1 and overlay2 can also be divided into group 1.

Optionally, the group identification information is used to determine a group to which the overlay belongs. The group identification information may be a group ID or a group name, which is not limited herein.

It should be understood that each overlay in the embodiment of the present application includes an overlay structure, and the overlay structure includes indication information for indicating an overlay operation function. For example, the operation function can be determined through the OverlayInteraction control structure. For example, rotation, free selection depth, window size can be changed, and so on.

Specifically, the server encodes the media data to obtain one or more overlays included in the media data stream, and then determines at least one operation function that each overlay has when the media data stream is encapsulated, so that for any two or two In the above overlay, if any two or more overlays have at least one of the same operation functions, the server may set the group identification information of each of them to be the same. For example, if the operation function corresponding to overlay1 and overlay2 is rotation, the server may use the first information / third information corresponding to overlay1 and overlay2 to indicate group 1.

In addition, the overlay control structure also defines a control structure of an area (eg, a spherical area) associated with the overlay, which is used to indicate that when an area in a video image is triggered, the overlay display associated with the area can be triggered. For example, the control structure of the area associated with the overlay is an overlay associated area control structure (AssociatedSphereRegionStruct) as an example. The syntax of AssociatedSphereRegionStruct is shown in Table 5 below:

table 5

SphereRegionStruct (1) in Table 5 defines a spherical area associated with the overlay.

When the spherical area defined in Table 5 appears in the user's perspective, depending on the configuration of the client or the prompt of the user interface, the user can click the spherical area to trigger the overlay associated with the spherical area. On or off.

It should be understood that the above-mentioned area of the overlay may refer to an area just covered or occupied by the area of the overlay, that is, the media data in the area of the overlay belong to the overlay, and the media data in the overlay are all in the area of the overlay.

The above-mentioned area spatial information of the overlay may also be referred to as area spatial information of the area of the overlay. The area spatial information of the overlay is used to indicate the spatial range or spatial position of the area associated with the overlay. In this way, when the user is watching a video image, the area associated with the area can be displayed in the video image by triggering the area.

The above-mentioned spatial position of the area associated with the overlay may specifically be directed to a coordinate system, and the coordinate system may be a three-dimensional coordinate system or a two-dimensional coordinate system. For example, when a three-dimensional coordinate system is used to represent the spatial position of the area associated with the overlay, the origin of the three-dimensional coordinate system may be the center point of the panoramic video image, the point in the upper left corner of the panoramic video image, or other fixed position points in the panoramic video image. In addition, the spatial position of the area associated with the overlay may also be the position of the overlay in the panoramic video image area. Spatial location).

The embodiments of the present application may be applicable to the following scenario 1 and scenario 2:

Scenario 1. One or more overlays need to be displayed or closed together in a triggered manner, then one or more overlays can be added to a group (the name of the group is used as a common display group as an example below. It can be understood that The group can also have other names). Then each overlay in the one or more overlays corresponds to the second information and the third information.

Scenario 2: You can add one or more overlays with a certain type of interaction to a group based on one or more overlays with the interaction defined in the OverlayInteraction control structure (you can call this group name for interaction) For: interaction group, it should be understood that the group for interaction operation may also be in another name). This enables the terminal to perform a certain type of defined operation function, such as an interactive operation, based on a trigger operation on the interaction group. Exemplarily, the interactive operation may be shown in Table 4, and details are not described herein again. At this time, each of the one or more overlays corresponds to the first information.

As a first possible implementation manner, step 102 in the embodiment of the present application may be specifically implemented in the following manner: S1. The server encodes the media data to obtain a media data code stream corresponding to the media data. S2. The server encapsulates the media data stream obtained after encoding. The encapsulated media data stream includes information of one or more overlays, and first information corresponding to each overlay in the one or more overlays. Or the second information and the third information corresponding to each overlay. Each overlay in the one or more overlays corresponds to a file format.

It should be understood that the overlay in the following embodiments may be part of the media content (ie, media data) of the background layer, and the overlay may not be separately encoded at this time. That is, when the server encodes the media data, the obtained media data stream includes one or more overlays. The server may then encapsulate the media data stream including one or more overlays. For example, make the encapsulated media data stream correspond to the file description. Or the server encapsulates the media data stream so that one or more overlays included in the media data stream have an overlay structure.

In the following embodiments, the overlay can also be separately encoded as media content. At this time, the server encodes the media data to obtain a media data stream, and then encodes the overlay included in the media data to obtain an overlay code stream. When the server encapsulates the media data stream and the overlay code stream, the encapsulated media data stream has overlay information. The overlay information may be an overlay structure.

Corresponding to scenario 1, the second information and the third information may be carried in a file format of an overlay that encapsulates a media data stream including the overlay.

The file format includes: an overlay structure, and an overlay-related area control structure and an overlay group box located in the overlay structure. The third information is located in the overlay group box, and the second information is located in the overlay associated area control structure. At this time, it can be understood that the second information may be an overlay associated area control structure. For the operation function of the control structure of the overlay associated area, refer to the above description. Exemplarily, the operation function indicated by the third information may be display or shutdown.

Corresponding to scenario 2, the first information may be carried in the file format of the overlay. That is, one or more overlay file formats obtained after encapsulation may not have an overlay-related area control structure.

The file format includes: overlay group box. The first information is located in an overlay group box. It should be understood that in scenario 2, the file format may also include an overlay structure.

Exemplarily, the server may encapsulate the media data stream including one or more overlays according to the OMAF standard file format. In the encapsulated file, the file format of each overlay in the one or more overlays has an overlay control structure. In addition, for scenario 1, the overlay structure may have an overlay-related area control structure. For scenario 2, the overlay structure may be provided with an OverlayInteraction control structure.

Specifically, during the encapsulation process, the server may add a box corresponding to the overlay control region control structure to the file format, so that the overlay file format has an overlay control structure.

In addition, in OMAF, entity groups are defined for multiple overlays. A type of group (eg, a switching group) including selective switching is defined. Multiple overlays in this switching group can be switched to each other. The specific syntax for switching groups is shown in Table 6 below:

Table 6

Among them, ref_overlay_id [i] represents identification information of other overlays that belong to the same group as an overlay.

In addition, for scenario 2, the server can also make it possible to define identification information of other overlays that belong to the same group as the overlay in the overlay structure of each overlay, so as to replace the above-mentioned group identification information.

The identification information of the overlay is used to identify the overlay. The identification information may be an ID number of the overlay.

It should be noted that if there is only one overlay in a group, identification information of other overlays may not be defined in the overlay.

For example, overlay1 belongs to group1 and overlay2 also belongs to group1. Therefore, the identification information of group1 and the identification information of overlay2 can be defined in overlay1. In overlay2, identification information of group 1 and identification information of overlay1 can be defined.

Wherein, by defining the group in which the overlay is located, when any overlay in the group performs an operation function corresponding to the overlay, all overlays in the group respond to the operation function.

In the embodiment of the present application, there may be an entity group and an overlay structure in a file format corresponding to each overlay. Among them, the entity entity group box EntityToGroupBox.

The file format in the embodiment of the present application further includes an overlay group box. The overlay group box is used to indicate an operation function corresponding to the overlay when overlaying any one of the overlay group boxes. The overlays all respond to this operation function.

For the specific name of the overlay group box, refer to the function name of the overlay control structure in the overlay group box. For example, if the overlay control structure is an overlay-related area control structure, the overlay group box can be defined as an OverlayConditionalShownGroupBox, which means that a group of overlays can be displayed or closed together when the user targets and triggers a certain overlay. For example, for an interactive operation, that is, an overlay group box with an OverlayInteraction control structure may be an OverlayRelationGroupBox.

It should be understood that the OverlayConditionalShownGroupBox and OverlayRelationGroupBox in the overlay group box in the embodiment of the present application may also have other names, which are not limited in the embodiment of the present application.

Exemplarily, there may be an overlay group box in the entity groups.

Exemplarily, if all overlays in a group have an overlay-related area control structure, the group may be named after a common display group. The common display group may indicate that multiple overlays in the common display group may be displayed or closed together. Or, the first prompt information corresponding to the group is used to indicate that the group can be expanded or closed together. If you perform interactive operations on all overlays in a group, you can define the group name as an interactive group, or the second prompt information corresponding to the group, which is used to prompt the group to perform the group together. The interactive operation indicated by the corresponding operation type. It can be understood that this is only an example, and the name of the group may also be another name, which is not limited in the embodiment of the present application.

Example 1-1. Taking the common display group as an example, the overlay group box in the embodiment of the present application may be OverlayConditionalShownGroupBox, which means that a group of overlays can be performed when the user triggers the display for any overlay or the group Show together. At this time, the third information may be carried in the OverlayConditionalShownGroupBox.

When the user closes the group or any overlay in the group, a common close is performed. The specific syntax is shown in Table 7:

Table 7

The ref_overlay_id [i] in the above Table 7 indicates that the overlay_id corresponding to the track or image item indicated by the i-th entity_id is an overlay that can be displayed under the trigger of the user in this group. There will be an overlay_id corresponding to ref_overlay_id [i] in the referenced i-th track or image item. When each track or image item represented by entity_id in this entity group contains only one overlay, then the ref_overlay_id [i] syntax element in the structure is also allowed to exist.

Use OverlayConditionalShownGroupBox to add multiple overlays to the same group.

Example 2-1. Taking interactive groups as an example, the file format of each overlay in one or more overlays obtained by the server after processing the media data has the group identification information of the overlay.

The overlay group box in the embodiment of the present application may be an OverlayRelationGroupBox, which is used to form multiple overlays into an interaction group, and all overlays in the interaction group may have the same interaction operation. At this time, the first information is carried in the OverlayRelationGroupBox.

It should be noted that the same interaction group specifies an interaction operation that can perform a certain type of operation function instruction for all overlays in the interaction group. When performing an operation function corresponding to the OverlayRelationGroupBox for any overlay in the OverlayRelationGroupBox, the other overlays in the OverlayRelationGroupBox also respond to the operation function corresponding to the OverlayRelationGroupBox. The specific syntax is shown in Table 8:

Table 8

Optionally, regarding the interactive information syntax element included in the OverlayInteraction control structure, when there are multiple overlays forming an interaction group OverlayRelationGroupBox. If any overlay in the interaction group is triggered, the operation functions defined in the OverlayInteraction control structure will be applied to each overlay in the interaction group together.

And the value of the syntax element of the OverlayInteraction control structure corresponding to the operation function defined by the OverlayRelationGroupBox of each overlay in the interaction group is the same.

For example, the OverlayRelationGroupBox defines an operation function for scaling all overlays in the OverlayRelationGroupBox as an example.

At this time, if overlayA, overlayB, and overlayC in the OverlayRelationGroupBox form interaction group 1, then resize_flag = 1 in the OverlayInteraction control structure corresponding to overlayA, overlayB, and overlayC, respectively. When this interaction group 1 is triggered, overlayA, overlayB, and overlayC will be scaled. Similarly, when the OverlayRelationGroupBox definition is an operation function for changing the position of all overlays in the OverlayRelationGroupBox, change_position_flag = 1 in the OverlayInteraction control structure corresponding to overlayA, overlayB, and overlayC respectively. When the group to which overlayA, overlayB, and overlayC belong is triggered, overlayA, overlayB, and overlayC will perform a position change operation.

It should be noted that all overlays of the same interaction group will perform the same operation function.

Exemplarily, the overlay in the embodiment of the present application can be displayed together with the background video, and the overlay and the background video can be bound for common display. For example, the syntax structure of overlay and background video is shown in Table 9:

Table 9

The second possible implementation manner, step 102 in the embodiment of the present application may be specifically implemented in the following manner: S3. The server encodes the media data to obtain a media data stream, and encodes one or more overlays included in the media data to obtain An overlay code stream corresponding to the overlay, and each overlay code stream includes a SEI.

S4. The server encapsulates the media data stream and the overlay code stream corresponding to each overlay to obtain a media data stream including one or more overlay information.

It should be noted that when the overlay can be separately encoded as the media content, the server can also separately encapsulate the overlay code stream. Then send the encapsulated overlay code stream to the terminal.

The SEI payload type is used to indicate that the SEI carries overlay group identification information.

Corresponding to scenario 1, the third information may be carried as an indication field in the SEI of the overlay code stream. Alternatively, the first information may be used as an SEI of an overlay code stream. At this time, the SEI has an indication field for indicating the group identification information of the overlay.

It should be understood that when the server executes S4, the encapsulated overlay may also include: an overlay associated area control structure. The specific encapsulation process can refer to the above S2, which will not be repeated here.

For example, the third information is carried in the SEI of the overlay code stream corresponding to the overlay, and the second information is carried in the overlay associated area control structure of the overlay.

It can also be understood that in the second implementation manner, the SEI corresponding to the scenario 1 may be named after a common group when carrying the group identification information, so that the second information may not be carried, that is, it is not defined in the overlay structure of the overlay overlay associated area control structure.

For the manner of carrying the third information, refer to the description in the first possible implementation manner, and details are not described herein again.

Corresponding to scenario 2, the first information may be carried as an indication field in the SEI of the overlay code stream corresponding to the overlay. At this time, the encapsulated overlay may not have an overlay-related area control structure. At this time, in order that the terminal can know the operation function of the overlay after receiving the overlay, when the SEI carries the group identification information, the operation function corresponding to each overlay in the group can be used to name the group.

For example, the first information is carried in the SEI of the overlay code stream corresponding to the overlay.

The SEI is used to indicate group identification information of the overlay. For example, the syntax structure of the SEI is shown in Table 10:

Table 10

The sei_payload in Table 10 defines the SEI payload information, including two parameters payloadType and payloadSize. Among them, the payloadType indicates the type of the SEI, and the payloadSize indicates the size of the SEI.

Among them, OLG in Table 10 is a variable, which represents the value of the payloadType of an SEI. For example, the value of OLG may be 190. For the following references to OLG, you can refer to the description here, and will not repeat them later. The embodiment of the present application does not limit the specific value of OLG. payloadSize indicates the payload size.

In Example 3-1, taking an operation function as a display or a shutdown as an example, an overlay group can be represented as an overlay condition display group (overlay_conditional_shown_group).

Therefore, the group identification information of the overlay in Table 10 may be replaced with overlay_conditional_shown_group_info (information). For example, the syntax structure of overlay_conditional_shown_group_info is shown in Table 11:

Table 11

overlay_conditional_shown_group_id This value indicates the ID number of the group of the overlay.

Example 4-1, taking the operation function as an interactive operation, the overlay group can be overlay_relation_group, and overlay_relation_group_info can be used to replace the overlay group identification information in Table 10.

The above interactive operations may refer to a common operation on a certain type of operation function, or a common operation on all operation functions supported by the overlay, which are not limited in the embodiments of the present application.

A third possible implementation manner, step S102 in the embodiment of the present application may be specifically implemented in the following manner: S5. Encode the media data to obtain a media data stream including one or more overlays. S6. The server encapsulates a media data stream including one or more overlays, and obtains a description file corresponding to the media data stream.

Specifically, for implementation of S5, reference may be made to S1, and details are not described herein again.

Exemplarily, S6 may be specifically implemented in the following manner: The server may encapsulate a media data stream including one or more overlays based on a DASH transmission protocol standard to obtain a media presentation description MPD of the media data stream as a description file. The overlay descriptor of the MPD's adaptation, level, or representation level carries the group identification information of the overlay.

Corresponding to scenario 1, each overlay in the one or more overlays corresponds to the second information and the third information. At this time, the description file includes at least third information of each overlay in one or more overlays, and the third information may be carried as an indication field in the description file of the media data stream. It should be understood that after the media data stream is encapsulated, one or more overlays included in the media data stream have an overlay associated area control structure. The specific encapsulation process can refer to the above S2, which will not be repeated here.

For example, the third information is carried in an MPD corresponding to a media data stream obtained by encapsulating a media data stream including one or more overlays, and the second information is carried in an overlay-related area control structure of the overlay.

In a third possible implementation manner, corresponding to scenario 1 can also be replaced in the following manner, that is, the overlay group in the description file containing the media data of the overlay is named after the operation function of the overlay. Has an overlay associated area control structure.

For the manner of carrying the second information, reference may be made to the description of the first scenario in the second possible implementation manner, and details are not described herein again.

Corresponding to scenario 2, each overlay in one or more overlays corresponds to the first information. At this time, the first information may be carried as an indication field in a description file containing the media data of the overlay. It can be understood that the name of the group can also be named by the operation function of the code stream.

For example, the first information is carried in a media presentation description MPD of the media data obtained by encapsulating the media data including the overlay.

In summary, the first information or the third information is located in an overlay description word of an adaptation set level or a representation level of the MPD.

Example 5-1. Take the operation function as the display or shutdown as an example, so you can define a new @schemeIdUri for the overlay descriptor, the value is: "urn: mpeg: mpegI: omaf: 2018: ocsg", the semantics is common to overlay OCSG descriptor. A maximum of one OCSG descriptor is allowed to appear at the adaptation level or the representation level.

The value of the OCSG descriptor is a comma-separated string. The specific values and semantics are defined in Table 12 below:

Table 12

Among them, M represents a required parameter, and O represents an optional parameter. An adaptation set that has the same overlay_relation_group_id value belongs to the same interaction group. The values in the adaptation set that belong to different groups can be different.

Exemplarily, Table 13 shows an example of an MPD carrying a group indicating an overlay as a common display group:

Table 13

Table 13 describes two common display groups, namely "urn: mpeg: mpegI: omaf: 2018: ocsg" value = "1" (can be referred to as common display group 1) and "urn: mpeg: mpegI: omaf: 2018: ocsg "value =" 2 "(may be referred to as common display group 2. Each of the two common display groups shown in Table 13 includes: two overlays. For example, a common display group Group 1 includes overlay1 and overlay2, and together shows that group 2 includes overlay3 and overlay4.

Example 6-1. Take the common interaction group of the group where the overlay is located as an example. The server can define a new @schemeIdUri for the overlay descriptor, whose value is: "urn: mpeg: mpegI: omaf: 2018: ovly", the semantics is overlay common interaction grouping information (OVLY) descriptor, which describes the group for the overlay. Groups perform some kind of interaction. If the position of the overlay moves. A maximum of one OVLY descriptor can appear at the adaptation level or the representation level.

The value of the OVLY descriptor is a string separated by commas. The specific values and semantic definitions are shown in Table 14 below:

Table 14

It should be understood that the adaptation sets having the same overlay_relation_group_id value belong to the same common interaction group, and the values in the adaptation sets belonging to different groups must be different.

Exemplarily, Table 15 shows an example of an MPD carrying a group indicating that the overlay is a common interaction group:

Table 15

Table 15 describes two common interaction groups, namely "urn: mpeg: mpegI: omaf: 2018: ovly" value "=" 1 "(can be referred to as common interaction group 1 for short) and" urn: mpeg: mpegI : omaf: 2018: ovly "value =" 2 "(can be referred to as common interaction group 2. For each of the two common interaction groups shown in Table 15, there are two overlays. For example, the common interaction group 1 includes overlay1 and overlay2, and common interaction group 2 includes overlay3 and overlay4.

It should be understood that after determining the operation function that each overlay has, the server may determine that one or more overlays having the same operation function belong to the same group. It can also be understood that if two or more overlays have the same operation function, two or more overlays with the same operation function can be divided into the same group. And the group can be named after the two or two overlays have a common operation. At this time, the group can correspond to an operation option, which is used to prompt the operation functions that the overlay in the group has in common.

For example, if overlay1 corresponds to rotation and reduction, and overlay2 corresponds to rotation, the server may carry identification information indicating group 1 in overlay1 and overlay2. It should be understood that the operation option corresponding to group 1 is used to indicate the operation function shared by overlay1 and overlay2.

It should be understood that the server may determine the respective operation function of each overlay through the respective overlay control structure of each overlay.

Step 103: The server sends one or more overlays to the terminal.

For example, the server may send one or more overlays to the terminal through a transmitting and transmitting device.

It should be noted that, in the embodiment of the present application, after the server processes one or more overlays, it may directly send the processed one or more overlays to the terminal. It is also possible to send the processed one or more overlays after receiving a request message for requesting an overlay sent by the terminal.

It should be understood that, for the foregoing first possible implementation manner and the second possible implementation manner, the first information, the second information, and the third information are included in the overlay. For the third possible implementation manner, the first information, the second information, and the third information are included in the MPD file. When the service uses the third possible implementation manner to process the media data, it is understood that the one or more overlays sent by the server in S103 also send the MPD corresponding to the one or more overlays to the terminal. The MPD corresponding to one or more overlays includes information of each overlay.

It should be noted that in the embodiment of the present application, when the first information or the third information is carried in the SEI, it may be carried in the SEI of the overlay code stream corresponding to the overlay. If the server sends an overlay stream to the terminal, the terminal can display the overlay when decoding and playing the overlay stream.

Step 104: The terminal receives one or more overlays sent by the server.

Exemplarily, the terminal may receive one or more overlays sent by the server through a receiving device. It should be understood that the one or more overlays sent by the server may be implemented in the following manner: the server sends the encapsulated media data stream and the one or more overlays included in the media data stream to the terminal. Or, the server sends an overlay code stream corresponding to each overlay in the encapsulated one or more overlays to the terminal.

It should be understood that if the server carries the first information in the MPD of the included media stream of one or more overlays, or carries the second information and the third information in the MPD of the included media stream of one or more overlays In step 104, the terminal also needs to receive the MPD of the included media data stream of one or more overlays.

Specifically, when the first information is located in the overlay group box, the terminal can obtain the group identification information from the overlay group box in the file format by analyzing the file format of the overlay. When the third information is located in the overlay group box, the terminal can obtain the group identification information from the overlay group box in the file format by analyzing the file format of the overlay.

Step 105: When the overlay corresponds to the first information, the terminal processes at least two overlays according to the first information of the at least two overlays; or, when the overlay corresponds to the second information and the first information, In the third information, the terminal processes the at least two overlays according to the second information and the third information corresponding to the at least two overlays.

Specifically, S105 may be implemented in the following manner: After receiving one or more overlays sent by the server, the terminal decapsulates the overlays to obtain first information corresponding to one or more overlays. Or after the terminal is decapsulated, the second information and the third information corresponding to one or more overlays are obtained. Then when the terminal decodes and plays media data, it can include in the client configuration or user interface prompts an operation option corresponding to the overlay in the same group, which is used to prompt all overlays in the group to perform common operations. Operational functions.

It should be understood that when each overlay corresponds to the first information, the terminal may determine each group of each overlay according to the first information corresponding to each overlay, and then may determine all overlays belonging to the same group. In addition, for interactive operations, the server may determine the interactive operation corresponding to each overlay according to the OverlayInteraction control structure of each overlay.

When each overlay corresponds to the third information and the second information, the terminal may determine a respective group of each overlay according to the third information corresponding to each overlay. You can then determine all overlays that belong to the same group. The terminal can determine the corresponding display or close operation function of each overlay according to the AssociatedSphereRegionStruct of each overlay.

In one example, the terminal may determine all overlays belonging to the same group in the following manner: The terminal divides the overlays with the same group identification information into the same group according to the group identification information corresponding to each overlay.

For example, the group identification information of overlay1 is group1, the group identification information of overlay2 is group2, the group identification information of overlay3 is group1, and the group identification information of overlay4 is group2. Then, the terminal may determine that there are two groups, that is, group 1 and group 2.

In another example, the terminal may determine all overlays belonging to the same group by: the terminal indicates the identification information of any overlay and other overlays corresponding to any overlay according to the identification information of any overlay corresponding to any overlay The other overlays are grouped into the same group.

As a possible embodiment, as shown in FIG. 4, the method provided in this embodiment of the present application further includes:

Step 106: The terminal displays at least one group, and information indicating an operation function corresponding to each group in the at least one group and an overlay belonging to each group.

In a possible implementation manner, at least one group is determined by first information corresponding to each overlay in at least two overlays; an operation function corresponding to one group is determined by an overlay structure included in the overlay in each group.

In another possible implementation manner, at least one group is determined by third information corresponding to each overlay in the at least two overlays, and an operation function corresponding to one group is an overlay associated area included in the overlay in the group The control structure is determined.

It should be understood that, when displaying at least one group, the terminal may also decode and play the received media data stream to display the media data. The at least one group may be displayed overlaid on the media data.

Step 107: When any group in at least one group is triggered, all overlays belonging to any one group respond to the operation function corresponding to the group. Or when any overlay is triggered, any overlay and other overlays belonging to the same group as any overlay respond to the operation function of any overlay being triggered.

It should be understood that the operation function corresponding to a group is determined by the operation function shared by all overlays in the group.

If two or more overlays in a group have multiple common operating functions, then a group can correspond to multiple operating functions. When any group is triggered, any group All overlays respond to the group's triggered operation functions. It should be understood that if multiple operation functions corresponding to the group are triggered, all overlays in any one group respond to the multiple operation functions. If any one of a plurality of operation functions corresponding to the group is triggered, all overlays in any one group respond to any one of the triggered operation functions.

For example, overlay1 and overlay2 belong to group 1, where the operation functions corresponding to overlay1 and overlay2 are rotation and size scaling. The operation function corresponding to group 1 is also rotation and size scaling. If both rotation and size scaling are triggered, overlay1 and overlay2 respond to the rotation and size scaling operations. If the triggered operation function is rotation, overlay1 and overlay2 respond to the rotation operation.

For example, in the embodiment of the present application, the terminal may assign an operation option to each group, and the operation option is used to prompt an operation function that all overlays in the group can respond to.

It should be understood that if a group corresponds to multiple operation functions, multiple operation options can be assigned to the group, and each operation option corresponds to one operation function. In addition, each group may be assigned an operation option 1 for instructing execution of all operation functions. When the operation option 1 is triggered, if the group has multiple operation functions, all overlays in the group respond to multiple operation functions.

If all overlays in the same group have multiple common operating functions, multiple overlay options can also be given to each overlay. For the specific process, refer to the process in which a group corresponds to multiple operation functions, and multiple operation options can be given to the group, which will not be repeated here.

It should be understood that when the user does not trigger the group and the user's operation is on the group, the user may be prompted for all overlays included in the group and the operation functions common to all overlays in the group. For example, when the mouse is on the operation option, but the click operation is not triggered, the user can also be prompted for all overlays included in the group and the operation functions common to all overlays in the group.

Example 1-2, corresponding to Example 1-1, as a possible implementation manner, step 105 may be specifically implemented in the following manner:

The terminal parses each overlay to obtain the respective AssociatedSphereRegionStruct of each overlay. The server determines whether each overlay's respective operation function is displayed or closed according to the AssociatedSphereRegionStruct. The terminal parses the entity group in the media data stream containing one or more overlays, and obtains the OverlayConditionalShownGroupBox.

Since the OverlayConditionalShownGroupBox can also be used to indicate identification information of other overlays that belong to the same common display group as an overlay, the terminal can further obtain ref_overlay_id. Therefore, the terminal can determine whether the operation function that can be performed on the one overlay and other overlays in the same common display group as the overlay is display or close.

Here, as a possible implementation manner, step 106 may be specifically implemented in the following manner (1-1):

In the method (1-1), when the terminal decodes the media data and plays the video on the display interface, the client configuration or the user interface prompt may include operation options for triggering the display or closing of the common display group.

It should be understood that the media data may be displayed on the client or user interface of the terminal.

Step 107 may be specifically implemented in the following manner (1-2) or (1-3):

Method (1-2). When any group is triggered, the terminal displays all overlays in the any group. That is, all overlays in the group are displayed on the display interface.

In the method (1-3), if multiple overlays in the first group are displayed on the display interface, when an operation option corresponding to the first group is triggered. Or, if any overlay is triggered, the terminal closes all overlays in the first group. That is, all overlays in the first group are canceled.

It should be understood that all overlays that are canceled are in the first group. As shown in Figure 5.

It should be understood that the operation option may be displayed on the display interface in the form of icons or text. When the operation option is displayed in the form of a chart, when the user's touch operation or click operation is located on the chart or near the chart, the text used to prompt the corresponding function of the operation option can be displayed on the display interface.

It should be understood that the foregoing operation option or overlay may be triggered by a touch operation or a click operation.

Exemplarily, taking the common display group as an example, as shown in FIG. 5, an operation option of triggering display or closing the display may be performed on the overlay in the common display group in the client configuration or the user interface prompt of the terminal. For example, group 1 in FIG. 5 corresponds to operation option 1 and group 2 corresponds to one operation option 2. In FIG. 5, the display mode of the operation options is text as an example.

Specifically, if only the first group is displayed on the display interface and all overlays in the first group are not displayed, for example, in FIG. 5, when an operation option corresponding to the first group is triggered, the terminal controls the first group. All overlays in a group perform operational functions.

Exemplarily, the common display is taken as an example, and the first group is taken as the group 1 shown in FIG. 5 as an example. When the operation option 1 of the group 1 is triggered, the terminal displays the information in the group 1 on the display interface. All overlays. As shown in FIG. 6, a touch operation is used as an example for the operation option 1 of the group 1 being triggered. When the operation option 1 of the group 1 is triggered, as shown in FIG. 6, the display of the group 1 includes : Name corresponding to media content 1, name 2 corresponding to media content 2, and name 3 corresponding to media content 3.

It can be understood that when all overlays in group 1 are displayed on the display interface, when the close operation function corresponding to the operation option of group 1 is triggered, the terminal closes all overlays included in group 1 and uses the group to 1 way is displayed on the display interface. That is, in response to closing the operation function of the display, the display interface at this time may be as shown in FIG. 5.

Specifically, when one or more overlays are not displayed on the display interface, if the one or more overlays that can be displayed are included in a group. When the corresponding operation option of the group is triggered, the terminal can simultaneously display the one or more overlays on the background video in response to the user's trigger operation. When one or more overlays are displayed on the display interface, when the corresponding operation option of the group is triggered or any overlay is triggered, the one or more overlays can be closed at the same time in response to the user's trigger operation.

Example 2-2 corresponds to Example 2-1. As a possible implementation manner, step 105 can be specifically implemented in the following manner:

The terminal parses the overlay structure of each overlay to obtain the first information carried in the overlay structure of each overlay, and then determines the group identification information of each overlay according to the first information. In addition, the terminal can parse the overlay structure of each overlay to obtain the operation function of each overlay.

It should be understood that if the first information carries identification information of other overlays that belong to the same group as the overlay, the terminal may determine other overlays that belong to the same group as the overlay.

For example, the terminal parses the entity group in the media data stream that contains one or more overlays. The terminal may obtain the OverlayRelationGroupBox from the entity group, thereby obtaining the respective group identification information of each overlay, and / or ref_overlay_id. At the same time, the terminal can also determine the operation function of the overlay according to the overlay structure, and then can determine the identification information of all overlays belonging to the same group. And all the syntax elements in the overlay's OverlayInteraction control structure.

Here, as another possible implementation manner, step 106 may be specifically implemented in the following manner (2-1):

In the method (2-1), when the terminal decodes and plays the media data, the terminal configuration or the user interface prompt may include operation options corresponding to a common interaction group for a corresponding interaction operation.

Exemplarily, when any overlay of the first group is triggered with an operation function corresponding to the OverlayRelationGroupBox semantics, the terminal determines all overlays in the first group according to the ref_overlay_id or the identification information of all overlays corresponding to the overlay. . The terminal then performs a common operation function on all overlays in the first group.

For example, the operation function corresponding to the OverlayRelationGroupBox semantics is size scaling. When the operation option of the first group is triggered, all overlays in the first group respond to the size scaling operation. If all overlays in the first group are displayed on the display interface, if the size scaling function corresponding to any of the overlays in the first group is triggered, then any of the triggered overlays responds to the size scaling operation. Other untriggered overlays in a group also respond to size scaling operations.

Example 3-2 corresponds to Example 3-1. As a possible implementation manner, step 105 can be specifically implemented in the following manner:

The terminal decodes the NALU of one or more overlays to obtain the SEI contained in each overlay stream in the one or more overlays. When the SEI payload type is a value represented by OLG, it means that the SEI carries a common display group message. The terminal continues to decode the SEI to obtain the overlay_conditional_shown_group_id, or the terminal continues to decode the SEI to obtain the ref_overlay_id corresponding to each overlay. After decoding the part, the terminal can determine all overlays belonging to the same common display group according to the overlay_conditional_shown_group_id or ref_overlay_id corresponding to each overlay. In addition, the terminal searches for and resolves to the AssociatedSphereRegionStruct in the overlay control structure of each overlay, and learns that the overlay is triggered by the user or turned off.

Specifically, for the specific implementation of steps 106 and 107 in Example 3-2, reference may be made to the description of steps 106 and 107 in Example 1-2, and details are not described herein again.

Example 4-2 corresponds to Example 4-1. As a possible implementation manner, step 105 may be specifically implemented in the following manner:

The terminal decodes the NALU of one or more overlays to obtain the SEI contained in each overlay stream in the one or more overlays. When the SEI payload type is a value represented by OLG, it means that the SEI carries a common interaction group message. The terminal continues to decode the SEI to obtain the overlay_relation_group_id, which indicates the ID number of the common interaction group of the overlay or the terminal continues to decode the SEI to obtain the ref_overlay_id. After decoding the part, the terminal may determine all overlays belonging to the same common interaction group according to the obtained overlay_relation_group_id corresponding to each overlay or ref_overlay_id corresponding to each overlay.

Specifically, for the specific implementation of steps 106 and 107 in Example 4-2, reference may be made to the description of steps 106 and 107 in Example 2-2, and details are not described herein again.

Example 5-2 corresponds to Example 5-1. As a possible implementation manner, step 105 can be specifically implemented in the following manner:

The terminal obtains one or an MPD corresponding to the overlay, and obtains an adaptation-level attribute by parsing the MPD corresponding to each overlay and obtains the overlay descriptor and the value of the attribute. Then, the identification information of the group to which each overlay belongs or the identification information of other overlays belonging to the same group can be obtained. In addition, when the terminal decapsulates each overlay, it can learn that the operation function of the overlay is to trigger the display or close the display by analyzing the AssociatedSphereRegionStruct in the overlay structure of each overlay.

In addition, if there is no AssociatedSphereRegionStruct in the overlay structure, the terminal can determine its operation function for the display or shutdown according to the group operation function of the overlay.

Specifically, for the specific implementation of steps 106 and 107 in Example 5-2, reference may be made to the description of steps 106 and 107 in Example 1-2, and details are not described herein again.

Example 6-2 corresponds to Example 6-1. As a possible implementation manner, step 105 may be specifically implemented in the following manner:

The terminal obtains one or an MPD corresponding to the overlay, and obtains an adaptation-level attribute by parsing the MPD corresponding to each overlay and obtains the overlay descriptor and the value of the attribute. Then, the identification information of the group to which each overlay belongs or the identification information of other overlays belonging to the same group can be obtained. And according to the overlay structure in each overlay, determine the interactive operation that each overlay has.

Specifically, for the specific implementation of steps 106 and 107 in Example 6-2, reference may be made to the description of steps 106 and 107 in Example 2-2, and details are not described herein again.

It should be noted that, for Example 3-2, Example 4-2, Example 5-2, and Example 6-2, the terminal may divide the overlay with the same group identification information into the same group. Or, the terminal converts it into the same group according to the identification information carried in each overlay and the identification information of other overlays that belong to the same group.

Although multiple overlays can be located in the same group, when any overlay in the group responds to the operation function corresponding to the overlay, all overlays in the same group can respond to the operation function of any overlay. Or, there is only an operation function for which any of the overlay responses is triggered. Therefore, corresponding to scenario 2, the embodiment of the present application can define conditions for performing a group operation on a group where an overlay is located.

In a possible implementation manner, one or more overlays in the embodiment of the present application further include fourth information, where the fourth information is used to indicate that in the case where the overlay responds to the first operation function, all overlays in the group to which the overlay belongs respond to An operation function (ie group operation). It can be understood that the first operation function is any one of a plurality of operation functions of the overlay. It should be noted that the first operation function is any operation function that is triggered among a plurality of operation functions corresponding to the overlay.

Wherein, the group operation refers to: in the case of performing an operation function on any overlay in a group, all overlays in the group to which the overlay belongs perform the operation function triggered by the overlay. For example, overlay1 and overlay2 are located in group 1. When overlay1 is triggered, overlay1 and overlay2 jointly respond to the operation function of overlay1 being triggered.

Correspondingly, step 105 in the embodiment of the present application may be specifically implemented in the following manner: the terminal processes the at least two overlays according to the first information and the fourth information of the at least two overlays. Specifically, if the overlay corresponds to the fourth information, step 106 further includes that the terminal may include an operation option for performing a group operation on the group to which the overlay belongs in a client configuration or a user interface prompt.

For a specific implementation of step 107 at this time, reference may be made to the foregoing description, that is, the terminal uses the group as a granularity to perform triggered operation functions on all overlays in the group, and details are not described herein again. That is, any overlay is triggered, and all overlays in the group corresponding to any overlay respond to the operation function of any overlay being triggered.

In another possible implementation manner, one or more overlays in the embodiments of the present application further include: fifth information. The fifth information is used to indicate that in the case where the overlay responds to the first operation function, all overlays in the group to which the overlay belongs respond to the first operation function (for example, group operation), or the overlay responds to the first operation function (single operation ).

Accordingly, step 105 may be specifically implemented in the following manner: the terminal processes the at least two overlays according to the first information and the fifth information of the at least two overlays.

The corresponding step 106 further specifically includes: the terminal displaying an operation option corresponding to the at least one group to perform a group operation and an operation option to perform an individual operation.

The corresponding step 107 further includes: if an individual operation is triggered, when any overlay is triggered, any overlay responds to the triggered operation function. If a group operation is triggered, when any overlay is triggered, any overlay and other overlays belonging to the same group as the overlay respond to the triggered operation function.

Understandably, all overlays in the same group have fourth information. Alternatively, all overlays in the same group have fifth information.

The individual operation refers to: in the case of performing an operation function on any overlay in a group, the any overlay responds to the triggered operation function. For example, when overlay1 is triggered, other overlays in group 1 do not respond to the operation function corresponding to overlay1, and only overlay1 responds to the operation function of overlay1.

Exemplarily, the fourth information or the fifth information may be carried in an MPD of a media data stream, an SEI of an overlay code stream, or a file format.

It should be understood that if the fourth information is carried in the overlay, the terminal may determine that the group to which the overlay belongs has only the permission with the granularity of the group operation. If the overlay carries the fifth information, the terminal may determine that the group to which the overlay belongs has permissions with group operations and individual operations as the granularity. The granularity of operation depends on the user's choice.

Example 2-3, combined with the above example 2-1, for example, the fourth information or the fifth information is located in the entity group defined in the overlay structure. The following uses the third information and the fourth information as a condition type (condition_type) as an example.

For example, in the entity file group in the OMAF standard file format, a new group box for mutual interaction based on overlays is OverlayRelationGroupBox, and then the condition_type is defined in its syntax structure. The syntax structure is shown in Table 16:

Table 16

Among them, condition_type is used to indicate a condition for the user to perform a certain type of common operation on the group.

It should be understood that condition_type may have different values, and different condition_type values indicate that the groups of the overlay have different permissions. For example, the condition_type value is 0, which means that it has group operation authority. That is, when any overlay in the group is triggered, other overlays in the group also respond to the operation function of any overlay. The condition_type value is 1, indicating that the group has group operation and individual operation permissions. If a group operation is triggered, when any overlay in the group is operated, other overlay codes in the group also respond to the operation function of any of the overlays being triggered. If an individual operation is triggered, when any overlay in the group is operated, only any overlay responds to the triggered operation function.

Exemplarily, the values of condition_type corresponding to all overlays in the same group are the same.

It should be understood that if the overlay corresponds to the fourth information or the fifth information, the terminal can obtain the fourth information or the fifth information by parsing to the entity group in the overlay structure.

Exemplarily, if condition_type = 1, the client configuration or the user interface prompt may include an operation option for performing overlay group operations or an operation option for performing individual operations. If condition_type = 1, when the operation option of the group operation is not triggered, only interactive operations are responded to the triggered overlay. When the operation option corresponding to the group operation is triggered, and any overlay in the group triggers the operation function corresponding to the OverlayRelationGroupBox semantics, according to all ref_overlay_id in the group, all overlays in the group will jointly respond to the triggered overlay. Interaction.

Example 4-3, combined with the above example 4-1, for example, the fourth information or the fifth information is located in the overlay code stream SEI. That is, the condition_type is defined in the SEI of the overlay stream corresponding to each overlay in one or more overlays, and the syntax structure is shown in Table 17:

Table 17

It should be noted that whether the fourth information or the fifth information is located in the overlay code stream SEI corresponding to each overlay or the description file or file format of the media data including one or more overlays, the specific implementation of step 105 to step 107 For the manner, refer to the description in the foregoing embodiment, and details are not described herein again.

Example 6-3, combined with the above example 6-1, for example, the fourth information or the fifth information is located in the MPD. Specifically, the fourth information and the fifth information may be located in the overlay description word of the overlay together with the group identification information.

For example, overlay_relation_group_id and condition_type are defined in the MPD corresponding to one or more overlays, and the syntax structure is shown in Table 18:

Table 18

Specifically, for the definition and value of condition_type, reference may be made to the description in Table 16 above, which is not repeated here.

Exemplarily, Table 19 shows the condition_type and syntax structure carried in the MPD file, as shown in Table 19:

In Table 19, a total of two groups that can be interacted together are described, namely, group 1 and group 2, each of which includes two overlays. Group 1 includes overlay1 and overlay2, and the condition_type = 0 of group 1 indicates that when any type of interactive operation is performed on overlay1 or overlay2 in group 1, all overlays in group 1 respond to the interaction at the same time operating. Group 2 includes overlay 3 and overlay 4. Condition_type = 1 in group 2 means that only when the user selects group 2 to perform a group operation, all overlays in the group 2 respond to a common interactive operation, otherwise only The triggered overlay responds to interactive operations.

Since an overlay may correspond to one or more groups, as another embodiment of the present application, the group identification information of the overlay in this embodiment of the present application includes one or more group identification information of the overlay. That is, the overlay structure, description file, or SEI of the overlay can be used to indicate that the overlay corresponds to multiple groups. At this time, the first information is also used to indicate the number of groups corresponding to the overlay.

Example A, when the information indicating that the overlay corresponds to multiple groups exists in the overlay structure, the overlay structure where each overlay is located is shown in Table 20:

Table 20

Among them, overlay_relation_group_number indicates the number of groups to which the overlay belongs.

overlay_relation_group_id [i] represents the ID number of the i-th group in which the overlay is located.

In the case shown in Table 20, when the terminal decodes and plays the media data, the client configuration or the user interface prompt may set operation options for common operations on the same group overlay, and different groups have different operation options. When the overlay in any one group is triggered, all overlays in any one group will jointly respond to the triggered operation function of the triggered overlay. If an overlay is in multiple groups, the overlay will respond to user operations for different groups in which the overlay is located in turn.

When the group identification information of the overlay includes one or more group identification information of the overlay, the overlay structure of each overlay may also have a syntax element of condition_type, as shown in Table 21 below:

Table 21

In the case of Table 21, if an overlay is in multiple groups, the overlay will sequentially respond to user operations for different groups in which the overlay is located according to the conditions specified by the trigger type information condition_type.

Example B. When the information indicating that multiple groups correspond to the overlay exists in the SEI of the overlay stream, the SEI syntax structure corresponding to each overlay stream is shown in Table 22:

Table 22

It should be understood that, in the case of Table 22, the operation manner of the terminal when one overlay corresponds to multiple groups may refer to the description in Table 20, and details are not described herein again.

Optionally, the SEI syntax shown in Table 22 may also have a syntax element of condition_type, as shown in Table 23 below:

Table 23

It should be understood that, in the case shown in Table 23, operations performed by the terminal on an overlay corresponding to multiple groups and syntax elements with condition_type can refer to the description in Table 21, and are not repeated here.

For example C, an information description file indicating multiple groups corresponding to the overlay is taken as an example. For example, the information of multiple groups is located in the OVLY description word, and the OVLY description word corresponding to each overlay is shown in Table 24:

Table 24

Among them, the adaptation sets having the same overlay_relation_group_id value belong to the same group, and the same overlay can belong to multiple different groups.

The number of groups to which the overlay belongs is specified by overlay_relation_group_number, and overlay_relation_group_id indicates the ID number of the group to which the overlay belongs.

In addition, when multiple groups are shown in Table 24, the OVLY descriptor has condition_type. Among them, condition_type is a condition type in which the groups corresponding to the overlay_relation_group_id interact together.

It should be understood that, in the case of Table 24, the operations of the terminal on one overlay corresponding to multiple groups and syntax elements with condition_type can refer to the descriptions in Tables 20 and 21, and are not repeated here.

Exemplarily, Table 25 shows a specific example of syntax elements with multiple group identification information and condition_type in the MPD, as shown in Table 25:

Table 25

In Table 25, three groups are described, which are group 1, group 2, and group 3, where group 1 includes overlay 1 and overlay 3, group 2 includes overlay 1, and group 3 includes overlay 2 and overlay 4.

It should be noted that when an overlay is located in multiple groups, the multiple groups correspond to different operation options. If the overlay has fourth information, the overlay is located in the second group and the third group as an example. When the overlay in the second group is triggered, the terminal performs an operation function corresponding to the second group on all overlays in the second group, and performs a third group on all overlays in the third group. Corresponding operation function. That is, all overlays in the second group respond to the operation function of the second group, and all overlays in the third group respond to the operation function corresponding to the third group.

If the overlay has the fifth information, if the group operation is triggered, when the overlay in the second group is triggered, the terminal performs an operation function corresponding to the second group on all overlays in the second group. (That is, all overlays in the second group respond to the operation functions corresponding to the second group), and perform the operation functions corresponding to the third group on all overlays in the third group (that is, the All overlays respond to the operation function corresponding to the third group). If a separate operation is triggered, when the overlay in the second group is triggered, the terminal performs an operation function corresponding to the second group on the overlay, and performs a third group on the overlay in the third group. Corresponding operation function.

In another possible example, when each overlay in one or more overlays in the embodiment of the present application corresponds to first information, the first information further includes first indication information, and an operation function corresponding to a group in which the overlay is located is The operation type indicated by the first instruction information is determined.

Specifically, the first indication information is used to indicate a type of an interaction operation (overlay_interaction_type). That is, it is used to indicate the specific operation function corresponding to the overlay.

Taking the first information carried in the SEI of the overlay code stream as an example, Table 26 shows an example of the first indication information carried in the SEI syntax.

Table 26

overlay_relation_group_info(payloadSize){overlay_relation_group_info (payloadSize) {	DescriptorDescriptor
overlay_relation_group_idoverlay_relation_group_id	Zh
overlay_interaction_typeoverlay_interaction_type	Zh
}}	Zh

The value of overlay_interaction_type indicates the type of the group that the current overlay can perform common interaction operations on. One representation is to indicate by bit, as shown in Table 27 below:

Table 27

When a bit of the overlay_interaction_type has a value, it means that the overlay can perform an interaction operation corresponding to the bit in a group operation mode.

It should be noted that the values of the overlay_interaction_type corresponding to the overlays in the same group must be the same.

It should be understood that Table 27 only exemplifies the types of partial interaction operations, and the types of common interaction operations of the overlay may not be limited to those shown in Table 27 above.

It should be noted that after the terminal resolves to overlay_interaction_type, the terminal determines the type of interaction operation that can be performed in a group operation based on the value of overlay_interaction_type. For example, when the terminal determines that the values of the overlay_interaction_type corresponding to all overlays in the same group are the bit index 6 in Table 24, the terminal determines that the operation function possessed by the group can be rotated together. If the terminal determines that one or more overlays belong to multiple groups, the terminal corresponds to each group with an operation option corresponding to a type of the interactive operation indicated by the first indication information. When an operation option corresponding to any one group is triggered, all overlays in any one group operate all overlays in any one group according to the operation function indicated by the first instruction information.

In the situation shown in Table 27, the terminal performs NALU decoding on one or more overlay code streams to obtain the SEI contained in the overlay code stream. When the SEI payload type is a value represented by OLG, it indicates that the SEI is an overlay common interaction group Group of messages. The terminal continues to decode to obtain the identification information of the overlay_relation_group_id in the SEI and other overlays that belong to the same group, and the overlay_interaction_type value, that is, the ID number of the common interaction group of the overlay and the type of group interaction . After the terminal decodes the part, it obtains the group identification information of all overlays, and determines that the overlays belong to the same group. In addition, the terminal can also determine the type of interaction that can be performed according to the overlay_interaction_type value. The client configuration or user interface prompt can set user interaction option information for overlays with the same ID number as a group, and specify the type of group operation that can be performed according to the overlay_interaction_type value. Different interaction options are given for overlay groups with different ID numbers. When the user clicks or activates the option corresponding to the ID number, the overlay of the group corresponding to the ID number can be operated simultaneously according to the operation type of the specified group.

It should be understood that the overlay_interaction_type may also be carried in the MPD. For the specific process, refer to the description at Table 27, which is not repeated here.

As another embodiment of the present application, the server may add common interaction group description information to a file format corresponding to each overlay. The difference from Example 1-1 is that in this embodiment, the group identification information of the overlay is located in the overlay control structure. That is, the third information is located in an overlay control structure included in the overlay structure.

Specifically, the server encodes one or more overlay media data to obtain one or more overlay media data streams, and then encapsulates the one or more overlay media data streams. In the encapsulated media data stream, each overlay has an overlay control structure, and each overlay has an overlay control structure, and may further include an Overlay control symbol Overlay group, which is used to indicate group identification information of the overlay. Specifically, the Overlay control symbols are shown in Table 28:

Table 28.Overlay control symbol semantics

Exemplarily, the group structure of the overlay is shown in Table 29:

Table 29

Among them, overlay_relation_group_id represents the ID number of the group to which the overlay belongs.

In the overlay group defined in Table 29, the group identification information of the overlays belonging to the same group is the same, and the group identification information of the overlays belonging to different groups is different.

Specifically, the terminal obtains the overlay control structure syntax element overlay_control_flag after decapsulating one or more overlays, thereby obtaining the Overlay group represented by the tenth bit in Table 29, and then obtains the OverlayGroup structure information to obtain the overlay group identification information . After the parsing is completed, obtain the group information of all overlays.

In summary, the embodiment of the present application adds the group identification information used to indicate the overlay to the file format, the SEI or the MPD of the overlay code stream, so that the terminal can display the same group identification information in groups. One or more overlays. In this way, for overlays belonging to the same group, the user can perform the corresponding operation function of the group's overlay at the same time, which reduces the steps when the user performs the same operation on one or more overlays, and improves the user's watching VR. The subjective experience of the video.

A possible implementation manner. For scenario 1, the second information and the third information in the embodiment of the present application may be carried in an overlay file format that encapsulates a media data stream including an overlay, and the file format includes an entity group. Box (EntityToGroupBox) and overlay structure, wherein the overlay structure has an overlay associated area control structure. The second information is located in the overlay association area control structure, and the third information is located in the EntityToGroupBox.

The syntax and semantics of EntityToGroupBox are shown in Table 30 below:

Table 30

The group_id in Table 30 indicates the unique ID number of the group, which is different from any other EntityToGroupBox structure ID number. num_entities_in_group represents the number of entities in the current group, and entity_id corresponds to the ID number of an entity in the file format.

Optionally, taking the common display group as an example, when the second information and the third information are located in a file format and the third information is located in an EntityToGroupBox, the file format in the embodiment of the present application may also have the following table 31 The syntax:

Table 31

Among them, ocsg represents a type of grouping_type, which is used to indicate that the group type is a common display group.

In this way, the terminal can resolve from EntityToGroupBox to OverlayConditionalShownGroupBox, which means that a group of overlays can be displayed together when the user triggers the display for any overlay or the group. When the user closes the group or any overlay in the group, a common close is performed. The ID number of the overlay contained in the current overlay group is represented by the entity_id in the current EntityToGroupBox.

The OverlayConditionalShownGroupBox can also contain information about the name of the overlay group. The syntax structure is shown in Table 32 below:

Table 32

overlay_group_label is a UTF-8 encoded string of unlimited length, representing the description of the overlay group. Can be null

overlay_group_label is used to give group description information, and it can be displayed on the user's display interface as group information.

For another possible implementation manner, for scenario 2, the first information in the embodiment of the present application may be carried in an overlay file format that encapsulates a media data stream including an overlay, and the file format includes an EntityToGroupBox and an overlay structure, where The overlay structure includes control symbols. The first information is in EntityToGroupBox.

Optionally, taking an interactive group as an example, when the first information is in a file format and the first information is in an EntityToGroupBox, the file format in the embodiment of the present application may also have the syntax shown in Table 32:

Table 33

Among them, ovrg also represents a type of grouping_type (group type), which is used to indicate that the group type is a common interaction group.

When the syntax structure of Table 33 is adopted, the overlay structure can adopt the above description, which is not limited here.

In this way, the terminal can resolve from EntityToGroupBox to OverlayRelationGroupBox, which means that when a group of overlays can perform interactive operations on any overlay or all overlays in the group, the operation function of the overlay interactive operation is determined based on the overlay structure of each overlay. . The ID number of the overlay contained in the current overlay group is represented by the entity_id in the current EntityToGroupBox.

The OverlayRelationGroupBox can also contain information about the name of the overlay group. The syntax structure is shown in Table 34 below:

Table 34

overlay_group_label is used to give group name information, which can be displayed on the user's display interface as group information.

It can be understood that, for Table 32 and Table 34, the name information of the overlay group can be carried in the overlay group box. In this way, after acquiring the overlay, the terminal can resolve the name information of the overlay group carried by the overlay group box, and display the group name indicated by the name information of the overlay group. In one example, when the overlay group box is defined as an OverlayRelationGroupBox, the name information of the overlay group is located in the OverlayRelationGroupBox.

In another example, when the overlay group box is defined as an OverlayConditionalShownGroupBox, the name information of the overlay group is located in the OverlayConditionalShownGroupBox.

The above mainly introduces the solution of the embodiment of the present application from the perspective of interaction between various network elements. It can be understood that, in order to implement the above functions, each network element, such as a device for processing media data, includes a hardware structure and / or a software module corresponding to each function. Those skilled in the art should easily realize that, with reference to the units and algorithm steps of each example described in the embodiments disclosed herein, this application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is performed by hardware or computer software-driven hardware depends on the specific application of the technical solution and design constraints. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

In the embodiment of the present application, functional units may be divided according to the apparatus for processing media data according to the foregoing method. For example, each functional unit may be divided corresponding to each function, or two or more functions may be integrated into one processing unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional unit. It should be noted that the division of the units in the embodiments of the present application is schematic, and is only a logical function division. There may be another division manner in actual implementation.

The following description is made by taking each functional module as an example:

FIG. 7 is a schematic block diagram of an apparatus for processing media data according to an embodiment of the present application. The apparatus for processing media data may be a terminal or a chip applied to the terminal. The apparatus 500 for processing media data shown in FIG. 7 includes an obtaining module 510 and a processing module 520.

The obtaining module 510 and the processing module 520 in the apparatus 500 for processing media data may perform various steps performed by the terminal in the methods shown in FIG. 3 and FIG. 4.

When the apparatus 500 for processing media data executes the method shown in FIG. 3, the specific functions of the obtaining module 510 and the processing module 520 are as follows:

The obtaining module 510 is configured to receive at least two overlay layers corresponding to the media data. Wherein, each overlay corresponds to first information, and the first information includes group identification information of the overlay; or, the overlay corresponds to second information and third information, and the second information is used to indicate all In the operation function corresponding to the overlay, the third information includes group identification information of the overlay or identification information of other overlays that belong to the same group as the overlay.

A processing module 520 is configured to process the at least two overlays according to the first information of the at least two overlays when the overlay corresponds to the first information. Alternatively, a processing module 520 is configured to, when the overlay corresponds to the second information and the third information, the terminal performs a pairing process on the at least two according to the second information and the third information corresponding to the at least two overlays. An overlay is processed.

When the apparatus 500 for processing media data executes the method shown in FIG. 4, the apparatus 500 for processing media data further includes a display module 530. The specific functions of the display module 530 and the processing module 520 are as follows:

The display module 530 is configured to display at least one group, and information indicating an operation function corresponding to each group in the at least one group and an overlay belonging to each group.

The processing module 520 is configured to process, when any one of the at least one group is triggered, all overlays belonging to any one group in response to an operation function corresponding to the group. Or when any overlay is triggered, it handles any overlay and other overlays that belong to the same group as any overlay in response to the operation function of any overlay being triggered.

FIG. 8 is a schematic block diagram of an apparatus 600 for processing media data according to an embodiment of the present application. The apparatus 600 for processing media data may be a server. Or a chip used in a server. The apparatus 600 for processing media data shown in FIG. 8 includes an obtaining module 610, a processing module 620, and a sending module 630.

The obtaining module 610, the processing module 620, and the sending module 630 in the apparatus 600 for processing media data may perform each step of the method shown in FIG. 3 and FIG. 4 by the server.

When the apparatus 600 for processing media data executes the method shown in FIG. 3 or FIG. 4, the specific functions of the obtaining module 610, the processing module 620, and the sending module 630 are as follows:

The obtaining module 610 is configured to obtain media data.

A processing module 620, configured to process media data to obtain at least two overlay layers corresponding to the media data;

The sending module 630 is configured to send one or more overlays to the terminal.

It should be understood that when the apparatus 600 for processing media data adopts the structure shown in FIG. 2, the input / output interface 130 is configured to acquire media data. The processor 110 is configured to process media data to obtain at least two overlay layers corresponding to the media data. The input / output interface 130 is also used to send one or more overlays to the terminal.

It should be understood that when the apparatus 500 for processing media data adopts the structure shown in FIG. 2, the input / output interface 130 is configured to receive at least two overlay layers corresponding to the media data. Wherein, each overlay corresponds to first information, and the first information includes group identification information of the overlay; or, the overlay corresponds to second information and third information, and the second information is used to indicate all In the operation function corresponding to the overlay, the third information includes group identification information of the overlay or identification information of other overlays that belong to the same group as the overlay.

The processor 110 is configured to process the at least two overlays according to the first information of the at least two overlays when the overlay corresponds to the first information. Alternatively, the processor 110 is configured to: when the overlay corresponds to the second information and the third information, the terminal performs, on the at least two, the second information and the third information corresponding to the at least two overlays. An overlay is processed.

Optionally, the display 160 is configured to display at least one group, and information used to indicate an operation function corresponding to each group in the at least one group and an overlay belonging to each group.

In another possible implementation manner, at least one group is determined by third information corresponding to each overlay in the at least two overlays, and an operation function corresponding to one group is an overlay association area included in the overlay in the group The control structure is determined.

The display 160 is configured to process all overlays belonging to any one group in response to an operation function corresponding to the group when any one of the groups is triggered. Or when any overlay is triggered, it handles any overlay and other overlays that belong to the same group as any overlay in response to the operation function of any overlay being triggered.

Those of ordinary skill in the art may realize that the units and algorithm steps of each example described in connection with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, and are not repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods may be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the unit is only a logical function division. In actual implementation, there may be another division manner. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, which may be electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, may be located in one place, or may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objective of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each of the units may exist separately physically, or two or more units may be integrated into one unit.

If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application is essentially a part that contributes to the existing technology or a part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage media include: U disks, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks or compact discs and other media that can store program codes .

The above is only a specific implementation of this application, but the scope of protection of this application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in this application. It should be covered by the protection scope of this application. Therefore, the protection scope of this application shall be subject to the protection scope of the claims.

Claims

A method for processing media data, comprising:

The terminal acquires at least two overlay layers corresponding to the media data;

The overlay corresponds to first information, and the first information includes group identification information of the overlay;

Alternatively, the overlay corresponds to second information and third information, wherein the second information is used to indicate an operation function corresponding to the overlay or identification information of other overlays that belong to the same group as the overlay;

When the overlay corresponds to the first information, the terminal processes the at least two overlays according to the first information of the at least two overlays,

or,

When the overlay corresponds to the second information and the third information, the terminal processes the at least two overlays according to the second information and the third information corresponding to the at least two overlays.
The method according to claim 1, wherein the processing the at least two overlays according to the first information of the at least two overlays comprises:

The terminal displays at least one group, and information for indicating an operation function corresponding to each group in the at least one group and an overlay belonging to each group; the at least one group is controlled by the at least one group. The first information corresponding to each overlay in the two overlays is determined; the operation function corresponding to a group is determined by the overlay structure included in the overlay in each group;

or,

The processing, by the terminal, according to the second information and the third information corresponding to the at least two overlays, includes:

The terminal displays at least one group, and information for indicating an operation function corresponding to each group in the at least one group and an overlay belonging to each group; the at least one group is controlled by the at least one group. The third information corresponding to each overlay in the two overlays is determined, and the operation function corresponding to a group is determined by the overlay associated area control structure included in the overlay in the group.
The method according to claim 2, characterized in that, when any one of the at least one group is triggered, all overlays belonging to the any one group respond to an operation corresponding to the any one group Features.
The method according to claim 2, characterized in that when any overlay is triggered, any of the overlays and other overlays belonging to the same group as any of the overlays respond to the operation function of the triggered overlay. .
The method according to any one of claims 1-4, wherein the overlay further corresponds to fourth information, and the fourth information is used to indicate that when the first operation function is performed on the overlay, all the All overlays in the group to which the overlay belongs respond to the first operation function, and the method further includes: when any one of the at least one group is triggered, all overlays belonging to the any group respond to the first operation function. Describe the operation function triggered by any group; or,

When any overlay is triggered, any overlay and other overlays belonging to the same group as any overlay respond to the operation function of any overlay being triggered.
The method according to any one of claims 1-4, wherein the overlay further corresponds to fifth information, and the fifth information is used to indicate that when the first operation function is performed on the overlay, all All overlays in the group to which the overlay belongs respond to the first operation function, or the overlay responds to the first operation function:

In the case that any overlay is triggered, any of the overlays and other overlays belonging to the same group as any of the overlays respond to the operation function of the overlay being triggered, or the overlay response is triggered. Operational functions.
The method according to any one of claims 1-6, wherein when the overlay corresponds to the first information, the overlay also corresponds to sixth information, and the sixth information is used to indicate an operation corresponding to the overlay. Features;

When the overlay is triggered, the overlay responds to the operation function indicated by the sixth information.
The method according to any one of claims 1-7, wherein the second information and the third information are carried in a file format of the overlay.
The method according to claim 8, wherein the file format comprises: an overlay structure, an overlay association area control structure and an overlay group box located in the overlay structure, and the third information is located in the overlay In the group box, the second information is located in the overlay associated area control structure.
The method according to any one of claims 1 to 7, wherein the first information is carried in a file format of the overlay.
The method according to claim 10, wherein the file format comprises: an overlay group box,

The first information is located in the overlay group box.
The method according to any one of claims 1 to 7, wherein the third information is carried in supplementary enhancement information (supplementary enhancement information) of an overlay code stream corresponding to the overlay, and the second information The information is carried in an overlay-related area control structure of the overlay.
The method according to any one of claims 1-7, wherein the first information is carried in auxiliary enhancement information SEI of an overlay code stream corresponding to the overlay.
The method according to claim 12 or 13, wherein a payload type of the SEI is used to indicate that the SEI carries group identification information of the overlay.
The method according to any one of claims 1 to 7, wherein the third information is carried in a Media Presentation Description (MPD) including a media data stream of overlay, and the second information is carried In the overlay associated area control structure of the overlay.
The method according to claim 15, wherein the third information is located in an adaptive set level (representation level) or an overlay description word of the representation level (representation level) of the MPD.
The method according to any one of claims 1-7, wherein the first information is carried in a media presentation description MPD including a media data stream of an overlay.
The method according to claim 16 or 17, wherein the first information is located in an adaptation description level or an overlay description word of the representation level of the MPD.
The method according to any one of claims 1-17, wherein when the first information includes group identification information of the overlay, the first information is carried in an entity group box EntityToGroupBox.
The method according to any one of claims 1-17, wherein when the third information includes group identification information of the overlay, the third information is carried in an EntityToGroupBox, and the second information is carried in the EntityToGroupBox The overlay's overlay association area control structure.
The method according to any one of claims 1 to 18, wherein the file format of the overlay further comprises: an overlay group box, and the overlay group box carries name information of the overlay group; The method further includes:

The terminal displays the group name indicated by the name information of the group of the overlay.
The method according to any one of claims 1 to 21, wherein the overlay corresponds to a plurality of groups, and when the overlay is triggered, the overlay responds to an operation function corresponding to the plurality of groups, respectively;

Alternatively, all overlays in the plurality of groups respectively respond to operation functions corresponding to the group.
A method for processing media data, comprising:

The server obtains the media data;

The server processes the media data to obtain at least two overlay overlays corresponding to the media data; the overlay corresponds to first information, wherein the first information includes group identification information of the overlay;

Alternatively, the overlay corresponds to second information and third information; the second information is used to indicate an operation function corresponding to the overlay, and the third information is used to indicate group identification information of the overlay or the same as the overlay. identification information of other overlays belonging to the same group;

The server sends the at least two overlay overlays.
The method according to claim 23, wherein the second information and the third information are carried in a file format of the overlay.
The method according to claim 24, wherein the file format comprises an overlay structure, an overlay association area control structure located in the overlay structure, and an overlay group box;

The third information is located in the overlay group box, and the second information is located in the overlay associated area control structure.
The method according to claim 23, wherein the first information is carried in a file format of the overlay.
The method according to claim 26, wherein the file format comprises an overlay group box, and wherein the first information is located in the overlay group box.
The method according to claim 23, wherein the third information is carried in an auxiliary enhanced information SEI of an overlay code stream corresponding to the overlay, and the second information is carried in an overlay associated area control of the overlay Structure.
The method according to claim 23, wherein the first information is carried in an SEI of an overlay code stream corresponding to the overlay.
The method according to claim 28 or 29, wherein a payload type of the SEI is used to indicate that the SEI carries group identification information of the overlay.
The method according to claim 23, wherein the third information is carried in a media presentation description MPD including a media data stream of the overlay, and the second information is carried in an overlay-related area control of the overlay Structure.
The method according to claim 23, wherein the third information is located in an overlay description word of an adaptation level or a representation level of the MPD.
The method according to claim 23, wherein the first information is carried in a media presentation description MPD including a media data stream of the overlay.
The method according to claim 33, wherein the first information is located in an overlay description word of an adaptation level or a representation level of the MPD.
The method according to claim 23, wherein when the first information includes group identification information of the overlay, the first information is carried in an entity group box EntityToGroupBox.
The method according to claim 23, wherein when the third information includes group identification information of the overlay, the third information is carried in an EntityToGroupBox, and the second information is carried in an overlay association area of the overlay. In the control structure.
The method according to any one of claims 23 to 36, wherein the overlay further corresponds to fourth information, and the fourth information is used to indicate a case of a first operation function triggered by the overlay response. All overlays in the group to which the overlay belongs respond to the first operation function.
The method according to any one of claims 23 to 36, wherein the overlay further corresponds to fifth information, and the fifth information is used to indicate that when the overlay is triggered by a first operation function, the overlay All overlays in the group to which the overlay belongs respond to the first operation function, or the overlay responds to the first operation function.
The method according to any one of claims 23 to 38, wherein, when the overlay corresponds to the first information, the overlay also corresponds to sixth information, and the sixth information is used to indicate an operation corresponding to the overlay. Features.
A terminal is characterized by comprising: an obtaining module and a processing module, wherein:

The acquiring module is configured to acquire at least two overlay layers corresponding to the media data;

The overlay corresponds to first information, and the first information includes group identification information of the overlay;

Alternatively, the overlay corresponds to second information and third information, wherein the second information is used to indicate an operation function corresponding to the overlay, and the third information includes group identification information of the overlay or is related to the overlay. identification information of other overlays belonging to the same group;

When the overlay corresponds to the first information, the processing module is configured to process the at least two overlays according to the first information of the at least two overlays,

or,

When the overlay corresponds to the second information and the third information, the processing module is configured to process the at least two overlays according to the second information and the third information corresponding to the at least two overlays.
The terminal according to claim 40, wherein the terminal further comprises a display module for displaying at least one group, and for indicating that each group in the at least one group corresponds to Information of the operation function and the overlay belonging to each group; the at least one group is determined by the first information corresponding to each overlay of the at least two overlays; the operation function corresponding to one group is determined by the each The overlay structure included in the overlay of each group is determined;

or,

The display module is configured to display at least one group, and information for indicating an operation function corresponding to each group in the at least one group and an overlay belonging to each group; the at least one group It is determined by the third information corresponding to each overlay in the at least two overlays, and an operation function corresponding to a group is determined by an overlay associated area control structure included in the overlay in the group.
The terminal according to claim 40, wherein when any one of the at least one group is triggered, all overlays belonging to the any one group respond to an operation corresponding to the any one group Features.
The terminal according to claim 40, characterized in that when any overlay is triggered, any of the overlays and other overlays belonging to the same group as any of the overlays respond to the operation function of the triggered overlay. .
The terminal according to any one of claims 40-43, wherein the overlay further corresponds to fourth information, and the fourth information is used to indicate that when the first operation function is performed on the overlay, all the All overlays in the group to which the overlay belongs respond to the first operation function,

When any one of the at least one group is triggered, all overlays belonging to the any group respond to the triggered operation function of the any group; or,

When any overlay is triggered, any overlay and other overlays belonging to the same group as any overlay respond to the operation function of any overlay being triggered.
The terminal according to any one of claims 40-43, wherein the overlay further corresponds to fifth information, and the fifth information is used to indicate that when the first operation function is performed on the overlay, all the All overlays in the group to which the overlay belongs respond to the first operation function, or the overlay responds to the first operation function:

In the case that any overlay is triggered, any of the overlays and other overlays belonging to the same group as any of the overlays respond to the operation function of the overlay being triggered, or the overlay response is triggered. Operational functions.
The terminal according to any one of claims 40-45, wherein when the overlay corresponds to the first information, the overlay also corresponds to sixth information, and the sixth information is used to indicate an operation corresponding to the overlay. Features;

When the overlay is triggered, the overlay responds to the operation function indicated by the sixth information.
The terminal according to any one of claims 40-46, wherein the second information and the third information are carried in a file format of the overlay.
The terminal according to claim 47, wherein the file format comprises: an overlay structure, an overlay association area control structure and an overlay group box located in the overlay structure, and the third information is located in the overlay In the group box, the second information is located in the overlay associated area control structure.
The terminal according to any one of claims 40-46, wherein the first information is carried in a file format of the overlay.
The terminal according to claim 49, wherein the file format comprises: an overlay group box,

The first information is located in the overlay group box.
The terminal according to any one of claims 40 to 46, wherein the third information is carried in supplementary enhancement information (SEI) of an overlay code stream corresponding to the overlay, and the second The information is carried in an overlay-related area control structure of the overlay.
The terminal according to any one of claims 40 to 46, wherein the first information is carried in auxiliary enhancement information SEI of an overlay code stream corresponding to the overlay.
The terminal according to claim 51 or 52, wherein a payload type of the SEI is used to indicate that the SEI carries group identification information of the overlay.
The terminal according to any one of claims 40 to 46, wherein the third information is carried in a media presentation description (MPD) including a media data stream of the overlay, and the second information The information is carried in an overlay-related area control structure of the overlay.
The terminal according to claim 54, wherein the third information is located in an overlay description word of an adaptation level or a representation level of the MPD.
The terminal according to any one of claims 40 to 46, wherein the first information is carried in a media presentation description MPD including a media data stream of the overlay.
The terminal according to claim 55 or 56, wherein the first information is located in an adaptation description level or an overlay description word of the representation level of the MPD.
The terminal according to any one of claims 40 to 46, wherein when the first information includes group identification information of the overlay, the first information is carried in an entity group box EntityToGroupBox.
The terminal according to any one of claims 40-46, wherein when the third information includes group identification information of the overlay, the third information is carried in an EntityToGroupBox, and the second information is carried in the EntityToGroupBox The overlay's overlay association area control structure.
The terminal according to any one of claims 40-59, wherein the overlay corresponds to a plurality of groups, and when the overlay is triggered, the overlay responds to an operation function corresponding to the plurality of groups, respectively;

Alternatively, all overlays in the plurality of groups respectively respond to operation functions corresponding to the group.
The terminal according to any one of claims 40 to 60, wherein the file format of the overlay further comprises: an overlay group box, and the overlay group box carries name information of the overlay group, The display unit is further configured to display the group name indicated by the name information of the group of the overlay.
A server is characterized in that it includes:

An acquisition module for acquiring media data;

A processing module, configured to process the media data to obtain at least two overlays corresponding to the media data; the overlay corresponds to first information, where the first information includes group identification information of the overlay;

Alternatively, the overlay corresponds to second information and third information; the second information is used to indicate an operation function corresponding to the overlay, and the third information is used to indicate group identification information of the overlay or the same as the overlay. identification information of other overlays belonging to the same group;

A sending module, configured to send the at least two overlay layers to a terminal.
The server according to claim 62, wherein the second information and the third information are carried in a file format of the overlay.
The server according to claim 63, wherein the file format includes an overlay structure, an overlay association area control structure located in the overlay structure, and an overlay group box;

The third information is located in the overlay group box, and the second information is located in the overlay associated area control structure.
The server according to claim 62, wherein the first information is carried in a file format of the overlay.
The server according to claim 65, wherein the file format comprises an overlay group box, and wherein the first information is located in the overlay group box.
The server according to claim 62, wherein the third information is carried in an auxiliary enhanced information SEI of an overlay code stream corresponding to the overlay, and the second information is carried in an overlay associated area control of the overlay Structure.
The server according to claim 62, wherein the first information is carried in an SEI of an overlay code stream corresponding to the overlay.
The server according to claim 67 or 68, wherein a payload type of the SEI is used to indicate that the SEI carries group identification information of the overlay.
The server according to claim 62, wherein the third information is carried in a media presentation description MPD including a media data stream of the overlay, and the second information is carried in an overlay-related area control of the overlay Structure.
The server according to claim 70, wherein the third information is located in an overlay description word of an adaptation level or a representation level of the MPD.
The server according to claim 62, wherein the first information is carried in a media presentation description MPD including a media data stream of the overlay.
The server according to claim 72, wherein the first information is located in an overlay description word of an adaptation level or a representation level of the MPD.
The server according to claim 62, wherein when the first information includes group identification information of the overlay, the first information is carried in an entity group box EntityToGroupBox.
The server according to claim 74, wherein when the third information includes group identification information of the overlay, the third information is carried in an EntityToGroupBox, and the second information is carried in an overlay association area of the overlay. In the control structure.
The server according to any one of claims 62 to 75, wherein the overlay further corresponds to fourth information, and the fourth information is used to indicate a case of a first operation function triggered by the overlay response. All overlays in the group to which the overlay belongs respond to the first operation function.
The server according to any one of claims 62 to 76, wherein the overlay further corresponds to fifth information, and the fifth information is used to indicate that in the case that the first triggering function is triggered by the overlay, All overlays in the group to which the overlay belongs respond to the first operation function, or the overlay responds to the first operation function.
The server according to any one of claims 62 to 77, wherein when the overlay corresponds to the first information, the overlay also corresponds to sixth information, and the sixth information is used to indicate an operation corresponding to the overlay. Features.