CN107222769B - Method, device and system for transmitting auxiliary data stream - Google Patents

Method, device and system for transmitting auxiliary data stream Download PDF

Info

Publication number
CN107222769B
CN107222769B CN201610162697.1A CN201610162697A CN107222769B CN 107222769 B CN107222769 B CN 107222769B CN 201610162697 A CN201610162697 A CN 201610162697A CN 107222769 B CN107222769 B CN 107222769B
Authority
CN
China
Prior art keywords
auxiliary data
data stream
stream
destination
support information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201610162697.1A
Other languages
Chinese (zh)
Other versions
CN107222769A (en
Inventor
石祚夫
赵旭
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Mobile Communications Group Co Ltd
Original Assignee
China Mobile Communications Group Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Mobile Communications Group Co Ltd filed Critical China Mobile Communications Group Co Ltd
Priority to CN201610162697.1A priority Critical patent/CN107222769B/en
Publication of CN107222769A publication Critical patent/CN107222769A/en
Application granted granted Critical
Publication of CN107222769B publication Critical patent/CN107222769B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/4104—Peripherals receiving signals from specially adapted client devices
    • H04N21/4126—The peripheral being portable, e.g. PDAs or mobile phones
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/4104—Peripherals receiving signals from specially adapted client devices
    • H04N21/4122—Peripherals receiving signals from specially adapted client devices additional display device, e.g. video projector
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/434—Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream
    • H04N21/4345—Extraction or processing of SI, e.g. extracting service information from an MPEG stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/436—Interfacing a local distribution network, e.g. communicating with another STB or one or more peripheral devices inside the home
    • H04N21/4363—Adapting the video or multiplex stream to a specific local network, e.g. a IEEE 1394 or Bluetooth® network
    • H04N21/43637—Adapting the video or multiplex stream to a specific local network, e.g. a IEEE 1394 or Bluetooth® network involving a wireless protocol, e.g. Bluetooth, RF or wireless LAN [IEEE 802.11]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/438—Interfacing the downstream path of the transmission network originating from a server, e.g. retrieving MPEG packets from an IP network
    • H04N21/4385—Multiplex stream processing, e.g. multiplex stream decrypting
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
    • H04N21/643—Communication protocols
    • H04N21/6437—Real-time Transport Protocol [RTP]

Abstract

The embodiment of the invention discloses a method, equipment and a system for transmitting auxiliary data stream; the method can comprise the following steps: in the Miracast capability negotiation stage, a source end sends a request message of auxiliary data flow support information; the request message is used for acquiring the support information of the destination to the auxiliary data stream; and after receiving the auxiliary data stream support information, the source end sends the parameter information of the auxiliary data stream according to the auxiliary data stream support information and the auxiliary data stream to be transmitted. The auxiliary data stream associated with the multimedia data stream can be transmitted simultaneously with the transmission of the multimedia data stream, so that the multimedia data stream and the associated auxiliary data stream can be synchronously played at the destination.

Description

Method, device and system for transmitting auxiliary data stream
Technical Field
The present invention relates to the field of wireless transmission technologies, and in particular, to a method, a device, and a system for transmitting an auxiliary data stream.
Background
Currently, video and audio are streamed directly from one wireless communication enabled device to another. Such a system is called "Miracast", which is also called Wi-Fi Display (WFD). The Miracast specification is designed for streaming any type of video or audio bitstream from a source device (source) to a destination device (Sink). As an example, the source terminal (source) may be a smart phone and the destination terminal (Sink) may be a television. Although in a typical IEEE 802.11 wireless network, client devices communicate through Access Point (AP) devices, there are protocols that support Direct device communication, such as Wi-Fi Direct. The Miracast system then uses these protocols to transmit display data from one device to another, such as from a smart phone to a television or computer, or vice versa.
The current Miracast protocol specification only specifies that a source end sends multimedia streams in three modes, namely Video-only, Audio-only and Video + Audio, and correspondingly realizes transmission of Video, Audio and Audio Video. In many cases, however, during actual multimedia playing, in addition to the above-mentioned three modes of multimedia streams, it is necessary to include corresponding multimedia auxiliary data streams, such as individual subtitle streams, individual UI streams, mixed audio streams, etc.; however, the current Miracast protocol specification does not specify the transmission of the auxiliary data stream, and therefore, the current device cannot transmit the multimedia auxiliary information stream through the Miracast protocol specification.
Disclosure of Invention
In order to solve the above technical problems, embodiments of the present invention desirably provide a method, device and system for transmitting an auxiliary data stream, which can transmit the auxiliary data stream associated with a multimedia data stream while transmitting the multimedia data stream, so that the multimedia data stream and the associated auxiliary data stream can be played synchronously at a destination.
The technical scheme of the invention is realized as follows:
in a first aspect, an embodiment of the present invention provides a method for transmitting an auxiliary data stream, where the method includes:
in the Miracast capability negotiation stage, a source end sends a request message of auxiliary data flow support information; the request message is used for acquiring the support information of the destination to the auxiliary data stream;
and after receiving the auxiliary data stream support information, the source end sends the parameter information of the auxiliary data stream according to the auxiliary data stream support information and the auxiliary data stream to be transmitted.
In the above solution, the auxiliary data stream support information includes: the format of the auxiliary data stream supported by the destination and the number of data packets of the auxiliary data stream supported by the destination.
In the above scheme, the sending, by the source end, the request message for auxiliary data stream support information specifically includes:
and the source end encapsulates the request message of the auxiliary data stream support information in a parameter acquisition request of a real-time streaming protocol (RTSP), and sends the encapsulated parameter acquisition request.
In the foregoing scheme, the sending, by the source end, the parameter information of the auxiliary data stream according to the auxiliary data stream support information and the auxiliary data stream to be transmitted specifically includes:
the source end determines the program identifier of the auxiliary data stream in the MPEG-TS stream and the channel identifier of the auxiliary data stream in the program of the MPEG-TS stream according to the packaging form of the auxiliary data stream to be transmitted;
the source end determines the priority, the data packet quantity and the file format of the auxiliary data stream according to the content of the auxiliary data stream to be transmitted;
and the source end transmits the program identification of the auxiliary data stream in the MPEG-TS stream, the channel identification of the auxiliary data stream in the program of the MPEG-TS stream, the priority of the auxiliary data stream, the data packet quantity and the file format.
In the above scheme, the sending, by the source end, the parameter information of the auxiliary data stream specifically includes:
and the source end encapsulates the parameter information of the auxiliary data stream in a parameter setting request of the RTSP and sends the encapsulated parameter setting request.
In a second aspect, an embodiment of the present invention provides a method for transmitting an auxiliary data stream, where the method includes:
in the Miracast capability negotiation stage, a destination receives a request message of auxiliary data flow support information; the request message is used for acquiring the support information of the destination to the auxiliary data stream;
the destination end sends the auxiliary data stream support information;
and the destination end receives the parameter information of the auxiliary data stream.
In the above aspect, the method further includes:
and the destination initializes the data stream receiving parameters of the destination according to the parameter information of the auxiliary data stream.
In the above solution, the auxiliary data stream support information includes: the format of the auxiliary data stream supported by the destination and the number of data packets of the auxiliary data stream supported by the destination.
In the foregoing solution, the sending, by the destination, the auxiliary data stream support information specifically includes:
and the destination end encapsulates the auxiliary data stream support information in a response message aiming at the parameter acquisition request of the RTSP and sends the encapsulated response message.
In the foregoing solution, the parameter information of the auxiliary data stream specifically includes: the program identification of the auxiliary data stream in the MPEG-TS stream, the channel identification of the auxiliary data stream in the program of the MPEG-TS stream, the priority of the auxiliary data stream, the number of data packets and the file format.
In a third aspect, an embodiment of the present invention provides a source device, where the source device includes: the device comprises a first sending module and a first receiving module; wherein the content of the first and second substances,
the first sending module is configured to send a request message for supporting information of an auxiliary data flow in a capability negotiation stage of Miracast; the request message is used for acquiring the support information of the destination to the auxiliary data stream;
the first receiving module is configured to receive auxiliary data stream support information;
the first sending module is further configured to send parameter information of the auxiliary data stream according to the auxiliary data stream support information and the auxiliary data stream to be transmitted.
In the above solution, the auxiliary data stream support information includes: the format of the auxiliary data stream supported by the destination and the number of data packets of the auxiliary data stream supported by the destination.
In the foregoing solution, the first sending module is configured to encapsulate the request message of the auxiliary data streaming support information in a parameter obtaining request of a real-time streaming protocol RTSP, and send the encapsulated parameter obtaining request.
In the above scheme, the first sending module is configured to determine, according to a packing form of the auxiliary data stream to be transmitted, a program identifier of the auxiliary data stream in an MPEG-TS stream and a channel identifier of the auxiliary data stream in a program of the MPEG-TS stream;
determining the priority, the data packet number and the file format of the auxiliary data stream according to the content of the auxiliary data stream to be transmitted;
and transmitting the program identification of the auxiliary data stream in the MPEG-TS stream, the channel identification of the auxiliary data stream in the program of the MPEG-TS stream, the priority of the auxiliary data stream, the number of data packets and the file format.
In the foregoing solution, the first sending module is configured to encapsulate the parameter information of the auxiliary data stream in a parameter setting request of the RTSP, and send the encapsulated parameter setting request.
In a fourth aspect, an embodiment of the present invention provides a destination device, where the destination device includes: the second receiving module and the second sending module; wherein the content of the first and second substances,
the second receiving module is configured to receive a request message for supporting information of an auxiliary data flow at a capability negotiation stage of Miracast; the request message is used for acquiring the support information of the destination to the auxiliary data stream;
the second sending module is configured to send the auxiliary data stream support information;
the second receiving module is further configured to receive parameter information of the auxiliary data stream.
In the above solution, the destination device further includes an initialization module, configured to initialize a data stream receiving parameter of the destination device according to the parameter information of the auxiliary data stream.
In the above solution, the auxiliary data stream support information includes: the format of the auxiliary data stream supported by the destination and the number of data packets of the auxiliary data stream supported by the destination.
In the foregoing solution, the second sending module is configured to encapsulate the auxiliary data stream support information in a response message for a parameter acquisition request of a real-time streaming protocol RTSP, and send the encapsulated response message.
In the foregoing solution, the parameter information of the auxiliary data stream specifically includes: the program identification of the auxiliary data stream in the MPEG-TS stream, the channel identification of the auxiliary data stream in the program of the MPEG-TS stream, the priority of the auxiliary data stream, the number of data packets and the file format.
In a fifth aspect, an embodiment of the present invention provides a system for transmitting an auxiliary data stream, where the system includes: a source end device and a destination end device; wherein the content of the first and second substances,
the source end device is configured to send a request message for supporting information of an auxiliary data flow at a capability negotiation stage of Miracast; the request message is used for acquiring the support information of the destination to the auxiliary data stream;
and, receiving auxiliary data stream support information;
and sending parameter information of the auxiliary data stream according to the auxiliary data stream support information and the auxiliary data stream to be transmitted;
the destination device is configured to receive, at a capability negotiation stage of Miracast, a request message of the auxiliary data flow support information;
and, transmitting the auxiliary data stream support information;
and receiving parameter information of the auxiliary data stream.
The embodiment of the invention provides a method, equipment and a system for transmitting auxiliary data streams; by negotiating the related information of the auxiliary data stream associated with the multimedia data stream to be transmitted in the WFD capability negotiation process, the auxiliary data stream associated with the multimedia data stream can be transmitted while the multimedia data stream is transmitted, and the multimedia data stream and the associated auxiliary data stream can be played synchronously at the destination.
Drawings
Fig. 1 is a schematic diagram of an application scenario provided in an embodiment of the present invention;
fig. 2 is a flowchart illustrating a method for transmitting an auxiliary data stream according to an embodiment of the present invention;
fig. 3 is a schematic flowchart of a process for sending parameter information of an auxiliary data stream from a source according to an embodiment of the present invention;
fig. 4 is a schematic diagram of a specific implementation process of a method for transmitting an auxiliary data stream according to an embodiment of the present invention;
fig. 5 is a schematic diagram of an M3 request message according to an embodiment of the present invention;
fig. 6 is a schematic diagram of an M3 response message according to an embodiment of the present invention;
fig. 7 is a schematic diagram of an M4 request message according to an embodiment of the present invention;
fig. 8 is a schematic diagram of an M4 response message according to an embodiment of the present invention;
fig. 9 is a schematic flowchart of a playing process performed by a destination according to an embodiment of the present invention;
fig. 10 is a schematic diagram of another M3 response message provided in the embodiment of the present invention;
fig. 11 is a schematic diagram of another M4 request message according to an embodiment of the present invention;
fig. 12 is a flowchart illustrating a method for transmitting an auxiliary data stream based on a source end according to an embodiment of the present invention;
fig. 13 is a flowchart illustrating a method for transmitting an auxiliary data stream based on a destination according to an embodiment of the present invention;
fig. 14 is a schematic structural diagram of a source device according to an embodiment of the present invention;
fig. 15 is a schematic structural diagram of a destination device according to an embodiment of the present invention;
fig. 16 is a schematic structural diagram of another destination device according to an embodiment of the present invention;
fig. 17 is a schematic structural diagram of a transmission system of an auxiliary data stream according to an embodiment of the present invention.
Detailed Description
The technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.
Referring to fig. 1, an application scenario provided by an embodiment of the present invention is shown, in which a source 10 and a destination 20 implement screen sharing through Miracast connection; among them, a device that performs video transmission is referred to as a source device, and a device that receives a transmitted video is referred to as a destination device. In this exemplary scenario, the source end 10 may be a smartphone, a tablet computer, a notebook computer, etc. supporting Miracast function; the destination 20 may be a television, a projection device, etc. supporting Miracast functionality. It should be understood that the application scenario is only used for explaining the technical solution provided by the embodiment of the present invention, and does not limit the embodiment of the present invention at all.
It can be understood that in the application scenario shown in fig. 1, only three multimedia streams, i.e., Video-only, Audio-only and Video + Audio, can be performed between the source 10 and the destination 20, which correspond to Video, Audio and Audio Video, respectively, and auxiliary data streams, such as auxiliary subtitle stream, auxiliary Audio stream and auxiliary Video stream, cannot be transmitted.
Example one
Referring to fig. 2, which illustrates a transmission method of an auxiliary data stream according to an embodiment of the present invention, the method may be applied to the scenario illustrated in fig. 1, and the method may include:
s201: in the Miracast capability negotiation stage, a source end sends a request message of auxiliary data flow support information;
the request message is used for acquiring the support information of the destination to the auxiliary data stream; and the auxiliary data stream support information includes: the format of the auxiliary data stream supported by the destination and the number of data packets of the auxiliary data stream supported by the destination; in particular, one data packet of the auxiliary data stream represents one transmission of the auxiliary data stream.
It should be noted that the technical solution of this embodiment may be applied to a WFD Capability Negotiation (WFD Capability Negotiation) phase in which a source end and a destination end follow the miracast1.0 specification flow. In the WFD capability negotiation stage before step S201, the source end first sends an M1 message of Real Time Streaming Protocol (RTSP) to the destination end to inquire about the RTSP method supported by the destination end, and the destination end replies to the M1 message; the destination then sends an M2 message to the source to query the RTSP methods supported by the source, which replies to the M2 message.
Next, the source end can send a request message for auxiliary data stream support information; in a specific implementation process, the source end encapsulates the request message of the auxiliary data stream support information into a PARAMETER obtaining request (RTSP get _ PARAMETER request, i.e., M3 message) of the real-time streaming protocol RTSP, and sends the encapsulated PARAMETER obtaining request.
S202: after receiving the request message of the auxiliary data stream support information, the destination terminal sends the auxiliary data stream support information;
for example, in a specific implementation process, step S202 may specifically include: the destination encapsulates the auxiliary data stream support information in a response message (i.e., a response message to the M3 message) of the parameter acquisition request for the real-time streaming protocol RTSP, and sends the encapsulated response message.
S203: and after receiving the auxiliary data stream support information, the source end sends the parameter information of the auxiliary data stream according to the auxiliary data stream support information and the auxiliary data stream to be transmitted.
Illustratively, the parameter information of the auxiliary data stream specifically includes: the program identification of the auxiliary data stream in the MPEG-TS stream, the channel identification of the auxiliary data stream in the program of the MPEG-TS stream, the priority of the auxiliary data stream, the number of data packets and the file format.
Specifically, as shown in fig. 3, the sending, by the source end, parameter information of the auxiliary data stream according to the auxiliary data stream support information and the auxiliary data stream to be transmitted specifically includes:
s2031: the source end determines the program identifier of the auxiliary data stream in the MPEG-TS stream and the channel identifier of the auxiliary data stream in the program of the MPEG-TS stream according to the packaging form of the auxiliary data stream to be transmitted;
s2032: the source end determines the priority, the data packet quantity and the file format of the auxiliary data stream according to the content of the auxiliary data stream to be transmitted;
s2033: the source end transmits the program identification of the auxiliary data stream in the MPEG-TS stream, the channel identification of the auxiliary data stream in the program of the MPEG-TS stream, the priority of the auxiliary data stream, the number of data packets and the file format.
Wherein, the program identifier of the auxiliary data stream in the MPEG-TS stream may be a PATID value; the channel identifier of the auxiliary data stream in the program of the MPEG-TS stream may be a PMTID value, and it should be noted that when different multimedia data streams and multimedia auxiliary data streams are packed into one MPEG-TS stream, the multimedia data streams and multimedia auxiliary data streams need to be packed into different programs, that is, different PATID values are provided; but the multimedia auxiliary data streams are packed into one program, i.e. with the same PATID value, different PMTID values.
In a specific implementation process, step S203 may be implemented as: the source end encapsulates the PARAMETER information of the auxiliary data stream in a PARAMETER setting request (RTSP SET _ PARAMETER request, i.e., M4 message) of the RTSP, and transmits the encapsulated PARAMETER setting request.
S204: and the destination end receives the parameter information of the auxiliary data stream and initializes the data stream receiving parameters of the destination end according to the parameter information of the auxiliary data stream.
S205: and after the destination terminal initializes the data stream of the destination terminal to receive the parameters successfully, returning an initialization success message.
In a specific implementation process, the destination may send an initialization success message through a response message to the M4 message.
It should be noted that after step S205, the M5-M16 messages between the source end and the destination end may interact according to the current Miracast standard message without any modification. And the source device can set and use the same synchronization information while packing the multimedia data stream and the multimedia auxiliary data stream.
After the destination device unpacks, if the destination device contains video media information, the auxiliary data stream of the video stream is superposed on the upper layer of the video stream; and if the audio media information is contained, mixing and playing the auxiliary data stream of the audio stream and the audio stream. The superposition of the video stream and the mixing of the audio stream are common implementation means in the art, and this is not particularly limited by the embodiment of the present invention.
The embodiment provides a transmission method of auxiliary data stream; by negotiating the related information of the auxiliary data stream associated with the multimedia data stream to be transmitted in the WFD capability negotiation process, the auxiliary data stream associated with the multimedia data stream can be transmitted while the multimedia data stream is transmitted, and the multimedia data stream and the associated auxiliary data stream can be played synchronously at the destination.
Example two
Based on the technical solution of the first embodiment, referring to fig. 4, this embodiment provides a specific implementation process of a method for transmitting an auxiliary data stream, in this embodiment, a multimedia data stream that a source device needs to send is a video stream of a foreign movie, the auxiliary data stream is an original subtitle and a chinese subtitle of the foreign movie, and it can be understood that common encoding formats of subtitles include unicode, UTF-8, and the like. The specific process can comprise the following steps:
s401: the source end may additionally add a message parameter in the M3 request message to obtain auxiliary data stream information supported by the destination end;
specifically, in addition to the parameters transmitted according to the existing Miracast standard, as shown in the schematic diagram of the M3 request message shown in fig. 5, in the content of the M3 request message, the bold italic parameter is used to indicate the message parameter additionally added by the source end in the M3 request message.
Wherein, the parameters wfd _ audio _ capture _ format, wfd _ audio _ video _ format, and wfd _ audio _ codec respectively represent the format of the subtitle stream supported by the destination, the format of the auxiliary video stream supported by the destination, and the format of the auxiliary audio stream supported by the destination;
wfd _ auxiliary _ capture _ count, wfd _ auxiliary _ video _ count, and wfd _ auxiliary _ audio _ count respectively indicate that the destination can support transmission of several auxiliary subtitle streams, the destination can support transmission of several auxiliary video streams, and the destination can support transmission of several auxiliary audio streams.
S402: the source end sends an M3 request message of additionally adding message parameters to the destination end;
s403: after receiving the M3 request message of the additional added message parameters, the destination end encapsulates the parameter values corresponding to the additional added message parameters into the M3 response message;
it should be noted that, the destination may respond to each parameter in the M3 request message according to the type and format of the auxiliary data stream that can be supported by the destination and the number of data packets of the auxiliary data stream, and generate an M3 response message; in this embodiment, the destination supports two subtitle streams, and the encoding format of the subtitle stream is unicode. Therefore, in addition to the parameters transmitted according to the existing Miracast standard, a schematic diagram of an M3 reply message as shown in fig. 6; in the content of the M3 response message, the parameters of the destination responding to the parameters in the source M3 request message are represented by bold italic parameters and corresponding numerical values.
S404: the destination end returns the M3 response message to the source end;
s405: the source end determines the parameters of the auxiliary caption stream to be transmitted according to the video stream to be transmitted and the auxiliary caption stream, and generates an M4 request message according to the parameters of the auxiliary caption stream to be transmitted;
specifically, in the implementation process, the parameters of the auxiliary data stream to be transmitted may include: wfd _ authority _ PATID and wfd _ authority _ AAA _ priority _ codec _ B; wherein wfd _ auxiliary _ PATID indicates the position of the auxiliary data stream in the MPEG-TS stream, i.e. the program identification PATID where the MPEG-TS stream is located; wfd _ auxiliary _ AAA _ priority _ codec _ B represents information such as priority and supported format corresponding to the auxiliary data stream, and the specific parameter values may include a channel identifier PMTID, priority and supported format codec of the auxiliary data stream in the program of the MPEG-TS stream; for the wfd _ audio _ AAA _ priority _ codec _ B parameter name itself, "AAA" is the type of auxiliary data stream, and takes values in three values of "capture, video, and audio"; b accumulates from 1 to indicate the data packet identity of the auxiliary data stream, and if X auxiliary data streams exist, each auxiliary data stream corresponds to a wfd _ auxiliary _ AAA _ priority _ codec _ B parameter value.
In this embodiment, the multimedia stream that the source end needs to transmit is a video stream, and the auxiliary data stream that needs to be transmitted is two auxiliary subtitle streams, so the parameters of the auxiliary subtitle stream to be transmitted may include: the program identifier wfd _ auxiary _ PATID, the priority and supported format information wfd _ auxiary _ capture _ priority _ codec _1 and wfd _ auxiary _ capture _ priority _ codec _2 corresponding to the auxiliary subtitle stream in the MPEG-TS stream; therefore, in addition to the parameters transmitted according to the existing Miracast standard, as shown in the M4 request message shown in fig. 7, in the content of the M4 request message, the parameter information of the source end transmission auxiliary data stream is represented by the bold italic parameters and the corresponding numerical values.
S406: the source end sends the M4 request message to the destination end;
s407: the destination initializes the receiving parameters of the destination according to the M4 request message;
s408: and after the destination terminal successfully initializes, sending an M4 response message for representing successful initialization to the source terminal.
Specifically, the content of the M4 reply message is as shown in fig. 8.
It is understood that by the process of S401 to S408, miracast1.0 standard messages can be used in the subsequent stages without any modification. When the source end packs the video stream and the two auxiliary subtitle streams into an MPEG-TS stream and sends the MPEG-TS stream to the destination end, the destination end can receive the packed MPEG-TS stream through Miracast1.0 standard information and analyze the MPEG-TS stream to obtain the video stream and the two auxiliary subtitle streams, and the destination end superimposes the auxiliary subtitle streams on the video stream when playing. The specific playing process performed by the destination is shown in fig. 9.
EXAMPLE III
Based on the technical solutions of the foregoing two embodiments, in this embodiment, when the multimedia data stream that the source device needs to transmit is a video stream of a movie, and the auxiliary data stream is two auxiliary audio streams, the specific process may be performed according to the technical solution shown in fig. 4, but the M3 response message and the M4 request message should be modified to the message contents shown in fig. 10 and fig. 11.
Example four
Based on the same technical concept as the foregoing embodiment, referring to fig. 12, which shows a flow of a method for transmitting an auxiliary data stream based on a source end according to an embodiment of the present invention, where the method may include:
s121: in the Miracast capability negotiation stage, a source end sends a request message of auxiliary data flow support information;
the request message is used for acquiring the support information of the destination to the auxiliary data stream;
s122: and after receiving the auxiliary data stream support information, the source end sends the parameter information of the auxiliary data stream according to the auxiliary data stream support information and the auxiliary data stream to be transmitted.
In the above solution, the auxiliary data stream support information includes: the format of the auxiliary data stream supported by the destination and the number of data packets of the auxiliary data stream supported by the destination.
In the above scheme, the sending, by the source end, the request message for auxiliary data stream support information specifically includes:
and the source end encapsulates the request message of the auxiliary data stream support information in a parameter acquisition request of a real-time streaming protocol (RTSP), and sends the encapsulated parameter acquisition request.
In the foregoing scheme, the sending, by the source end, the parameter information of the auxiliary data stream according to the auxiliary data stream support information and the auxiliary data stream to be transmitted specifically includes:
the source end determines the program identifier of the auxiliary data stream in the MPEG-TS stream and the channel identifier of the auxiliary data stream in the program of the MPEG-TS stream according to the packaging form of the auxiliary data stream to be transmitted;
the source end determines the priority, the data packet quantity and the file format of the auxiliary data stream according to the content of the auxiliary data stream to be transmitted;
and the source end transmits the program identification of the auxiliary data stream in the MPEG-TS stream, the channel identification of the auxiliary data stream in the program of the MPEG-TS stream, the priority of the auxiliary data stream, the data packet quantity and the file format.
In the above scheme, the sending, by the source end, the parameter information of the auxiliary data stream specifically includes:
and the source end encapsulates the parameter information of the auxiliary data stream in a parameter setting request of the RTSP and sends the encapsulated parameter setting request.
EXAMPLE five
Based on the same technical concept as the foregoing embodiment, referring to fig. 13, a flow of a method for transmitting an auxiliary data stream based on a destination according to an embodiment of the present invention is shown, where the method may include:
s131: in the Miracast capability negotiation stage, a destination receives a request message of auxiliary data flow support information;
the request message is used for acquiring the support information of the destination to the auxiliary data stream;
s132: the destination end sends the auxiliary data stream support information;
s133: and the destination end receives the parameter information of the auxiliary data stream.
In the above aspect, the method further includes: and the destination terminal initializes the data stream receiving parameters of the destination terminal according to the parameter information of the auxiliary data stream.
In the above solution, the auxiliary data stream support information includes: the format of the auxiliary data stream supported by the destination and the number of data packets of the auxiliary data stream supported by the destination.
In the foregoing solution, the sending, by the destination, the auxiliary data stream support information specifically includes:
and the destination end encapsulates the auxiliary data stream support information in a response message aiming at the parameter acquisition request of the RTSP and sends the encapsulated response message.
In the foregoing solution, the parameter information of the auxiliary data stream specifically includes: the program identification of the auxiliary data stream in the MPEG-TS stream, the channel identification of the auxiliary data stream in the program of the MPEG-TS stream, the priority of the auxiliary data stream, the number of data packets and the file format.
EXAMPLE six
Based on the same technical concept of the foregoing embodiment, referring to fig. 14, it illustrates a source device 140 provided by an embodiment of the present invention, which may include: a first transmitting module 1401 and a first receiving module 1402; wherein the content of the first and second substances,
a first sending module 1401, configured to send a request message of support information of an auxiliary data flow in a capability negotiation stage of Miracast; the request message is used for acquiring the support information of the destination to the auxiliary data stream;
a first receiving module 1402, configured to receive auxiliary data stream support information;
the first sending module 1401 is further configured to send parameter information of the auxiliary data stream according to the auxiliary data stream support information and the auxiliary data stream to be transmitted.
In the above solution, the auxiliary data stream support information includes: the format of the auxiliary data stream supported by the destination and the number of data packets of the auxiliary data stream supported by the destination.
In the above solution, the first sending module 1401 is configured to encapsulate the request message of the auxiliary data stream support information into the parameter obtaining request of the real-time streaming protocol RTSP, and send the encapsulated parameter obtaining request.
In the above solution, the first sending module 1401 is configured to determine, according to a packing form of an auxiliary data stream to be transmitted, a program identifier of the auxiliary data stream in an MPEG-TS stream and a channel identifier of the auxiliary data stream in a program of the MPEG-TS stream;
determining the priority, the data packet number and the file format of the auxiliary data stream according to the content of the auxiliary data stream to be transmitted;
and transmitting the program identification of the auxiliary data stream in the MPEG-TS stream, the channel identification of the auxiliary data stream in the program of the MPEG-TS stream, the priority of the auxiliary data stream, the number of data packets and the file format.
In the above solution, the first sending module 1401 is configured to encapsulate the parameter information of the auxiliary data stream in the parameter setting request of the RTSP, and send the encapsulated parameter setting request.
EXAMPLE seven
Based on the same technical concept of the foregoing embodiment, referring to fig. 15, it illustrates a destination device 150 according to an embodiment of the present invention, which may include: a second receiving module 1501, a second sending module 1502; wherein the content of the first and second substances,
a second receiving module 1501, configured to receive, in a capability negotiation stage of Miracast, a request message of support information of an auxiliary data flow; the request message is used for acquiring the support information of the destination to the auxiliary data stream;
a second sending module 1502, configured to send auxiliary data stream support information;
the second receiving module 1501 is further configured to receive parameter information of the auxiliary data stream.
In the above solution, as shown in fig. 16, the destination device 150 further includes an initializing module 1503, configured to initialize its data stream receiving parameters according to the parameter information of the auxiliary data stream.
In the above solution, the auxiliary data stream support information includes: the format of the auxiliary data stream supported by the destination and the number of data packets of the auxiliary data stream supported by the destination.
In the foregoing solution, the second sending module 1502 is configured to encapsulate the auxiliary data stream support information in a response message of a parameter acquisition request for a real-time streaming protocol RTSP, and send the encapsulated response message.
In the above scheme, the parameter information of the auxiliary data stream specifically includes: the program identification of the auxiliary data stream in the MPEG-TS stream, the channel identification of the auxiliary data stream in the program of the MPEG-TS stream, the priority of the auxiliary data stream, the number of data packets and the file format.
Example eight
Based on the same technical concept as the previous embodiment, referring to fig. 17, a transmission system 170 of an auxiliary data stream according to an embodiment of the present invention is shown, where the system includes: a source device 140 and a destination device 150; wherein the content of the first and second substances,
the source device 140 is configured to send a request message for supporting information of an auxiliary data flow at a capability negotiation stage of Miracast; the request message is used for acquiring the support information of the destination to the auxiliary data stream;
and, receiving auxiliary data stream support information;
and sending parameter information of the auxiliary data stream according to the auxiliary data stream support information and the auxiliary data stream to be transmitted;
the destination device 150 is configured to receive, in a capability negotiation stage of Miracast, a request message of the auxiliary data flow support information;
and, transmitting the auxiliary data stream support information;
and receiving parameter information of the auxiliary data stream.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention.

Claims (17)

1. A method for transmitting an auxiliary data stream, the method comprising:
in the Miracast capability negotiation stage, a source end sends a request message of auxiliary data flow support information; the request message is used for acquiring the support information of the destination to the auxiliary data stream;
after receiving the auxiliary data stream support information, the source end sends parameter information of the auxiliary data stream according to the auxiliary data stream support information and the auxiliary data stream to be transmitted;
the sending, by the source end, parameter information of the auxiliary data stream according to the auxiliary data stream support information and the auxiliary data stream to be transmitted specifically includes:
the source end determines the program identifier of the auxiliary data stream in the MPEG-TS stream and the channel identifier of the auxiliary data stream in the program of the MPEG-TS stream according to the packaging form of the auxiliary data stream to be transmitted;
the source end determines the priority, the data packet quantity and the file format of the auxiliary data stream according to the content of the auxiliary data stream to be transmitted;
and the source end transmits the program identification of the auxiliary data stream in the MPEG-TS stream, the channel identification of the auxiliary data stream in the program of the MPEG-TS stream, the priority of the auxiliary data stream, the data packet quantity and the file format.
2. The method of claim 1, wherein the auxiliary data flow support information comprises: the format of the auxiliary data stream supported by the destination and the number of data packets of the auxiliary data stream supported by the destination.
3. The method according to claim 1, wherein the sending, by the source, the request message for auxiliary data flow support information specifically includes:
and the source end encapsulates the request message of the auxiliary data stream support information in a parameter acquisition request of a real-time streaming protocol (RTSP), and sends the encapsulated parameter acquisition request.
4. The method according to claim 1, wherein the sending, by the source, the parameter information of the auxiliary data stream specifically includes:
and the source end encapsulates the parameter information of the auxiliary data stream in a parameter setting request of the RTSP and sends the encapsulated parameter setting request.
5. A method for transmitting an auxiliary data stream, the method comprising:
in the Miracast capability negotiation stage, a destination receives a request message of auxiliary data flow support information; the request message is used for acquiring the support information of the destination to the auxiliary data stream;
the destination end sends the auxiliary data stream support information;
the destination end receives parameter information of the auxiliary data stream; the parameter information of the auxiliary data stream specifically includes: the program identification of the auxiliary data stream in the MPEG-TS stream, the channel identification of the auxiliary data stream in the program of the MPEG-TS stream, the priority of the auxiliary data stream, the number of data packets and the file format.
6. The method of claim 5, further comprising:
and the destination initializes the data stream receiving parameters of the destination according to the parameter information of the auxiliary data stream.
7. The method according to claim 5 or 6, wherein the auxiliary data stream support information comprises: the format of the auxiliary data stream supported by the destination and the number of data packets of the auxiliary data stream supported by the destination.
8. The method according to claim 5 or 6, wherein the sending, by the destination, the auxiliary data stream support information specifically includes:
and the destination end encapsulates the auxiliary data stream support information in a response message aiming at the parameter acquisition request of the RTSP and sends the encapsulated response message.
9. A source device, the source device comprising: the device comprises a first sending module and a first receiving module; wherein the content of the first and second substances,
the first sending module is configured to send a request message for supporting information of an auxiliary data flow in a capability negotiation stage of Miracast; the request message is used for acquiring the support information of the destination to the auxiliary data stream;
the first receiving module is configured to receive auxiliary data stream support information;
the first sending module is further configured to send parameter information of an auxiliary data stream according to the auxiliary data stream support information and the auxiliary data stream to be transmitted;
the first sending module is configured to determine, according to a packing form of the auxiliary data stream to be transmitted, a program identifier of the auxiliary data stream in an MPEG-TS stream and a channel identifier of the auxiliary data stream in a program of the MPEG-TS stream;
determining the priority, the data packet number and the file format of the auxiliary data stream according to the content of the auxiliary data stream to be transmitted;
and transmitting the program identification of the auxiliary data stream in the MPEG-TS stream, the channel identification of the auxiliary data stream in the program of the MPEG-TS stream, the priority of the auxiliary data stream, the number of data packets and the file format.
10. The source device of claim 9, wherein the auxiliary data stream support information comprises: the format of the auxiliary data stream supported by the destination and the number of data packets of the auxiliary data stream supported by the destination.
11. The source device of claim 9, wherein the first sending module is configured to encapsulate the request message of the auxiliary data streaming support information into a parameter obtaining request of a real-time streaming protocol RTSP, and send the encapsulated parameter obtaining request.
12. The source device of claim 9, wherein the first sending module is configured to encapsulate parameter information of the auxiliary data stream into a parameter setting request of RTSP, and send the encapsulated parameter setting request.
13. A destination device, comprising: the second receiving module and the second sending module; wherein the content of the first and second substances,
the second receiving module is configured to receive a request message for supporting information of an auxiliary data flow at a capability negotiation stage of Miracast; the request message is used for acquiring the support information of the destination to the auxiliary data stream;
the second sending module is configured to send the auxiliary data stream support information;
the second receiving module is further configured to receive parameter information of the auxiliary data stream; the parameter information of the auxiliary data stream specifically includes: the program identification of the auxiliary data stream in the MPEG-TS stream, the channel identification of the auxiliary data stream in the program of the MPEG-TS stream, the priority of the auxiliary data stream, the number of data packets and the file format.
14. The destination device according to claim 13, wherein the destination device further comprises an initialization module, configured to initialize its data stream receiving parameters according to the parameter information of the auxiliary data stream.
15. The destination device according to claim 13 or 14, wherein the auxiliary data stream support information comprises: the format of the auxiliary data stream supported by the destination and the number of data packets of the auxiliary data stream supported by the destination.
16. The destination device according to claim 13 or 14, wherein the second sending module is configured to encapsulate the auxiliary data streaming support information in a response message of a parameter acquisition request for a real-time streaming protocol RTSP, and send the encapsulated response message.
17. A system for transmitting an auxiliary data stream, the system comprising: a source end device and a destination end device; wherein the content of the first and second substances,
the source end device is configured to send a request message for supporting information of an auxiliary data flow at a capability negotiation stage of Miracast; the request message is used for acquiring the support information of the destination to the auxiliary data stream;
and, receiving auxiliary data stream support information;
and sending parameter information of the auxiliary data stream according to the auxiliary data stream support information and the auxiliary data stream to be transmitted; the sending, by the source end, parameter information of the auxiliary data stream according to the auxiliary data stream support information and the auxiliary data stream to be transmitted specifically includes:
the source end determines the program identifier of the auxiliary data stream in the MPEG-TS stream and the channel identifier of the auxiliary data stream in the program of the MPEG-TS stream according to the packaging form of the auxiliary data stream to be transmitted;
the source end determines the priority, the data packet quantity and the file format of the auxiliary data stream according to the content of the auxiliary data stream to be transmitted;
the source end sends the program identification of the auxiliary data stream in the MPEG-TS stream, the channel identification of the auxiliary data stream in the program of the MPEG-TS stream, the priority of the auxiliary data stream, the data packet quantity and the file format;
the destination device is configured to receive, at a capability negotiation stage of Miracast, a request message of the auxiliary data flow support information;
and, transmitting the auxiliary data stream support information;
and receiving parameter information of the auxiliary data stream; the parameter information of the auxiliary data stream specifically includes: the program identification of the auxiliary data stream in the MPEG-TS stream, the channel identification of the auxiliary data stream in the program of the MPEG-TS stream, the priority of the auxiliary data stream, the number of data packets and the file format.
CN201610162697.1A 2016-03-21 2016-03-21 Method, device and system for transmitting auxiliary data stream Active CN107222769B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610162697.1A CN107222769B (en) 2016-03-21 2016-03-21 Method, device and system for transmitting auxiliary data stream

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610162697.1A CN107222769B (en) 2016-03-21 2016-03-21 Method, device and system for transmitting auxiliary data stream

Publications (2)

Publication Number Publication Date
CN107222769A CN107222769A (en) 2017-09-29
CN107222769B true CN107222769B (en) 2020-06-05

Family

ID=59927834

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610162697.1A Active CN107222769B (en) 2016-03-21 2016-03-21 Method, device and system for transmitting auxiliary data stream

Country Status (1)

Country Link
CN (1) CN107222769B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109672943A (en) * 2018-12-29 2019-04-23 深圳Tcl数字技术有限公司 Multichannel caption presentation method, device and playback equipment

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101977324A (en) * 2010-11-09 2011-02-16 青岛海信宽带多媒体技术有限公司 Method for realizing screen sharing
CN104219551A (en) * 2014-09-29 2014-12-17 武汉诚迈科技有限公司 Method for screen sharing and remote control of intelligent device

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101483578B (en) * 2008-01-09 2012-11-07 华为技术有限公司 Media negotiation method, system and apparatus
CN104284246B (en) * 2013-07-08 2018-11-06 华为终端（东莞）有限公司 A kind of method and terminal of transmission data
CN103648015A (en) * 2013-11-18 2014-03-19 乐视致新电子科技（天津）有限公司 Data display method and apparatus
JP6662784B2 (en) * 2014-02-28 2020-03-11 サムスン エレクトロニクス カンパニー リミテッド Method and apparatus for displaying application data in a wireless communication system
CN105791367A (en) * 2014-12-25 2016-07-20 中国移动通信集团公司 Method, system and related equipment for sharing auxiliary media information in screen sharing

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101977324A (en) * 2010-11-09 2011-02-16 青岛海信宽带多媒体技术有限公司 Method for realizing screen sharing
CN104219551A (en) * 2014-09-29 2014-12-17 武汉诚迈科技有限公司 Method for screen sharing and remote control of intelligent device

Also Published As

Publication number Publication date
CN107222769A (en) 2017-09-29

Similar Documents

Publication Publication Date Title
US9872051B2 (en) Method and apparatus for transceiving data for multimedia transmission system
US20140040496A1 (en) On-demand http stream generation
US11343549B2 (en) Reception apparatus, reception method, transmission apparatus, and transmission method
US20150020138A1 (en) Mmt asset structure for supporting random access to systems for transmitting coded media data in heterogeneous networks, and method and apparatus for generating same
US9998798B2 (en) Reception device, reception method, transmission device, and transmission method
US8935362B2 (en) Multi-media file emulation device
CA3018476A1 (en) Systems and methods for signaling of information associated with audio content
CN107787586B (en) Method and apparatus for transmitting and receiving signal in multimedia system
CN107079188A (en) Media content reception and playback controls
US20190141361A1 (en) Systems and methods for signaling of an identifier of a data channel
WO2014207899A1 (en) Communication device and communication method
US11444885B2 (en) Transmission device, transmission method, reception device, and reception method
CN107222769B (en) Method, device and system for transmitting auxiliary data stream
US9681116B2 (en) System and method for delivering 3DTV content to variety of receivers
CN111107282B (en) Video switching method, video switcher, electronic device and storage medium
CN111970541A (en) Data transmission method, device, terminal equipment and storage medium
CN113316013B (en) Video screen projection method and system
JPWO2005071960A1 (en) Streaming system
EP3197173B1 (en) Reception device, reception method, transmission device and transmission method
KR102349451B1 (en) The method for transmitting or receiving multimedia and apparatus thereof
KR20060018840A (en) Receiver apparatus for mpeg-4 ipmp extended isma media stream
De Oliveira et al. Methodology to Control IoT Smart Devices Driven by Digital TV Video Scene
CN110876115B (en) Projection method, terminal, projection equipment and storage medium
JP7007293B2 (en) Transmission device for wirelessly transmitting an MPEG-TS (transport stream) compatible data stream
JP2017537533A (en) Media content reception and switching management

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant