CN113068064B - Method, device, computer equipment and storage medium for processing transmission stream media file - Google Patents
Method, device, computer equipment and storage medium for processing transmission stream media file Download PDFInfo
- Publication number
- CN113068064B CN113068064B CN202110290400.0A CN202110290400A CN113068064B CN 113068064 B CN113068064 B CN 113068064B CN 202110290400 A CN202110290400 A CN 202110290400A CN 113068064 B CN113068064 B CN 113068064B
- Authority
- CN
- China
- Prior art keywords
- media
- file
- key frame
- offset
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 66
- 238000000034 method Methods 0.000 title claims abstract description 57
- 238000004590 computer program Methods 0.000 claims description 31
- 230000008569 process Effects 0.000 claims description 28
- 238000003672 processing method Methods 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 5
- 238000005538 encapsulation Methods 0.000 description 3
- 230000009191 jumping Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/23412—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/234309—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4 or from Quicktime to Realvideo
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/234345—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/238—Interfacing the downstream path of the transmission network, e.g. adapting the transmission rate of a video stream to network bandwidth; Processing of multiplex streams
- H04N21/2387—Stream processing in response to a playback request from an end-user, e.g. for trick-play
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/239—Interfacing the upstream path of the transmission network, e.g. prioritizing client content requests
- H04N21/2393—Interfacing the upstream path of the transmission network, e.g. prioritizing client content requests involving handling client requests
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/27—Server based end-user applications
- H04N21/278—Content descriptor database or directory service for end-user access
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Databases & Information Systems (AREA)
- Television Signal Processing For Recording (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a method, a device, a computer device and a storage medium for processing a transmission stream media file. The method comprises the following steps: acquiring a transmission stream media file; generating an intermediate media file; the intermediate media file is packaged with the transport stream media file and the custom data media track; acquiring a key frame index of the intermediate media file; and writing the key frame index into the custom data media track in the intermediate media file to obtain a target media file. By adopting the method, the processing efficiency of progress skip during online media playing can be improved.
Description
Technical Field
The present application relates to the field of multimedia technologies, and in particular, to a method, an apparatus, a computer device, a storage medium, and a computer program product for processing a transport stream media file.
Background
With the development of multimedia technology, a transport streaming technology is developed, and the transport streaming technology is often used for online playing of high-definition video. Among them, a common transport streaming technology is to transmit video data in an encapsulation format of MPEG2-TS (Transport Stream). The MPEG2-TS can enable any section of data packet in the video stream to be independently decoded and played without a complete video stream.
In an application scenario of online video playing, a user may request to skip to a video clip with a specified progress time to start playing by dragging a video progress bar.
Aiming at a jump request of video progress, in the prior art, a key frame associated with the progress time is estimated through the requested progress time and the total video duration, then the key frame is decoded according to the key frame to obtain an image frame, when the image frame is not the image frame of the requested jump progress time, the accurate image frame can be obtained only by repeatedly estimating and decoding according to the error of the actual progress time of the decoded image frame and the requested progress time, and the processing efficiency of video progress jump is lower.
Therefore, the current online video playing mode has the problem of low processing efficiency of progress skip.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a method, an apparatus, a computer device, a storage medium, and a computer program product for processing a transport stream media file, which can improve the processing efficiency of progress skip.
A method for processing a transport stream media file, comprising:
Acquiring a transmission stream media file;
generating an intermediate media file; the intermediate media file is packaged with the transport stream media file and the custom data media track;
acquiring a key frame index of the intermediate media file; the key frame index is used for recording the corresponding relation between the time offset of the key frame of the medium and the file offset; the media key frames are key frames in the intermediate media file;
writing the key frame index into the custom data media track in the intermediate media file to obtain a target media file; and the target media file is used for determining a target file offset corresponding to the target time offset requested by the user according to the key frame index in the target media file in the online playing process, and playing the target media file according to the target file offset and the corresponding media key frame.
In one embodiment, the custom data media track is at a file header of the intermediate media file, the method further comprising:
acquiring the total media duration of the intermediate media file;
writing the total media duration to the custom data media track in the intermediate media file.
In one embodiment, the generating an intermediate media file includes:
generating a blank transmission streaming media file;
writing the custom data media track into the file header of the blank transport stream media file;
writing the media data in the transmission stream media file into a file body of the blank transmission stream media file;
and taking the blank transmission stream media file written with the custom data media track and the media data in the transmission stream media file as the intermediate media file.
In one embodiment, the obtaining the key frame index of the intermediate media file includes:
acquiring time offsets of all media key frames in the intermediate media file, and acquiring file offsets of all media key frames in the intermediate media file;
counting the difference value between at least one media key frame in the intermediate media file and the respective time offset of the media key frame and the last media key frame to obtain a time offset difference value, and counting the difference value between the respective file offset of the at least one media key frame and the last media key frame in the intermediate media file to obtain a file offset difference value;
And establishing a corresponding relation between the time offset difference value of the at least one media key frame and the file offset difference value of the at least one media key frame to obtain the key frame index.
In one embodiment, further comprising:
in the process of online playing, determining the time offset and the file offset of the last media key frame according to the time offset and the file offset of the first media key frame in the key frame index;
and restoring the time offset and the file offset of the at least one media key frame according to the time offset and the file offset of the last media key frame and the time offset difference and the file offset difference of the at least one media key frame so as to determine the corresponding target file offset when a user requests to jump to the target time offset of the at least one media key frame, and requesting a media server to feed back the corresponding target media data according to the target file offset.
In one embodiment, further comprising:
determining a target media key frame corresponding to the target file offset in the key frame index;
And decoding the target media data by adopting the target media key frame, and playing the decoded target media data.
A transport stream media file processing apparatus comprising:
the file acquisition module is used for acquiring the transmission stream media file;
an intermediate file module for generating an intermediate media file; the intermediate media file is packaged with the transport stream media file and the custom data media track;
the index acquisition module is used for acquiring a key frame index of the intermediate media file; the key frame index is used for recording the corresponding relation between the time offset of the key frame of the medium and the file offset; the media key frames are key frames in the intermediate media file;
the index writing module is used for writing the key frame index into the custom data media track in the intermediate media file to obtain a target media file; and the target media file is used for determining a target file offset corresponding to the target time offset requested by the user according to the key frame index in the target media file in the online playing process, and playing the target media file according to the target file offset and the corresponding media key frame.
A computer device comprising a memory storing a computer program and a processor which when executing the computer program performs the steps of:
acquiring a transmission stream media file;
generating an intermediate media file; the intermediate media file is packaged with the transport stream media file and the custom data media track;
acquiring a key frame index of the intermediate media file; the key frame index is used for recording the corresponding relation between the time offset of the key frame of the medium and the file offset; the media key frames are key frames in the intermediate media file;
writing the key frame index into the custom data media track in the intermediate media file to obtain a target media file; and the target media file is used for determining a target file offset corresponding to the target time offset requested by the user according to the key frame index in the target media file in the online playing process, and playing the target media file according to the target file offset and the corresponding media key frame.
A computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:
Acquiring a transmission stream media file;
generating an intermediate media file; the intermediate media file is packaged with the transport stream media file and the custom data media track;
acquiring a key frame index of the intermediate media file; the key frame index is used for recording the corresponding relation between the time offset of the key frame of the medium and the file offset; the media key frames are key frames in the intermediate media file;
writing the key frame index into the custom data media track in the intermediate media file to obtain a target media file; and the target media file is used for determining a target file offset corresponding to the target time offset requested by the user according to the key frame index in the target media file in the online playing process, and playing the target media file according to the target file offset and the corresponding media key frame.
A computer program product comprising a computer program which when executed by a processor performs the steps of:
acquiring a transmission stream media file;
generating an intermediate media file; the intermediate media file is packaged with the transport stream media file and the custom data media track;
Acquiring a key frame index of the intermediate media file; the key frame index is used for recording the corresponding relation between the time offset of the key frame of the medium and the file offset; the media key frames are key frames in the intermediate media file;
writing the key frame index into the custom data media track in the intermediate media file to obtain a target media file; and the target media file is used for determining a target file offset corresponding to the target time offset requested by the user according to the key frame index in the target media file in the online playing process, and playing the target media file according to the target file offset and the corresponding media key frame.
According to the transmission stream media file processing method, the transmission stream media file processing device, the computer equipment, the storage medium and the computer program product, the characteristics that the custom data can be stored in the custom data track of the transmission stream media file are utilized, the original transmission stream media file is packaged in a turning mode and added into the custom data media track, then the time offset of the key frame of the recording media and the key frame index of the file offset are written into the custom data media track, when the obtained target media file is played online, the correct file offset can be positioned rapidly according to the time offset, multiple times of pre-estimation and decoding are not needed, and the processing efficiency of progress skip in the online media playing process is improved.
And the user-defined data media track is added to the file head, and then the total media duration is written into the user-defined data media track, so that the total media duration can be obtained without two file offsets in the progress jumping process, and the processing efficiency of the progress jumping is further improved.
Drawings
FIG. 1 is an application environment diagram of a method for processing a transport stream file in one embodiment;
FIG. 2 is a flow chart illustrating a method for processing a transport stream file according to one embodiment;
FIG. 3 is a schematic diagram of a media file generation flow in MPEG2-TS format in accordance with one embodiment;
FIG. 4 is a block diagram of a transport stream file processing apparatus in one embodiment;
fig. 5 is an internal structural diagram of a computer device in one embodiment.
Detailed Description
The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.
The method for processing the transport stream media file can be applied to an application environment shown in fig. 1. Wherein the user terminal 102 communicates with the media server 104 via a network. The user terminal 102 may be, but not limited to, various personal computers, notebook computers, smartphones, tablet computers, and portable wearable devices, and the media server 104 may be implemented as a stand-alone server or a server cluster composed of a plurality of servers.
A user may request certain media data from the media server 104 through the user terminal 102 for media playback on the user terminal 102. The media server 104 may respond to the request of the user terminal 102, and transmit the plurality of media data in the corresponding transport stream media file to the user terminal 102 according to a specific sequence, and the user terminal 102 may play after receiving the media data, so as to realize real-time online playing of the media.
For a scene where video is played online, in the prior art, when a user drags a video progress bar on the user terminal 102 to initiate a progress skip request, a skip to a video clip of a specified progress time is requested. The user terminal may determine a corresponding time offset (time offset), i.e. the distance in time between the progress time of a certain image frame in the transport stream file and the progress time of the starting image frame. The ue 102 calculates a file offset value file offset= (time offset/media duration) file size using a time offset (time offset) by the following formula;
the file size is the total data amount of the transmission stream media file, the file offset (file offset), that is, the distance between the data packet of a certain image frame and the data packet of the starting image frame in the transmission stream media file at the data position, and the total media duration (media duration), that is, the playing duration of the media file.
Then, the user terminal 102 requests the corresponding media data from the media server 104 according to the determined file offset to jump to play the video clip of the corresponding progress time.
However, the calculated file offset value file offset is an estimated value, and according to the image frame obtained by the estimated value, the image frame may not be the image frame corresponding to the requested time offset value time offset, if the image frame is not hit, the image frame needs to be estimated again according to the error, in practical application, multiple predictions (about 5 times are required on average) may be required to hit the correct file offset value file offset, so that the processing efficiency of the video progress skip in the online video playing process is lower.
In one embodiment, as shown in fig. 2, a method for processing a transport stream media file is provided, which is illustrated by applying the method to the media server 104 in fig. 1, and includes the following steps:
step 202, a transport stream media file is acquired.
The transport stream media file may be a file in a transport stream media package format. Such as an MPEG2-TS video file.
Specifically, the media server 104 may obtain the original transport stream media file, so as to perform a subsequent encapsulation process on the original transport stream media file. In practical applications, a user may create the above-mentioned transport stream media file through a recording device for creating the transport stream media file, and upload it to the media server 104.
Step 204, generating an intermediate media file; the intermediate media file encapsulates the transport stream media file and the custom data media track.
The custom data media track may be a media track supported by a transport stream media format for storing custom data. The media track may be a data section in a transport stream media file for implementing a specific function.
It should be noted that, for example, a transport stream media file such as an MPEG2-TS file may support at least four media tracks (tracks) such as video (video), audio (audio), subtitle (subtitle), and data (data). Wherein a data media track (data track) may be used to store custom private data.
Specifically, using the above-described features of the transport stream media file, the media server 104 may repackage the transport stream media file into a new transport stream media file. To distinguish between descriptions, the new transport stream media file is named an intermediate media file.
When a new transport stream media file is encapsulated, a custom data media track can be written in the new transport stream media file, thereby obtaining an intermediate media file encapsulating the original transport stream media file and the custom data media track.
Step 206, obtaining a key frame index of the intermediate media file; the key frame index is used for recording the corresponding relation between the time offset of the key frame of the medium and the file offset; the media key frames are key frames in the intermediate media file.
The time offset (time offset) may be a distance in time between a progress time of a certain image frame in the transport stream media file and a progress time of a start image frame, and may be s (seconds).
The file offset may be a distance between a data packet of a certain image frame and a data packet of a starting image frame in the transport stream media file at a data position, and the unit may be bytes.
The key frame may be data that can decode a complete image frame according to its own data without depending on the previous and subsequent image frames during decoding, and is also commonly referred to as an I frame.
Specifically, the media server 104 may obtain the time offset and the file offset of each key frame in the intermediate media file according to the progress time and the corresponding data packet corresponding to each key frame in the intermediate media file, and form an index list according to the time offset and the file offset of each key frame, as the key frame index.
Step 208, writing the key frame index into the custom data media track in the intermediate media file to obtain a target media file; and the target media file is used for determining a target file offset corresponding to the target time offset requested by the user according to the key frame index in the target media file in the online playing process, and playing the target media file according to the target file offset and the corresponding media key frame.
Specifically, the media server 104 may write the key frame index to the custom data media track already encapsulated in the intermediate media file, resulting in the target media file described above.
For example, for the original MPEG2-TS file_0, it may be encapsulated into MPEG2-TS file_1, a data track with a fixed length (e.g., 3000 bytes) is added to MPEG2-TS file_1, and then the key frame index is stored into the data track, thereby obtaining the MPEG2-TS file containing the key frame index.
When online playing is performed on a target media file, when a user initiates a progress skip request, the user terminal 102 may find a corresponding target file offset value file offset according to a target time offset value and a key frame index in the target media file specified by the skip request, request a key frame corresponding to the determined target file offset value file offset and a corresponding media data packet thereof from the media server 104, and then decode according to the key frame and the corresponding media data packet thereof, thereby obtaining a plurality of media data such as image frames and audio data, and the user terminal 102 may skip from the current progress to play the plurality of image frames and audio data.
According to the transmission stream media file processing method, the characteristic that the custom data track of the transmission stream media file can store custom data is utilized, the original transmission stream media file is encapsulated in a turning mode and added into the custom data media track, then the time offset of the key frame of the recording media and the key frame index of the file offset are written into the custom data media track, when the obtained target media file is played online, the correct file offset can be positioned rapidly according to the time offset, multiple times of pre-estimation and decoding are not needed, and the processing efficiency of progress skip in the online media playing process is improved.
In one embodiment, the custom data media track is located at a file header of the intermediate media file, and the method for processing a transport stream media file further includes:
acquiring the total media duration of the intermediate media file; writing the total media duration to the custom data media track in the intermediate media file.
The media duration (media duration) may be a playing duration of the intermediate media file.
It should be noted that, when decoding video and audio according to the key frame and its associated media data packet, it is necessary to first obtain the total media duration of the transport stream media file, and information such as the coding types of the video and audio, and then create a decoder for video and audio, and decode the media data such as image and audio by the decoder.
However, according to the transport stream media format, the total media duration of the transport stream media file is recorded in the last complete data packet in the entire transport stream media file, so if the total media duration is read in response to the progress skip request, it is necessary to shift from the current file location to the file location of the last complete data packet to read the total media duration, and then shift the file to the original file location.
In the progress skip process, file offset is generated twice, and the processing efficiency of progress skip is affected. In view of this, the present embodiment proposes a method for writing the total duration of media to the header of a file transmitting streaming media, so as to improve the processing efficiency of progress skip.
Specifically, when the transport stream media file is transcoded, the media server 104 may first generate a file in a blank transport stream media format, then write the custom data media track preferentially to the file header of the file, and then write the media data such as video and audio in the transport stream media file in the file body of the file, where the file header of the obtained intermediate media file includes the custom data media track.
Next, the media server 104 may open the intermediate media file to obtain a media total duration of the intermediate media file, and then write the media total duration to the custom data media track located at the file header of the intermediate media file by utilizing the feature that the custom data media track may store the private data custom by the user.
In the method for processing the transport stream media file, the user-defined data media track is added to the file header, and then the total media duration is written into the user-defined data media track, so that the total media duration can be obtained without two file offsets in the progress skip process, and the processing efficiency of the progress skip is further improved.
In one embodiment, step 204 may specifically include:
generating a blank transmission streaming media file; writing the custom data media track into the file header of the blank transport stream media file; writing the media data in the transmission stream media file into a file body of the blank transmission stream media file; and taking the blank transmission stream media file written with the custom data media track and the media data in the transmission stream media file as the intermediate media file.
The blank transport stream media file may be a file in a transport stream media format and does not contain media data such as video and audio.
Specifically, when the transport stream media file is transcoded, the media server 104 may first generate a blank transport stream media file and then write the custom data media track preferentially to the file header of the blank transport stream media file.
Next, the media server 104 may write media data of video, audio, etc. in the transport stream media file to the file body of the blank transport stream media file.
Therefore, the transmission stream media file written with the custom data media track and the media data in the transmission stream media file can be used as the above-mentioned intermediate media file.
In the above method for processing a transport stream media file, the custom data media track is written into the file header of a blank transport stream media file, and then the media data of the original transport stream media file is written into the file body, so that when the total media duration is written into the custom data media track, the information of the total media duration is ensured to be in the file header of the final transport stream media file, thereby improving the processing efficiency of progress skip.
In one embodiment, step 206 includes:
acquiring time offsets of all media key frames in the intermediate media file, and acquiring file offsets of all media key frames in the intermediate media file; counting the difference value between at least one media key frame in the intermediate media file and the respective time offset of the media key frame and the last media key frame to obtain a time offset difference value, and counting the difference value between the respective file offset of the at least one media key frame and the last media key frame in the intermediate media file to obtain a file offset difference value; and establishing a corresponding relation between the time offset difference value of the at least one media key frame and the file offset difference value of the at least one media key frame to obtain the key frame index.
The time offset difference may be the difference between the time offsets of two consecutive media key frames. Similarly, the file offset difference may be the difference in file offsets of two consecutive media key frames.
Specifically, when the key frame index of the intermediate media file needs to be acquired, the corresponding relationship between the time offsets and the file offsets corresponding to the plurality of related image frames and the audio data of the media key frame can be established based on the difference value between the respective time offset and the file offset of any one media key frame and the media key frame of the continuous previous frame.
In the subsequent online playing process, the time offset and the file offset of the media key frames of each subsequent frame can be sequentially restored based on the time offset and the file offset of the media key frames of the first frame according to the time offset difference and the file offset difference of the media key frames of each subsequent frame.
Table 1 below is an example of a key frame index list for one embodiment. As shown in the following table, the first column is the field size of the list information, the third column is the identification of each media key frame, and the second column is the time offset difference (time offset [ n-1] -time offset [ n-2 ]) and the file offset difference (file offset [ n-2] -file offset [ n-3 ]) of the media key frame.
TABLE 1
In the above method for processing the transport stream media file, by establishing the time offset difference value, the file offset difference value and the corresponding relation between the media key frames, the corresponding relation can be recorded by a small amount of data content, so that the index content is compressed, the index speed of the key frames is improved, and the processing efficiency of progress skip is further improved.
In one embodiment, the above method for processing a transport stream media file further includes:
In the process of online playing, determining the time offset and the file offset of the last media key frame according to the time offset and the file offset of the first media key frame in the key frame index; and restoring the time offset and the file offset of the at least one media key frame according to the time offset and the file offset of the last media key frame and the time offset difference and the file offset difference of the at least one media key frame so as to determine the corresponding target file offset when a user requests to jump to the target time offset of the at least one media key frame, and requesting a media server to feed back the corresponding target media data according to the target file offset.
Specifically, in the process of online playing, the media server 104 or the user terminal 102 may sequentially restore the time offset and the file offset of the media key frame of each subsequent frame according to the time offset difference and the file offset difference of the media key frame of each subsequent frame based on the time offset and the file offset of the media key frame of the first frame. For example, the time offset of the first media key frame is to 0 The file offset is fo 0 The second frame media key frame has a time offset difference of Deltato 0 The file offset difference is Δfo 0 Thus, the time offset of the second frame media key frame is (to) 0 +Δto 0 ) The second frame media key frame has a file offset of (fo 0 +Δfo 0 ) And so on, the time offset and the file offset of each media key frame can be restored, so that when the user requests to skip, the corresponding target file offset can be determined according to the requested target time offset, and the target media data corresponding to the target file offset can be requested to the media server 104.
Specifically, when a user requests progress skip through the user terminal 102, the media server 104 or the user terminal 102 may determine a time offset corresponding to the user requested progress time according to the request. For example, from 0 th second to 20 second of progress time from the start of the streaming media, the target time offset is 20 seconds.
The media server 104 or the user terminal 102 can find the target time offset interval in which the media server or the user terminal is located in the key frame index according to the target progress jump request, then can determine the corresponding target file offset interval according to the key frame index, can obtain corresponding target media data according to the target file offset interval, and provide the corresponding target media data to the user terminal 102 for performing progress jumps of online media playing, without performing multiple file offset prediction as in the prior art, and improves the processing efficiency of progress jumps.
In one embodiment, the method may further include:
determining a target media key frame corresponding to the target file offset in the key frame index; and decoding the target media data by adopting the target media key frame, and playing the decoded target media data.
Specifically, after determining the target file offset, the media server 104 or the user terminal 102 may determine a corresponding target media key frame according to the key frame index, decode the target media data obtained by requesting according to the target file offset by using the target media key frame, and play the decoded target media data (image frame and audio data) on the user terminal 102, so that the user terminal 102 implements progress skip of online media playing, and the whole progress skip Cheng Moxu repeatedly estimates the file offset and decodes the media data packet multiple times, thereby improving the processing efficiency of the progress skip.
In order to facilitate a thorough understanding of the above-described embodiments by those skilled in the art, a description will be given below with reference to specific examples of the drawings.
Fig. 3 is a schematic diagram of a media file generation flow in MPEG2-TS format according to one embodiment. As shown, in step 301, a TS file_0 in the original MPEG2-TS format is first acquired.
In step 302, the TS file_0 is encapsulated into a new TS file_1, and a data media track is added during encapsulation, and an empty packet of a data stream with a fixed length (for example, 3000 bytes) is preferentially added in the data media track, and at the same time, a video data packet and an audio data packet in the TS file_0 are written into the file body of the TS file_1.
Then, in step 303, the media duration and key frame index of the TS file_1 are acquired.
Finally, in step 304, the media duration and the key frame index are written into the data media track of the TS file_1, thereby obtaining the TS file including the media duration and the key frame index in the data media track of the file header.
It should be understood that, although the steps in the flowcharts of fig. 2-3 are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in fig. 2-3 may include multiple steps or stages that are not necessarily performed at the same time, but may be performed at different times, nor does the order in which the steps or stages are performed necessarily performed in sequence, but may be performed alternately or alternately with at least a portion of the steps or stages in other steps or other steps.
In one embodiment, as shown in fig. 4, there is provided a transport stream media file processing apparatus, including:
a file acquisition module 402, configured to acquire a transport stream media file;
an intermediate file module 404 for generating an intermediate media file; the intermediate media file is packaged with the transport stream media file and the custom data media track;
an index obtaining module 406, configured to obtain a key frame index of the intermediate media file; the key frame index is used for recording the corresponding relation between the time offset of the key frame of the medium and the file offset; the media key frames are key frames in the intermediate media file;
an index writing module 408, configured to write the key frame index to the custom data media track in the intermediate media file to obtain a target media file; and the target media file is used for determining a target file offset corresponding to the target time offset requested by the user according to the key frame index in the target media file in the online playing process, and playing the target media file according to the target file offset and the corresponding media key frame.
In one embodiment, the custom data media track is at a file header of the intermediate media file, the apparatus further comprising:
the total duration acquisition module is used for acquiring the total duration of the medium of the intermediate media file;
and the duration writing module is used for writing the total duration of the media into the custom data media track in the intermediate media file.
In one embodiment, the intermediate file module 404 is specifically configured to:
generating a blank transmission streaming media file; writing the custom data media track into the file header of the blank transport stream media file; writing the media data in the transmission stream media file into a file body of the blank transmission stream media file; and taking the blank transmission stream media file written with the custom data media track and the media data in the transmission stream media file as the intermediate media file.
In one embodiment, the index acquisition module 406 is specifically configured to:
acquiring time offsets of all media key frames in the intermediate media file, and acquiring file offsets of all media key frames in the intermediate media file; counting the difference value between at least one media key frame in the intermediate media file and the respective time offset of the media key frame and the last media key frame to obtain a time offset difference value, and counting the difference value between the respective file offset of the at least one media key frame and the last media key frame in the intermediate media file to obtain a file offset difference value; and establishing a corresponding relation between the time offset difference value of the at least one media key frame and the file offset difference value of the at least one media key frame to obtain the key frame index.
In one embodiment, the transport stream media file processing apparatus is further specifically configured to:
in the process of online playing, determining the time offset and the file offset of the last media key frame according to the time offset and the file offset of the first media key frame in the key frame index; and restoring the time offset and the file offset of the at least one media key frame according to the time offset and the file offset of the last media key frame and the time offset difference and the file offset difference of the at least one media key frame so as to determine the corresponding target file offset when a user requests to jump to the target time offset of the at least one media key frame, and requesting a media server to feed back the corresponding target media data according to the target file offset.
In one embodiment, the transport stream media file processing apparatus is further specifically configured to:
determining a target media key frame corresponding to the target file offset in the key frame index; and decoding the target media data by adopting the target media key frame, and playing the decoded target media data.
The specific limitation of the transport stream media file processing device may be referred to as limitation of the transport stream media file processing method hereinabove, and will not be described herein. The above-mentioned various modules in the transport stream media file processing apparatus may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 5. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is used to store data for transport streaming media files, intermediate media files, key frame indexes, custom data media tracks, and the like. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a method of processing a transport stream media file.
It will be appreciated by those skilled in the art that the structure shown in FIG. 5 is merely a block diagram of some of the structures associated with the present inventive arrangements and is not limiting of the computer device to which the present inventive arrangements may be applied, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.
In one embodiment, a computer device is provided comprising a memory and a processor, the memory having stored therein a computer program, the processor when executing the computer program performing the steps of:
acquiring a transmission stream media file;
generating an intermediate media file; the intermediate media file is packaged with the transport stream media file and the custom data media track;
acquiring a key frame index of the intermediate media file; the key frame index is used for recording the corresponding relation between the time offset of the key frame of the medium and the file offset; the media key frames are key frames in the intermediate media file;
writing the key frame index into the custom data media track in the intermediate media file to obtain a target media file; and the target media file is used for determining a target file offset corresponding to the target time offset requested by the user according to the key frame index in the target media file in the online playing process, and playing the target media file according to the target file offset and the corresponding media key frame.
In one embodiment, the custom data media track is at a file header of the intermediate media file, and the processor when executing the computer program further performs the steps of: acquiring the total media duration of the intermediate media file; writing the total media duration to the custom data media track in the intermediate media file.
In one embodiment, the processor when executing the computer program further performs the steps of: generating a blank transmission streaming media file; writing the custom data media track into the file header of the blank transport stream media file; writing the media data in the transmission stream media file into a file body of the blank transmission stream media file; and taking the blank transmission stream media file written with the custom data media track and the media data in the transmission stream media file as the intermediate media file.
In one embodiment, the processor when executing the computer program further performs the steps of: acquiring time offsets of all media key frames in the intermediate media file, and acquiring file offsets of all media key frames in the intermediate media file; counting the difference value between at least one media key frame in the intermediate media file and the respective time offset of the media key frame and the last media key frame to obtain a time offset difference value, and counting the difference value between the respective file offset of the at least one media key frame and the last media key frame in the intermediate media file to obtain a file offset difference value; and establishing a corresponding relation between the time offset difference value of the at least one media key frame and the file offset difference value of the at least one media key frame to obtain the key frame index.
In one embodiment, the processor when executing the computer program further performs the steps of: in the process of online playing, determining the time offset and the file offset of the last media key frame according to the time offset and the file offset of the first media key frame in the key frame index; and restoring the time offset and the file offset of the at least one media key frame according to the time offset and the file offset of the last media key frame and the time offset difference and the file offset difference of the at least one media key frame so as to determine the corresponding target file offset when a user requests to jump to the target time offset of the at least one media key frame, and requesting a media server to feed back the corresponding target media data according to the target file offset.
In one embodiment, the processor when executing the computer program further performs the steps of: determining a target media key frame corresponding to the target file offset in the key frame index; and decoding the target media data by adopting the target media key frame, and playing the decoded target media data.
In one embodiment, a computer readable storage medium is provided having a computer program stored thereon, which when executed by a processor, performs the steps of:
acquiring a transmission stream media file;
generating an intermediate media file; the intermediate media file is packaged with the transport stream media file and the custom data media track;
acquiring a key frame index of the intermediate media file; the key frame index is used for recording the corresponding relation between the time offset of the key frame of the medium and the file offset; the media key frames are key frames in the intermediate media file;
writing the key frame index into the custom data media track in the intermediate media file to obtain a target media file; and the target media file is used for determining a target file offset corresponding to the target time offset requested by the user according to the key frame index in the target media file in the online playing process, and playing the target media file according to the target file offset and the corresponding media key frame.
In one embodiment, the custom data media track is at the file header of the intermediate media file, and the computer program when executed by the processor further performs the steps of: acquiring the total media duration of the intermediate media file; writing the total media duration to the custom data media track in the intermediate media file.
In one embodiment, the computer program when executed by the processor further performs the steps of: generating a blank transmission streaming media file; writing the custom data media track into the file header of the blank transport stream media file; writing the media data in the transmission stream media file into a file body of the blank transmission stream media file; and taking the blank transmission stream media file written with the custom data media track and the media data in the transmission stream media file as the intermediate media file.
In one embodiment, the computer program when executed by the processor further performs the steps of: acquiring time offsets of all media key frames in the intermediate media file, and acquiring file offsets of all media key frames in the intermediate media file; counting the difference value between at least one media key frame in the intermediate media file and the respective time offset of the media key frame and the last media key frame to obtain a time offset difference value, and counting the difference value between the respective file offset of the at least one media key frame and the last media key frame in the intermediate media file to obtain a file offset difference value; and establishing a corresponding relation between the time offset difference value of the at least one media key frame and the file offset difference value of the at least one media key frame to obtain the key frame index.
In one embodiment, the computer program when executed by the processor further performs the steps of: in the process of online playing, determining the time offset and the file offset of the last media key frame according to the time offset and the file offset of the first media key frame in the key frame index; and restoring the time offset and the file offset of the at least one media key frame according to the time offset and the file offset of the last media key frame and the time offset difference and the file offset difference of the at least one media key frame so as to determine the corresponding target file offset when a user requests to jump to the target time offset of the at least one media key frame, and requesting a media server to feed back the corresponding target media data according to the target file offset.
In one embodiment, the computer program when executed by the processor further performs the steps of: determining a target media key frame corresponding to the target file offset in the key frame index; and decoding the target media data by adopting the target media key frame, and playing the decoded target media data.
In one embodiment, a computer program product is provided comprising a computer program which, when executed by a processor, performs the steps of:
acquiring a transmission stream media file;
generating an intermediate media file; the intermediate media file is packaged with the transport stream media file and the custom data media track;
acquiring a key frame index of the intermediate media file; the key frame index is used for recording the corresponding relation between the time offset of the key frame of the medium and the file offset; the media key frames are key frames in the intermediate media file;
writing the key frame index into the custom data media track in the intermediate media file to obtain a target media file; and the target media file is used for determining a target file offset corresponding to the target time offset requested by the user according to the key frame index in the target media file in the online playing process, and playing the target media file according to the target file offset and the corresponding media key frame.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, or the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples illustrate only a few embodiments of the application, which are described in detail and are not to be construed as limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of protection of the present application is to be determined by the appended claims.
Claims (10)
1. A method for processing a transport stream media file, comprising:
acquiring a transmission stream media file;
generating an intermediate media file; the intermediate media file is packaged with the transmission stream media file and a custom data media track, wherein the custom data media track is a media track which is supported by the transmission stream media file format and is used for storing custom data;
Acquiring a key frame index of the intermediate media file; the key frame index is used for recording the corresponding relation between the time offset of the key frame of the medium and the file offset; the media key frames are key frames in the intermediate media file;
writing the key frame index into the custom data media track in the intermediate media file to obtain a target media file; the target media file is used for determining a target file offset corresponding to a target time offset requested by a user according to the key frame index in the target media file in the online playing process, and playing the target media file according to the target file offset and the corresponding media key frame;
the generating an intermediate media file includes:
generating a blank transmission streaming media file;
writing the custom data media track into the file head of the blank transmission stream media file, and adding a blank packet of a section of data stream with fixed length into the custom data media track;
writing the media data in the transmission stream media file into a file body of the blank transmission stream media file;
Taking a blank transmission streaming media file written with the custom data media track and the media data in the transmission streaming media file as the intermediate media file;
the obtaining the key frame index of the intermediate media file includes:
acquiring time offsets of all media key frames in the intermediate media file, and acquiring file offsets of all media key frames in the intermediate media file;
counting the difference value between at least one media key frame in the intermediate media file and the respective time offset of the media key frame and the last media key frame to obtain a time offset difference value, and counting the difference value between the respective file offset of the at least one media key frame and the last media key frame in the intermediate media file to obtain a file offset difference value;
and establishing a corresponding relation between the time offset difference value of the at least one media key frame and the file offset difference value of the at least one media key frame to obtain the key frame index.
2. The method of claim 1, wherein the custom data media track is at a file header of the intermediate media file, the method further comprising:
Acquiring the total media duration of the intermediate media file;
writing the total media duration to the custom data media track in the intermediate media file.
3. The method as recited in claim 1, further comprising:
in the process of online playing, determining the time offset and the file offset of the last media key frame according to the time offset and the file offset of the first media key frame in the key frame index;
and restoring the time offset and the file offset of the at least one media key frame according to the time offset and the file offset of the last media key frame and the time offset difference and the file offset difference of the at least one media key frame so as to determine the corresponding target file offset when a user requests to jump to the target time offset of the at least one media key frame, and requesting a media server to feed back the corresponding target media data according to the target file offset.
4. A method according to claim 3, further comprising:
determining a target media key frame corresponding to the target file offset in the key frame index;
And decoding the target media data by adopting the target media key frame, and playing the decoded target media data.
5. A transport stream media file processing apparatus, comprising:
the file acquisition module is used for acquiring the transmission stream media file;
an intermediate file module for generating an intermediate media file; the intermediate media file is packaged with the transmission stream media file and a custom data media track, wherein the custom data media track is a media track which is supported by the transmission stream media file format and is used for storing custom data;
the index acquisition module is used for acquiring a key frame index of the intermediate media file; the key frame index is used for recording the corresponding relation between the time offset of the key frame of the medium and the file offset; the media key frames are key frames in the intermediate media file;
the index writing module is used for writing the key frame index into the custom data media track in the intermediate media file to obtain a target media file; the target media file is used for determining a target file offset corresponding to a target time offset requested by a user according to the key frame index in the target media file in the online playing process, and playing the target media file according to the target file offset and the corresponding media key frame;
The intermediate file module is specifically configured to:
generating a blank transmission streaming media file;
writing the custom data media track into the file head of the blank transmission stream media file, and adding a blank packet of a section of data stream with fixed length into the custom data media track;
writing the media data in the transmission stream media file into a file body of the blank transmission stream media file;
taking a blank transmission streaming media file written with the custom data media track and the media data in the transmission streaming media file as the intermediate media file;
the obtaining the key frame index of the intermediate media file includes:
acquiring time offsets of all media key frames in the intermediate media file, and acquiring file offsets of all media key frames in the intermediate media file;
counting the difference value between at least one media key frame in the intermediate media file and the respective time offset of the media key frame and the last media key frame to obtain a time offset difference value, and counting the difference value between the respective file offset of the at least one media key frame and the last media key frame in the intermediate media file to obtain a file offset difference value;
And establishing a corresponding relation between the time offset difference value of the at least one media key frame and the file offset difference value of the at least one media key frame to obtain the key frame index.
6. The apparatus of claim 5, wherein the apparatus further comprises:
the total duration acquisition module is used for acquiring the total duration of the medium of the intermediate media file;
and the duration writing module is used for writing the total duration of the media into the custom data media track in the intermediate media file.
7. The apparatus according to claim 5, wherein the transport stream media file processing apparatus is further specifically configured to: in the process of online playing, determining the time offset and the file offset of the last media key frame according to the time offset and the file offset of the first media key frame in the key frame index; and restoring the time offset and the file offset of the at least one media key frame according to the time offset and the file offset of the last media key frame and the time offset difference and the file offset difference of the at least one media key frame so as to determine the corresponding target file offset when a user requests to jump to the target time offset of the at least one media key frame, and requesting a media server to feed back the corresponding target media data according to the target file offset.
8. The apparatus of claim 7, wherein the transport stream media file processing apparatus is further specifically configured to:
determining a target media key frame corresponding to the target file offset in the key frame index; and decoding the target media data by adopting the target media key frame, and playing the decoded target media data.
9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the transport stream media file processing method of any of claims 1 to 4 when the computer program is executed.
10. A computer readable storage medium having stored thereon a computer program, characterized in that the computer program when executed by a processor realizes the steps of the transport stream media file processing method of any of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110290400.0A CN113068064B (en) | 2021-03-18 | 2021-03-18 | Method, device, computer equipment and storage medium for processing transmission stream media file |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110290400.0A CN113068064B (en) | 2021-03-18 | 2021-03-18 | Method, device, computer equipment and storage medium for processing transmission stream media file |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113068064A CN113068064A (en) | 2021-07-02 |
| CN113068064B true CN113068064B (en) | 2023-09-05 |
Family
ID=76561585
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110290400.0A Active CN113068064B (en) | 2021-03-18 | 2021-03-18 | Method, device, computer equipment and storage medium for processing transmission stream media file |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113068064B (en) |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101287107B (en) * | 2008-05-29 | 2010-10-13 | 腾讯科技（深圳）有限公司 | Demand method, system and device of media file |
| CN102510519A (en) * | 2011-10-11 | 2012-06-20 | 成都市华为赛门铁克科技有限公司 | Streaming media data processing method, playing method and device |
| CN105205153B (en) * | 2015-09-24 | 2018-10-19 | 小米科技有限责任公司 | File memory method and device |
| CN105979404A (en) * | 2015-12-01 | 2016-09-28 | 乐视致新电子科技（天津）有限公司 | Streaming media processing method and device |
-
2021
- 2021-03-18 CN CN202110290400.0A patent/CN113068064B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN113068064A (en) | 2021-07-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2017092336A1 (en) | Streaming media processing method and apparatus | |
| CN110933482B (en) | Video loading method and device, computer readable storage medium and computer equipment | |
| US20210136464A1 (en) | Providing Advanced Playback And Control Functionality To Video Client | |
| JP4264833B2 (en) | Recording apparatus and method, program, and recording medium | |
| CN108076377B (en) | Video storage and playing method and device, electronic equipment and storage medium | |
| JP6314252B2 (en) | Network video playback method and apparatus | |
| CN110418186A (en) | Audio and video playing method, apparatus, computer equipment and storage medium | |
| US11250863B2 (en) | Frame coding for spatial audio data | |
| WO2005059694A2 (en) | Method and apparatus for buffering streaming media | |
| JP2002529884A (en) | Signal processing on information files to obtain characteristic point information sequences | |
| CN111356023B (en) | Playing mode determining method and device | |
| CN113068064B (en) | Method, device, computer equipment and storage medium for processing transmission stream media file | |
| US20170264685A1 (en) | Distribution method, distribution request method, and information processing device | |
| JP2006074391A (en) | File recording method and recorder | |
| US20090268811A1 (en) | Dynamic Image Reproducing Method And Device | |
| JP2010123252A (en) | Apparatus and record carrier for, and method of recording sequence of video data signal | |
| CN113438503B (en) | Video file restoring method, device, computer equipment and storage medium | |
| CN110636368B (en) | Media playing method, system, device and storage medium | |
| CN112866745B (en) | Streaming video data processing method, device, computer equipment and storage medium | |
| CN112468847B (en) | Trick play method and device of multimedia stream and multimedia player | |
| CN111356003B (en) | Data writing method, system and terminal equipment | |
| US10832505B2 (en) | Processing video and sensor data associated with a vehicle | |
| CN111813994B (en) | Data processing and file playback method and device based on interactive whiteboard | |
| US11212573B2 (en) | Systems, methods, and devices for managing segmented media content | |
| US20230345073A1 (en) | Systems and methods for providing auxiliary manifests for media items |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |