CN116126487A - Multi-instance codec scheduling method and device, storage medium and terminal - Google Patents
Multi-instance codec scheduling method and device, storage medium and terminal Download PDFInfo
- Publication number
- CN116126487A CN116126487A CN202211483010.6A CN202211483010A CN116126487A CN 116126487 A CN116126487 A CN 116126487A CN 202211483010 A CN202211483010 A CN 202211483010A CN 116126487 A CN116126487 A CN 116126487A
- Authority
- CN
- China
- Prior art keywords
- real
- time
- codec
- decoder
- coder
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
- G11B20/00—Signal processing not specific to the method of recording or reproducing; Circuits therefor
- G11B20/10—Digital recording or reproducing
- G11B20/10009—Improvement or modification of read or write signals
- G11B20/10018—Improvement or modification of read or write signals analog processing for digital recording or reproduction
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
A method and a device for scheduling a coder-decoder under multiple instances, a storage medium and a terminal are provided, wherein the method for scheduling the coder-decoder under multiple instances comprises the following steps: when a plurality of examples are concurrent, acquiring real-time performance requirement parameters of the coder and the decoder corresponding to the plurality of examples respectively, wherein the real-time performance requirement parameters are used for determining whether the coder and the decoder are real-time coder and decoder or not; and scheduling the work of each codec according to the current performance meeting condition of the actual real-time codec and the real-time performance requirement parameter of each codec, wherein the scheduling priority of the real-time codec is higher than that of the non-real-time codec. The scheme can schedule the work of the real-time coder and decoder preferentially so as to schedule the work of the non-real-time coder and decoder under the condition of meeting the performance requirement of the real-time coder and decoder, thereby fully utilizing the system performance and improving the user experience.
Description
Technical Field
The embodiment of the invention relates to the technical field of encoding and decoding, in particular to a method and a device for scheduling an encoding and decoding device under multiple instances, a storage medium and a terminal.
Background
The performance of codecs (Codec) is currently mainly focused on devices related to codecs such as mobile phones in the single instance, where a single instance is to watch only a certain video or use only a video recording function at the same time. However, with the upgrade of hardware, the complexity of application scenarios, the high demands of users and the appearance of various evaluation software, the demands of simultaneous operation of multiple codecs are becoming urgent.
When a plurality of codecs work simultaneously, most of the current codec hardware meets the requirements of codec under multiple instances in a time-sharing multiplexing mode, however, the system performance is poor in this mode, and the user experience is affected.
Disclosure of Invention
The technical problem solved by the embodiment of the invention is that when a plurality of codecs work simultaneously, the system performance is poor, and the user experience is affected.
In order to solve the above technical problems, an embodiment of the present invention provides a method for scheduling a codec in a multi-instance, including: when a plurality of examples are concurrent, acquiring real-time performance requirement parameters of the coder and the decoder corresponding to the plurality of examples respectively, wherein the real-time performance requirement parameters are used for determining whether the coder and the decoder are real-time coder and decoder or not; and scheduling the work of each codec according to the current performance meeting condition of the actual real-time codec and the real-time performance requirement parameter of each codec, wherein the scheduling priority of the real-time codec is higher than that of the non-real-time codec.
Optionally, the scheduling the operation of each codec according to the performance meeting condition of the current actual real-time codec and the real-time performance requirement parameter of each codec includes: determining that the codecs corresponding to the multiple instances respectively comprise a real-time codec and a non-real-time codec according to the real-time performance requirement parameter of each codec; before the first frame data encoding and decoding or after the last frame data encoding and decoding processing is finished, judging whether the current actual performance meets the performance requirement of the real-time encoder-decoder; and if the current actual performance meets the performance requirement of the real-time codec, scheduling the work of the non-real-time codec according to the residual performance condition.
Optionally, the number of the non-real-time codecs is multiple, and if the current actual performance meets the performance requirement of the real-time codec, scheduling the work of the non-real-time codec according to the remaining performance condition, including: and if the current actual performance meets the performance requirement of the real-time codec, scheduling a plurality of non-real-time codecs to work alternately according to the residual performance condition.
Optionally, if the current actual performance meets the performance requirement of the real-time codec, scheduling the multiple non-real-time codecs to work alternately according to the remaining performance conditions, including: the current actual performance meets the performance requirement of the real-time coder and decoder, and the identification of the current real-time coder and decoder is deleted from the real-time coder and decoder sequence, wherein the real-time coder and decoder sequence is used for indicating the working real-time coder and decoder; and scheduling the non-real-time coder and decoder in waiting to work if the number of the non-real-time coder and decoder which wait to meet the condition that the real-time coder and decoder sequence is empty and the number of the non-real-time coder and decoder which work meets the specified number.
Optionally, the method for scheduling the codec in the multiple instances further includes: and if the currently working non-real-time coder/decoder finishes coding/decoding one frame of data, notifying the waiting non-real-time coder/decoder to work.
Optionally, the method for scheduling the codec in the multiple instances further includes: if the current actual performance does not meet the performance requirement of the real-time codec, suspending the operation of the non-real-time codec so that the non-real-time codec is waiting.
Optionally, the following manner is adopted to determine whether the current actual performance meets the performance requirement of the real-time codec: judging whether the actual time required by encoding and decoding a frame at present is greater than theoretical time or not; if the actual time is greater than the theoretical time, judging that the actual performance does not meet the performance requirement of the real-time codec; and if the actual time is not greater than the theoretical time, judging that the actual performance meets the performance requirement of the real-time coder-decoder.
Optionally, the method for scheduling the codec in the multiple instances further includes: when a plurality of examples are concurrent, determining the working frequency of a coder-decoder corresponding to each example according to the frame size and the frame rate required by each example; determining a final working frequency according to the working frequency of the corresponding codec of each instance; and controlling the operation of the codec by adopting the final operation frequency, wherein the final operation frequency meets the frequency requirements of all the operated codecs.
Optionally, the obtaining real-time performance requirement parameters of the codec corresponding to the multiple instances includes: and acquiring real-time performance requirement parameters of the codecs corresponding to the multiple instances respectively through a call interface preconfigured by the hardware abstraction layer.
In order to solve the above problem, an embodiment of the present invention further provides a scheduling apparatus for a codec in a multi-instance, including: the system comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring real-time performance requirement parameters of a coder and a decoder corresponding to a plurality of examples respectively when the plurality of examples are concurrent, and the real-time performance requirement parameters are used for determining whether the coder and the decoder are real-time coder and decoder or not; and the scheduling unit is used for scheduling the work of each coder and decoder according to the current performance meeting condition of the actual real-time coder and decoder and the real-time performance demand parameter of each coder and decoder, wherein the scheduling priority of the real-time coder and decoder is higher than that of the non-real-time coder and decoder.
To solve the above-mentioned problems, an embodiment of the present invention further provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the codec scheduling method in any of the multiple instances described above.
To solve the above-mentioned problems, an embodiment of the present invention further provides a terminal, including a memory and a processor, where the memory stores a computer program capable of running on the processor, and when the processor runs the computer program, the processor executes the steps of the method for scheduling a codec under any of the above-mentioned multiple instances.
Compared with the prior art, the technical scheme of the embodiment of the invention has the following beneficial effects:
in the embodiment of the invention, when a plurality of examples are concurrent, the real-time performance requirement parameters of the codec corresponding to the plurality of examples are acquired, and the real-time performance requirement parameters are used for determining whether the codec is a real-time codec. And scheduling the work of each coder-decoder according to the current performance meeting condition of the actual real-time coder-decoder and the real-time performance requirement parameter of each coder-decoder. Because the scheduling priority of the real-time codec is higher than that of the non-real-time codec, the scheduling of the multi-codec is performed according to the current actual performance meeting condition of the real-time codec and the real-time performance requirement parameter of each codec, the work of the real-time codec can be scheduled preferentially, so that the work of the non-real-time codec can be scheduled under the condition that the performance requirement of the real-time codec is met, the system performance can be fully utilized, and the user experience is improved.
Drawings
FIG. 1 is a flow chart of a method of scheduling a codec in multiple instances in an embodiment of the present invention;
FIG. 2 is a flow chart of a real-time performance demand parameter acquisition;
FIG. 3 is a flow chart of a method of scheduling a codec in a multiple instance scenario in a typical application scenario;
fig. 4 is a schematic diagram of a scheduling apparatus of a codec in a multi-instance embodiment of the present invention.
Detailed Description
As described above, most of the current Codec hardware meets the demands of the Codec under multiple instances by time-division multiplexing, and in this way, since the system is identical to all codecs, the performance of the system running multiple codecs is not guaranteed, the resource allocation rationality is poor, and the user experience is affected.
In the embodiment of the present invention, when multiple instances are concurrent, the real-time performance requirement parameters of the codecs corresponding to the multiple instances are obtained, where the real-time performance requirement parameters are used to determine whether the codec is a real-time codec. And scheduling the work of each coder-decoder according to the current performance meeting condition of the actual real-time coder-decoder and the real-time performance requirement parameter of each coder-decoder. Because the scheduling priority of the real-time codec is higher than that of the non-real-time codec, the scheduling of the multi-codec is performed according to the current actual performance meeting condition of the real-time codec and the real-time performance requirement parameter of each codec, the work of the real-time codec can be scheduled preferentially, so that the work of the non-real-time codec can be scheduled under the condition that the performance requirement of the real-time codec is met, the system performance can be fully utilized, and the user experience is improved.
In order to make the above objects, features and advantages of the embodiments of the present invention more comprehensible, the following detailed description of the embodiments of the present invention refers to the accompanying drawings.
The embodiment of the invention provides a scheduling method of a coder-decoder under multiple instances, which can be executed by a terminal, a chip or a chip module with a scheduling function of the coder-decoder under multiple instances in the terminal, a chip or a chip module with a data processing function in the terminal, and a baseband chip in the terminal. The terminal can be various suitable devices such as a mobile phone, a digital television, a smart watch, a vehicle recorder, a tablet personal computer and the like.
Referring to fig. 1, a flowchart of a method for scheduling a codec under multiple instances in an embodiment of the present invention is provided, which may specifically include the following steps:
and step 12, scheduling the work of each codec according to the performance meeting condition of the current actual real-time codec and the real-time performance requirement parameter of each codec, wherein the scheduling priority of the real-time codec is higher than that of the non-real-time codec.
In implementations, multiple operations related to using a codec at the same time will be concurrent with multiple instances. I.e. using a number of codec related operations at a certain moment or within a certain period of time. The codec-related operations may or may not be triggered simultaneously.
For example, recording a display screen to play video at the same time as video is played will be concurrent with two instances. Specifically, in response to a video play operation, an instance is created. In the video playing process, if a user triggers the operation of playing the video by the recording display screen, an instance is created in response to the operation of playing the video by the recording display screen. At this time, two examples are concurrent. And when the video is played, recording the video played by the display screen. The codec is required to decode the instance of playing video and the codec is also required to encode the instance of recording video on the display.
For another example, playing a video while operating a video call would be concurrent with two instances.
For another example, in the process of using video call, two or more different playing platforms are used to play video, and three or more instances will be concurrent.
For another example, for some devices, the performance is not enough to support a new coding format, and the conversion of the unsupported coding format into the supported coding format needs to be performed for playing, and at this time, the conversion of the coding format needs to be performed in the background. In this way, multiple instances will be concurrent if there are still other instances that require the codec to perform the codec. Wherein, the encoding and decoding refer to encoding or decoding.
It will be appreciated that there are other situations where multiple instances may be concurrent, and this is not an example.
The real-time performance requirement parameter obtained in step 11 is used to indicate the requirements of the Codec (Codec) for system performance. The real-time performance requirement parameter is used to indicate whether the codec is a real-time codec. A real-time codec refers to a codec that the system must meet the operating rate requirements. Non-real-time codecs require the system to try to meet the requirements and do not make strict demands, i.e. the system does not necessarily guarantee to meet its requirements.
The real-time performance requirement parameter may be a codec Priority (Priority), an operation rate (operation rate) of a codec, and a decoding rate of the codec. It should be noted that, different manufacturers or different types of codecs may have different names of real-time performance requirement parameters or parameters of specific configurations.
For example, if the codec priority of the real-time performance requirement parameter configuration of the codec is higher than the set priority, or a specific operation rate is configured, or a specific decoding rate is configured, or the real-time performance requirement parameter is configured to specify the first flag, the real-time codec is determined.
For another example, the codec priority is not configured by the real-time performance requirement parameter of the codec, or the configured codec priority is lower than the set priority, or the operation rate is a default value, or the decoding rate is a default value, or the real-time performance requirement parameter is configured to specify the second identifier, and then the non-real-time codec is determined.
The first designation may be 0 and the second designation may be 1. It will be appreciated that the first designation may be 1 and the second designation may be 0. Or other characters are adopted as the first specified identification or the second specified identification.
In some non-limiting embodiments, the real-time performance requirement parameters of the codecs corresponding to the multiple instances respectively may be obtained through a call interface preconfigured by a hardware abstraction layer (Hardware Abstraction Layer, HAL).
For example, the real-time performance requirement parameter is Priority, and a Priority calling interface for calling Priority may be configured in advance.
As another example, the real-time performance requirement parameter is OperatingRate, and an OperatingRate calling interface for calling the OperatingRate may be preconfigured.
A codec may generally comprise two parts, an encoder and a decoder. Wherein the encoder has an encoding component that performs a compression (encoding) function. The decoder has a decoding component that performs a decompression (decoding) function. It will be appreciated that some codecs may have both an encoder and a decoder, and that some may have only an encoder, or only a decoder.
Taking the real-time performance requirement parameters as Priority and OperatingRate as examples, the Priority and OperatingRate are configured by the upper layer application. A common class of parameters may be encapsulated for storing or managing the parameters of the codec components (coding and decoding components). The encoding and decoding may define the Priority and OperatingRate of the codec, which may be stored as component parameter parent classes in the parameter common class. The encoding component or decoding component inherits from the component parameter parent, i.e., the Priority and OperatingRate can be obtained from the component parameter parent. Specifically, the interfaces getpriority_l (), getoperatingrate_l () are defined in the member functions of the parameter common class at the same time, the Codec component can obtain the Priority of the corresponding Codec by calling the getpriority_l () interface, and can obtain the OperatingRate of the Codec by calling the interface getoperatingrate_l ().
Referring to fig. 2, a flowchart of acquiring real-time performance requirement parameters is provided, and a specific process of acquiring real-time performance requirement parameters is described below by taking the real-time performance requirement parameters as Priority and OperatingRate as an example. The method specifically comprises the following steps:
and step 21, acquiring the Priority and operatingRate parameters set by the upper layer application.
The Codec parameters (Codec configuration) may include Priority and OperatingRate.
In step 25, the encoding component or decoding component inherits from the component parameter parent.
In step 26, the encoding component or decoding component obtains the Priority and operatingRate from the component parameter parent.
In a specific implementation, when multiple instances are concurrent, the codec corresponding to the multiple instances determined according to the real-time performance requirement parameter may include three cases: only real-time codecs; only non-real time codecs; and includes both real-time codecs and non-real-time codecs.
In the implementation of step 12, it is determined that the codecs corresponding to the multiple instances respectively include a real-time codec and a non-real-time codec according to the real-time performance requirement parameter of each codec. Before the first frame data encoding and decoding or after the last frame data encoding and decoding processing is finished, judging whether the current actual performance meets the performance requirement of the real-time encoder-decoder; and if the current actual performance meets the performance requirement of the real-time codec, scheduling the work of the non-real-time codec according to the residual performance condition.
Further, if the current actual performance meets the performance requirement of the real-time codec, scheduling a plurality of non-real-time codecs to work alternately according to the remaining performance condition. For example, if the number of non-real-time codecs is two, denoted as non-real-time Codec1 and non-real-time Codec2, respectively, and the current actual performance meets the performance requirement of the real-time Codec, the non-real-time Codec1 and the non-real-time Codec2 are scheduled alternately.
In some non-limiting embodiments, the current actual performance meets the performance requirements of the real-time codec, and the identification of the current real-time codec is deleted from the real-time codec sequence, which is used to indicate the active real-time encoder; and scheduling the non-real-time coder and decoder in waiting to work if the number of the non-real-time coder and decoder which wait to meet the condition that the real-time coder and decoder sequence is empty and the number of the non-real-time coder and decoder which work meets the specified number.
Further, waiting for the number of non-real-time codecs that satisfy the real-time codec sequence being empty and working to satisfy a specified number includes: waiting to satisfy the number of non-real-time codecs that the real-time codec sequence is empty and that is working is zero, i.e. to satisfy the real-time codec sequence that is empty and that no non-real-time codec is working.
And if the currently working non-real-time coder/decoder finishes coding/decoding one frame of data, notifying the waiting non-real-time coder/decoder to work. Wherein the non-real time codec is suspended while waiting to avoid occupation of resources (e.g., central Processing Unit (CPU) resources).
In an implementation, if the current actual performance does not meet the performance requirement of the real-time codec, suspending the operation of the non-real-time codec so that the non-real-time codec is waiting. The non-real-time codec in standby will temporarily stop the codec operation.
In some non-limiting embodiments, it is determined whether the current actual performance meets the performance requirements of the real-time codec in the following manner: judging whether the actual time required by encoding and decoding a frame at present is greater than theoretical time or not; if the actual time is greater than the theoretical time, judging that the actual performance does not meet the performance requirement of the real-time codec; and if the actual time is not greater than the theoretical time, judging that the actual performance meets the performance requirement of the real-time coder-decoder.
In other non-limiting embodiments, taking decoding as an example, it is determined whether the current actual decoding frame number meets the frame rate requirement of the real-time codec, and if the current actual decoding frame number meets the frame rate requirement of the real-time codec, it is determined that the actual performance meets the performance requirement of the real-time codec. Accordingly, if the current actual decoding frame number does not meet the frame rate requirement of the real-time codec, it is determined that the actual performance does not meet the performance requirement of the real-time codec.
For example, the real-time codec requires a frame rate of 30 frames per second, and at 2 seconds, it is determined whether 60 frames have been decoded, and if 60 frames are not reached, it is determined that the actual performance does not meet the performance requirements of the real-time codec. Scheduling the real-time codec operation, suspending the non-real-time codec operation such that the non-real-time codec is waiting. If 60 frames have been decoded, it is determined that the actual performance meets the performance requirements of the real-time codec, and non-real-time codec operation may be scheduled.
In a specific implementation, after the instance is created, the Priority and the OperatingRate of the codec corresponding to the current instance may be obtained. And scheduling the codec according to the two parameters, namely Priority and operatingRate. The scheduling of the codec may be performed using a resource scheduling module. For example, the Codec component may invoke a resource scheduling function to determine whether the currently operating Codec is a real-time Codec or a non-real-time Codec based on Priority and OperatingRate. For example, if the OperatingRate value is 0 or less, it is determined as the non-real-time Codec. If the real-time Codec is the real-time Codec, judging whether the actual time required by actually encoding and decoding one frame is larger than the theoretical time, if the actual time is larger than the theoretical time, the system cannot meet the current real-time Codec performance requirement, and adding the current Codec into the real-time Codec sequence. The process () function is executed, and is called once every time a frame of data is processed, and the process () function is executed in a loop until the codec task ends. The resource scheduling module is arranged on the HAL layer and is used for scheduling the coder and the decoder before coding and decoding each frame of data. Each frame of data may be each frame of image data or each frame of audio data.
Otherwise, the system can meet the current real-time Codec performance, and then the current Codec is deleted in the real-time Codec sequence, namely the system performance is surplus, and the non-real-time Codec is scheduled to start working.
After the non-real-time Codec one frame is finished (i.e., the process () function is executed), deleting the non-real-time Codec of the one frame after the non-real-time Codec is finished in the non-real-time Codec sequence, and notifying (notify) the waiting non-real-time Codec for realizing the alternate Codec of the non-real-time Codec, wherein the waiting non-real-time Codec receiving the notification can work when the condition is satisfied (e.g., the real-time Codec sequence is empty and the number of the working non-real-time codecs satisfies the specified number).
In specific implementation, when multiple instances are concurrent, determining the working frequency of the codec corresponding to each instance according to the frame size and the frame rate required by each instance; determining a final working frequency according to the working frequency of the corresponding codec of each instance; and controlling the operation of the codec by adopting the final operation frequency, wherein the final operation frequency meets the frequency requirements of all the operated codecs.
Taking video codec as an example, the frame size may be the size of an image, such as a pixel of an image.
In some embodiments, the threshold may be preset, and different thresholds correspond to different frequency levels, respectively. For each instance, a product of a frame size (frame_size) and a frame rate (frame_rate) is calculated, a frequency level is determined according to a relationship between the product and a threshold, and then an operating frequency corresponding to the codec corresponding to the instance is determined according to the frequency level.
For example, 4 frequency classes are defined, class 1, class 2, class 3 and class 4 respectively. The operating frequencies of the different frequency classes are different.
4 stages: frame_rate_size is greater than or equal to 2560 x 1440 x 30;
3 stages: frame_rate frame_size is equal to or greater than 1920×1080×30;
2 stages: frame_rate_size is equal to or greater than 1280 x 720 x 30;
stage 1: else, where else is the other case, refers to the case where no level 2, level 3, or level 4 is satisfied.
In other embodiments, the working frequency corresponding to the codec may also be determined according to the codec priority corresponding to the instance and the operation rate. Specifically, under the minimum requirement of the working frequency of the single instance rate, the priority of the codec and the corresponding mapping relation between the operation rate and the working frequency can be preconfigured and stored. And then searching the corresponding working frequency from the mapping relation according to the priority of the coder and the decoder and the operation rate.
In still other embodiments, for each instance, a frequency level corresponding to the codec corresponding to the instance may be determined according to a product of a frame size (frame_size) and a frame rate (frame_rate), a frequency level corresponding to the codec corresponding to the instance may be determined according to a relationship between the product and a threshold, and a final operating frequency may be determined according to a codec priority and an operating rate corresponding to the instance. Because the chip performance of different terminals is different, the final working frequency is determined by firstly determining the frequency grade and then comprehensively considering the priority of the coder and the decoder and the operation rate, the working frequency matched with the terminal can be determined, and the coding and decoding performance of the system can be fully exerted.
For example, 4 examples are concurrent, example 1, example 2, example 3, and example 4, respectively. Wherein example 1 is playing video in the format: advanced video coding is abbreviated (Advanced Video Coding, AVC), resolution 1080p, transmission frames per second (fps) 30. Example 2 is: video is recorded in AVC format with a resolution of 2k, transmission frames per second (fps) 30. Example 3 is video transcoding, converting resolution from 1080p to 720p. Example 4 video transcoding converts resolution from 2k to 1080p.
The real-time performance requirement parameter of example 1 is Priority 0 and operatingrate 30. The real-time performance requirement parameter of example 2 is Priority 0 and operatingrate 30. The real-time performance requirement parameter of example 3 is Priority 1. Example 4 did not set the real-time performance requirement parameters, i.e., did not set the Priority and OperatingRate.
Based on the real-time performance requirement parameter determination of each example, the codec corresponding to example 1 is a real-time codec, the codec corresponding to example 2 is a real-time codec, the codec corresponding to example 3 is a non-real-time codec, and the codec corresponding to example 4 is a non-real-time codec.
The frequency class division manner may be the manner of 1 level, 2 level, 3 level and 4 level, or may be the manner of Low frequency band (Low FRQ), intermediate frequency band (Mid FRQ), high frequency band (high FRQ), etc., where the working frequency corresponding to the Low frequency band satisfies the first frequency range, the working frequency corresponding to the intermediate frequency band satisfies the second frequency range, the working frequency corresponding to the high frequency band satisfies the third frequency range, and the working frequencies corresponding to the Low frequency band, the intermediate frequency band, and the high frequency band are sequentially increased.
For example, priority is 1, 2, 3, respectively corresponding to Highest FRQ, mid FRQ, low FRQ. The Codec corresponding to example 1 is a real-time Codec, and the minimum operating frequency requirement for single-instance rate is met only by running on Low FRQ. The corresponding Codec of example 2 is also a real-time Codec, again requiring only Low FRQ at the lowest. The corresponding Codec of example 3 is a non-real time Codec, running at Highest FRQ according to the set priority parameters. Example 4 the corresponding Codec was a non-real time Codec, run on Mid FRQ.
If the Codec for example 1, example 2, example 3 and example 4 are running simultaneously, the final operating frequency setting needs to be high FRQ required for the Codec for example 3 according to the maximum requirement.
If the real-time codecs of examples 1, 2 and the non-real-time Codec of example 4 (Priority set to 2) are running simultaneously, the non-real-time Codec of example 4 needs to run at Mid FRQ according to the Highest frequency requirement of a single Codec, but since Mid FRQ does not meet the rate requirements of the real-time codecs of examples 1 and 2, the final operating frequency needs to be set to Highest FRQ.
In some embodiments, when a new instance is detected, the final operating frequency is redetermined based on multiple instances concurrent after the new instance.
In some embodiments, a hardware codec may be used, where the codec component invokes a hardware codec library to implement a codec function, i.e., the codec component invokes a driver side, and ultimately invokes a system-on-a-chip (SOC) to perform the codec, so selecting an appropriate hardware codec operating frequency plays a vital role in fully exploiting the system codec performance. The final working frequency can be selected according to parameters of frame size, frame rate, priority and OperatingRate, and the codec component obtains the Priority and OperatingRate, and the two parameters are transmitted to the driving side through a DecInit () function and an encInit () function to be used for decoding, and the encInit () function is used for encoding.
As can be seen from the above, when multiple instances are concurrent, the real-time performance requirement parameters of the codecs corresponding to the multiple instances are obtained, where the real-time performance requirement parameters are used to determine whether the codec is a real-time codec. And scheduling the work of each coder-decoder according to the current performance meeting condition of the actual real-time coder-decoder and the real-time performance requirement parameter of each coder-decoder. Because the scheduling priority of the real-time codec is higher than that of the non-real-time codec, the scheduling of the multi-codec is performed according to the current actual performance meeting condition of the real-time codec and the real-time performance requirement parameter of each codec, the work of the real-time codec can be scheduled preferentially, so that the work of the non-real-time codec can be scheduled under the condition that the performance requirement of the real-time codec is met, the system performance can be fully utilized, and the user experience is improved.
Referring to fig. 3, a flowchart of a method for scheduling a codec in a multi-instance case in a typical application scenario is presented. For easy understanding, in this embodiment, three concurrent examples, namely, example 1, example 2 and example 3 are taken as examples, and the scheduling method may specifically include the following steps.
In step 303, example 3 corresponds to non-real time Codec2.
In some non-limiting embodiments, the use of codec dependent operations are typically detected in accordance with the corresponding instance of the use of codec dependent operations, i.e. the corresponding instance is created.
In this case, according to the creation time of example 1, example 2, and example 3, steps 301, 302, and 303 may be performed simultaneously or may not be performed simultaneously. Examples 1, 2 and 3 coexist at least for a certain period of time.
After creating the corresponding instance, it may be determined whether the codec is a real-time codec according to real-time performance requirement parameters Priority and/or OperatingRate of the respective corresponding codec of each instance. Wherein, the Codec corresponding to example 1 is real-time Codec1, the Codec corresponding to example 2 is non-real-time Codec1, and the Codec corresponding to example 3 is non-real-time Codec2.
If the determination result is negative, i.e. the codec is not finished, step 305 is executed. If yes, namely the encoding and decoding are finished, ending the flow. Taking video as an example, the codec ending may include several cases, for example, detecting that video playback is canceled, such as closing the played video, ending the codec. For another example, when the definition of the video playing is switched, the encoding and decoding of the instance corresponding to the definition of the previous video are finished, and the instance corresponding to the definition after the switching is created again. For another example, after the video is played, the encoding and decoding corresponding to the example is finished.
If yes, go to step 307; when the determination result is negative, step 308 is performed.
In a specific implementation, the theoretical time refers to the codec time required for an instance of the real-time codec, which can be calculated from the frame rate required for the instance.
In particular implementations, a real-time Codec queue may be configured to indicate the operation of the real-time Codec and a non-real-time Codec queue to indicate the operation of the non-real-time Codec. The real-time Codec queue has a scheduled real-time Codec identification. The non-real time Codec queue has a non-real time Codec identification. The Codec identifiers are in one-to-one correspondence with the codecs and are used for identifying the codecs.
In step 307, real-time Codec performance cannot be guaranteed, limiting the operation of non-real-time codecs.
In a specific implementation, when the real-time Codec performance cannot be met, that is, the system cannot guarantee the real-time Codec performance, if the real-time Codec identifier exists in the real-time Codec queue, the real-time Codec identifier is maintained in the real-time Codec queue. If the real-time Codec queue has no real-time Codec identifier, the real-time Codec identifier is added to the real-time Codec queue.
When there are multiple real-time codecs, if the performance of all the real-time codecs is not satisfied, all the real-time Codec identifiers are in the real-time Codec queue.
In some non-limiting embodiments, if none of the real-time Codec's performance in the plurality of real-time Codecs is met, a real-time Codec identification that does not meet the performance of the real-time Codec may be added to the real-time Codec queue.
Further, the non-real-time Codec may be notified that it is currently in a restricted operating state. Wherein limiting the non-real time Codec operation may be suspending a process function, such as a process () function, that executes an instance of the non-real time Codec. Since the process () function in the codec component is called once every frame data is processed when the codec is operated, the process () function is circulated until the codec task is completed. Suspending execution of the process () function suspends the codec operation of the codec.
In step 308, real-time Codec performance is met and non-real-time Codec operation is scheduled.
In a specific implementation, when the real-time Codec performance is satisfied, the real-time Codec identifier may be deleted from the real-time Codec queue, and the non-real-time Codec identifier may be added to the non-real-time Codec queue to schedule the non-real-time Codec operation.
Further, non-real time Codec operation may also be notified.
In implementations, a process () function may be performed, and when the process () function is performed, the frame data codec is completed and the next frame data codec is entered.
After the execution of step 309, the execution of step 304 is continued.
After completion of the execution of step 308, a jump may be made to step 312 and/or step 317.
In particular implementations, whether step 308 is followed by step 312, step 317, or both steps 312 and 317 are performed, is related to the non-real time Codec operating configuration conditions. For example, if the operation configuration condition is that the non-real-time Codec is operated alternately, the previous frame schedules the non-real-time Codec1 to operate, the next frame schedules the non-real-time Codec2 to operate, and so on, until the Codec of the instance is finished. For another example, if the operation configuration condition is that the non-real-time Codec is operated simultaneously, the non-real-time Codec1 and the non-real-time Codec2 are scheduled to be operated simultaneously.
Taking alternate scheduling of non-real time Codec operations as an example, step 312 is performed when scheduling non-real time Codec1 operations.
Step 312 waits for the real-time Codec sequence to be empty and no non-real-time Codec to operate.
Step 313, process each frame of data.
Each frame of data is processed, i.e., a process () function corresponding to the non-real-time Codec is performed.
That is, when the process () function execution is completed, the executed non-real-time Codec function is deleted from the non-real-time Codec sequence.
After the execution of step 314 is completed, the steps 310 and 311 are continued. Step 310 and step 304 and step 315 may be the same step, i.e., step 310, step 304 and step 315 are shared. Step 311 and steps 305 and 316 may be the same step, i.e. step 311 is shared with steps 305 and 316. In this way, the system operation overhead can be saved. Thus, whether step 305 or step 311 or step 316 is performed, step 306 is entered.
After the execution of step 314 is completed, the encoding and decoding of the current frame may also have been completed by non-real-time Codec 2.
When non-real time Codec2 operation is scheduled, step 317 is performed.
At step 317, the real-time Codec sequence is waiting to be empty and no non-real-time Codec is working.
At step 318, each frame of data is processed.
Each frame of data is processed, i.e., a process () function corresponding to the non-real-time Codec is performed.
That is, when the process () function execution is completed, the executed non-real-time Codec function is deleted from the non-real-time Codec sequence.
After completion of the execution of step 319, the encoding and decoding of the current frame may also have been completed by non-real-time Codec 1.
The step 312 is described as a suitability of alternately scheduling non-real-time Codec operation when real-time Codec performance is satisfied. When the scheduling conditions of the non-real-time Codec are configured as other conditions, the specific content of step 312 may be adjusted to the corresponding configuration conditions, which are not exemplified here.
The embodiment of the present invention further provides a device for scheduling a codec in multiple instances, and referring to fig. 4, a schematic structural diagram of a device for scheduling a codec in multiple instances in the embodiment of the present invention is given, where a device 40 for scheduling a codec in multiple instances includes:
an obtaining unit 41, configured to obtain real-time performance requirement parameters of codecs corresponding to the multiple instances when the multiple instances are concurrent, where the real-time performance requirement parameters are used to determine whether the codec is a real-time codec;
And a scheduling unit 42, configured to schedule the operation of each codec according to the performance meeting condition of the current actual real-time codec and the real-time performance requirement parameter of each codec, where the scheduling priority of the real-time codec is higher than that of the non-real-time codec.
In specific implementation, the specific working principle and workflow of the codec scheduling device 40 under multiple instances may be referred to the description of the codec scheduling method under multiple instances provided in any one of the foregoing embodiments, which is not repeated herein.
In a specific implementation, the above-described scheduling apparatus 40 of the codec under multiple instances may correspond to a Chip in the terminal having a scheduling function of the codec under multiple instances, such as an SOC (System-On-a-Chip), a baseband Chip, etc.; or corresponds to a chip module including a scheduling function having a codec in multiple instances in the terminal; or corresponds to a chip module having a chip with a data processing function, or corresponds to a terminal.
The embodiments of the present invention further provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the method for scheduling codecs under multiple instances provided by any of the embodiments of the present invention.
The computer readable storage medium may include non-volatile memory (non-volatile) or non-transitory memory, and may also include optical disks, mechanical hard disks, solid state disks, and the like.
Specifically, in the embodiment of the present invention, the processor may be a central processing unit (central processing unit, abbreviated as CPU), and the processor may also be other general purpose processors, digital signal processors (digital signal processor, abbreviated as DSP), application specific integrated circuits (application specific integrated circuit, abbreviated as ASIC), field programmable gate arrays (field programmable gate array, abbreviated as FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
It should also be appreciated that the memory in embodiments of the present application may be either volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The nonvolatile Memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an electrically Erasable ROM (Electrically EPROM, EEPROM), or a flash Memory. The volatile memory may be a random access memory (Random Access Memory, RAM for short) which acts as an external cache. By way of example but not limitation, many forms of random access memory (Random Access Memory, abbreviated as RAM) are available, such as Static random access memory (Static RAM, abbreviated as SRAM), dynamic Random Access Memory (DRAM), synchronous dynamic random access memory (Synchronous DRAM, abbreviated as SDRAM), double data rate Synchronous dynamic random access memory (Double Data Rate SDRAM, abbreviated as DDR SDRAM), enhanced Synchronous dynamic random access memory (Enhanced SDRAM, abbreviated as ESDRAM), synchronous link dynamic random access memory (Synchronous DRAM, abbreviated as SLDRAM), and direct memory bus random access memory (Direct Rambus RAM, abbreviated as DR RAM).
The embodiment of the invention also provides a terminal, which comprises a memory and a processor, wherein the memory stores a computer program capable of running on the processor, and the processor executes the steps of the method for scheduling the codec under the multiple instances provided by any embodiment when running the computer program.
The memory is coupled to the processor and may be located within the terminal or external to the terminal. The memory and the processor may be connected by a communication bus.
The terminal can include, but is not limited to, terminal equipment such as a mobile phone, a computer, a tablet personal computer, a digital television, a smart watch, a vehicle recorder and the like, and can also be a server, a cloud platform and the like.
The above embodiments may be implemented in whole or in part by software, hardware, firmware, or any other combination. When implemented in software, the above-described embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present application are all or partially produced. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer program may be stored in or transmitted from one computer readable storage medium to another, for example, by wired or wireless means from one website, computer, server, or data center.
In the several embodiments provided in the present application, it should be understood that the disclosed method, apparatus, and system may be implemented in other manners. For example, the device embodiments described above are merely illustrative; for example, the division of the units is only one logic function division, and other division modes can be adopted in actual implementation; for example, multiple units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may be physically included separately, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in hardware plus software functional units. For example, for each device or product applied to or integrated on a chip, each module/unit included in the device or product may be implemented in hardware such as a circuit, or at least part of the modules/units may be implemented in software program, where the software program runs on a processor integrated inside the chip, and the rest (if any) of the modules/units may be implemented in hardware such as a circuit; for each device and product applied to or integrated in the chip module, each module/unit contained in the device and product can be realized in a hardware manner such as a circuit, different modules/units can be located in the same component (such as a chip, a circuit module and the like) or different components of the chip module, or at least part of the modules/units can be realized in a software program, the software program runs on a processor integrated in the chip module, and the rest (if any) of the modules/units can be realized in a hardware manner such as a circuit; for each device, product, or application to or integrated with the terminal, each module/unit included in the device, product, or application may be implemented by using hardware such as a circuit, different modules/units may be located in the same component (for example, a chip, a circuit module, or the like) or different components in the terminal, or at least part of the modules/units may be implemented by using a software program, where the software program runs on a processor integrated inside the terminal, and the remaining (if any) part of the modules/units may be implemented by using hardware such as a circuit.
It should be understood that the term "and/or" is merely an association relationship describing the associated object, and means that three relationships may exist, for example, a and/or B may mean: a exists alone, A and B exist together, and B exists alone. In this context, the character "/" indicates that the front and rear associated objects are an "or" relationship.
The term "plurality" as used in the embodiments herein refers to two or more.
The first, second, third, etc. descriptions in the embodiments of the present application are only used for illustrating and distinguishing description objects, and no order division is used, nor does it indicate that the number of devices in the embodiments of the present application is particularly limited, and no limitation in the embodiments of the present application should be construed.
It should be noted that the serial numbers of the steps in the present embodiment do not represent a limitation on the execution sequence of the steps.
Although the present invention is disclosed above, the present invention is not limited thereto. Various changes and modifications may be made by one skilled in the art without departing from the spirit and scope of the invention, and the scope of the invention should be assessed accordingly to that of the appended claims.
Claims (12)
1. A method for scheduling a codec in a multi-instance system, comprising:
When a plurality of examples are concurrent, acquiring real-time performance requirement parameters of the coder and the decoder corresponding to the plurality of examples respectively, wherein the real-time performance requirement parameters are used for determining whether the coder and the decoder are real-time coder and decoder or not;
and scheduling the work of each codec according to the current performance meeting condition of the actual real-time codec and the real-time performance requirement parameter of each codec, wherein the scheduling priority of the real-time codec is higher than that of the non-real-time codec.
2. The multi-instance codec scheduling method of claim 1, wherein the scheduling operation of each codec based on the performance satisfaction of the current actual real-time codec and the real-time performance requirement parameter of each codec comprises:
determining that the codecs corresponding to the multiple instances respectively comprise a real-time codec and a non-real-time codec according to the real-time performance requirement parameter of each codec;
before the first frame data encoding and decoding or after the last frame data encoding and decoding processing is finished, judging whether the current actual performance meets the performance requirement of the real-time encoder-decoder;
and if the current actual performance meets the performance requirement of the real-time codec, scheduling the work of the non-real-time codec according to the residual performance condition.
3. The multi-instance codec scheduling method of claim 2, wherein the number of non-real-time codecs is plural, and the scheduling the non-real-time codec according to the remaining performance if the current actual performance meets the performance requirement of the real-time codec comprises:
and if the current actual performance meets the performance requirement of the real-time codec, scheduling a plurality of non-real-time codecs to work alternately according to the residual performance condition.
4. A method for scheduling a multi-instance codec as recited in claim 3, wherein said scheduling a plurality of non-real-time codecs to operate alternately based on the remaining performance if the current actual performance meets the performance requirement of said real-time codec comprises:
the current actual performance meets the performance requirement of the real-time coder and decoder, and the identification of the current real-time coder and decoder is deleted from the real-time coder and decoder sequence, wherein the real-time coder and decoder sequence is used for indicating the working real-time coder and decoder;
and scheduling the non-real-time coder and decoder in waiting to work if the number of the non-real-time coder and decoder which wait to meet the condition that the real-time coder and decoder sequence is empty and the number of the non-real-time coder and decoder which work meets the specified number.
5. The multi-instance codec scheduling method of claim 4, further comprising:
and if the currently working non-real-time coder/decoder finishes coding/decoding one frame of data, notifying the waiting non-real-time coder/decoder to work.
6. The multi-instance codec scheduling method of claim 2, further comprising:
if the current actual performance does not meet the performance requirement of the real-time codec, suspending the operation of the non-real-time codec so that the non-real-time codec is waiting.
7. The multi-instance codec scheduling method of claim 2, wherein determining whether the current actual performance meets the performance requirement of the real-time codec is performed by:
judging whether the actual time required by encoding and decoding a frame at present is greater than theoretical time or not;
if the actual time is greater than the theoretical time, judging that the actual performance does not meet the performance requirement of the real-time codec;
and if the actual time is not greater than the theoretical time, judging that the actual performance meets the performance requirement of the real-time coder-decoder.
8. The multi-instance codec scheduling method of claim 2, further comprising:
when a plurality of examples are concurrent, determining the working frequency of a coder-decoder corresponding to each example according to the frame size and the frame rate required by each example;
determining a final working frequency according to the working frequency of the corresponding codec of each instance;
and controlling the operation of the codec by adopting the final operation frequency, wherein the final operation frequency meets the frequency requirements of all the operated codecs.
9. The method for scheduling a codec in multiple instances according to claim 1, wherein the obtaining real-time performance requirement parameters of the codec corresponding to each of the multiple instances includes:
and acquiring real-time performance requirement parameters of the codecs corresponding to the multiple instances respectively through a call interface preconfigured by the hardware abstraction layer.
10. A multi-instance codec scheduling apparatus, comprising:
the system comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring real-time performance requirement parameters of a coder and a decoder corresponding to a plurality of examples respectively when the plurality of examples are concurrent, and the real-time performance requirement parameters are used for determining whether the coder and the decoder are real-time coder and decoder or not;
And the scheduling unit is used for scheduling the work of each coder and decoder according to the current performance meeting condition of the actual real-time coder and decoder and the real-time performance demand parameter of each coder and decoder, wherein the scheduling priority of the real-time coder and decoder is higher than that of the non-real-time coder and decoder.
11. A computer readable storage medium having stored thereon a computer program, characterized in that the computer program when run by a processor performs the steps of the method of scheduling a codec in a multi-instance according to any one of claims 1 to 9.
12. A terminal comprising a memory and a processor, the memory having stored thereon a computer program capable of being run on the processor, characterized in that the processor, when running the computer program, performs the steps of the method of scheduling codecs under multiple instances of any of claims 1 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211483010.6A CN116126487A (en) | 2022-11-24 | 2022-11-24 | Multi-instance codec scheduling method and device, storage medium and terminal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211483010.6A CN116126487A (en) | 2022-11-24 | 2022-11-24 | Multi-instance codec scheduling method and device, storage medium and terminal |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116126487A true CN116126487A (en) | 2023-05-16 |
Family
ID=86299890
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211483010.6A Pending CN116126487A (en) | 2022-11-24 | 2022-11-24 | Multi-instance codec scheduling method and device, storage medium and terminal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116126487A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116055715A (en) * | 2022-05-30 | 2023-05-02 | 荣耀终端有限公司 | Scheduling method of coder and decoder and electronic equipment |
-
2022
- 2022-11-24 CN CN202211483010.6A patent/CN116126487A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116055715A (en) * | 2022-05-30 | 2023-05-02 | 荣耀终端有限公司 | Scheduling method of coder and decoder and electronic equipment |
| CN116055715B (en) * | 2022-05-30 | 2023-10-20 | 荣耀终端有限公司 | Scheduling method of coder and decoder and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220038724A1 (en) | Video stream decoding method and apparatus, terminal device, and storage medium | |
| US20200371820A1 (en) | Method for Rendering User Interface and Terminal | |
| US11336953B2 (en) | Video processing method, electronic device, and computer-readable medium | |
| CN106936671B (en) | Multi-screen interaction method and device and terminal equipment | |
| CN109032680B (en) | Real-time operating system quick start method and real-time operating system | |
| US9323545B2 (en) | Method, system, module, and storage medium for automatic adaptation of hardware encoding and decoding | |
| AU2019256257B2 (en) | Processor core scheduling method and apparatus, terminal, and storage medium | |
| CN111163345A (en) | Image rendering method and device | |
| CN109445918B (en) | Task scheduling method, device, terminal and storage medium | |
| US11949853B2 (en) | Data decoding method and apparatus, and data coding method and apparatus | |
| KR102528877B1 (en) | Video decoding control method, device, electronic device and storage medium | |
| US20220058772A1 (en) | Image Processing Method and Device | |
| CN115550500A (en) | Incoming call reminding method and terminal | |
| CN116126487A (en) | Multi-instance codec scheduling method and device, storage medium and terminal | |
| JP2022514198A (en) | Image enhancement control methods, devices, electronic devices and storage media | |
| KR20230039723A (en) | Projection data processing method and apparatus | |
| CN114339412A (en) | Video quality enhancement method, mobile terminal, storage medium and device | |
| US20230221969A1 (en) | Encoding scheduling method, server, client, and system for acquiring remote desktop | |
| CN114300007A (en) | WebRTC-based audio and video recording method and device | |
| CN113641431A (en) | Method and terminal equipment for enhancing display of two-dimensional code | |
| CN114079823A (en) | Video rendering method, device, equipment and medium based on Flutter | |
| CN117278796B (en) | Multi-area image data display method, device, playing equipment and storage medium | |
| CN109788233B (en) | Video layer updating method, device, equipment and medium | |
| US10687013B2 (en) | Proxy-pattern-based multimedia displaying method, device and apparatus | |
| US20240073415A1 (en) | Encoding Method, Electronic Device, Communication System, Storage Medium, and Program Product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |