CN107885596B

CN107885596B - Program optimization method for high-frequency access memory for audio and video decoding and encoding

Info

Publication number: CN107885596B
Application number: CN201710482436.2A
Authority: CN
Inventors: 叶雅敏
Original assignee: Individual
Current assignee: Individual
Priority date: 2017-06-22
Filing date: 2017-06-22
Publication date: 2021-06-15
Anticipated expiration: 2037-06-22
Also published as: CN107885596A

Abstract

The invention discloses a program optimization method for high-frequency access memory of audio and video coding, which comprises the following steps: the CPU synchronously checks the use condition of the virtual page; the CPU timely delivers the process control without authority to an operating system; the operating system gives control right to the application program through the runtime environment and provides the application program with abnormal supporting behaviors; an application program applies for virtual storage areas with various permission combinations and changes partial permissions of the virtual storage areas; secondly, pre-configuring the audio and video data in a section of continuous memory according to the audio and video type, and setting the last page of the continuous memory as a non-read-write permission; and controlling the code stream and returning information in time in a static mode or a dynamic mode, wherein the dynamic mode is completed by stopping the encoding process and adjusting the encoding parameters. The invention can solve the problems of system performance, stability and flexibility.

Description

Program optimization method for high-frequency access memory for audio and video decoding and encoding

Technical Field

The invention relates to the field of computer software, in particular to a program optimization method for high-frequency memory access.

Background

The high-frequency access, namely the access to the validity of the program memory area, greatly reduces the performance of the system, does not check the validity of the memory area, and has the problems of incorrect software, abnormal software and even no-end breakdown caused by the memory boundary crossing. This problem is also referred to as stability. Due to the limitation of system resources, stability and performance can be selected alternatively. This is particularly evident in the audio-visual field. In the process of audio and video coding and decoding, if the validity of the memory is checked, the CPU pipeline is interrupted, and the utilization rate of the CPU is increased. If a large amount of memories are applied in advance, the memory resources are greatly wasted, and the problem cannot be thoroughly solved.

Disclosure of Invention

The invention aims to overcome the defects in the prior art, and provides a program optimization method for high-frequency memory access in order to solve the problems of system performance, stability and flexibility.

The purpose of the invention is realized by the following technical scheme:

a program optimization method for high-frequency access memory of audio and video coding is characterized by firstly setting the following conditions: the CPU synchronously checks the use condition of the virtual page; the CPU timely delivers the process control without authority to an operating system; the operating system gives control right to the application program through the runtime environment and provides the application program with abnormal supporting behaviors; an application program applies for virtual storage areas with various permission combinations and changes partial permissions of the virtual storage areas;

secondly, pre-configuring the audio and video data in a section of continuous memory according to the audio and video type, and setting the last page of the continuous memory as a non-read-write permission; and controlling the code stream and returning information in time in a static mode or a dynamic mode, wherein the dynamic mode is completed by stopping the encoding process and adjusting the encoding parameters.

The authority comprises no read-write authority, read-only authority and read-write authority.

The dynamic mode comprises the following steps: when the CPU/GPU utilization rate is high, calling a corresponding function to reduce the CPU/GPU utilization rate; under the condition of network bandwidth shortage, the quantization step length is enlarged, a layering thought is adopted, layering measures are reduced, and the quality of audio and video is reduced until the range which can be accepted by vision and hearing is adjusted; under the condition of sufficient network bandwidth, the step length of quantization is reduced, a layering thought is adopted, layering measures are added, and the quality of the audio and video is enhanced until the interval which can be accepted by vision and hearing is adjusted; and preferentially ensuring the audio and video quality under the condition that CPU/GPU resources are idle.

When a section of continuous memory is applied in advance according to the type of the audio and video, a guard memory is included, and then the guard memory is set to have no read-write permission.

A program optimization method for high-frequency access memory for audio and video decoding comprises the following steps of firstly, detecting that the following conditions are met: the CPU synchronously checks the use condition of the virtual page; the CPU timely delivers the process control without authority to an operating system; the operating system gives control right to the application program through the runtime environment and provides the application program with abnormal supporting behaviors; an application program applies for virtual storage areas with various permission combinations and changes partial permissions of the virtual storage areas;

decoding is the reverse engineering of encoding, and then code stream analysis is carried out: if the video is the video, analyzing the width, height and YUV/RGB type of the video, and applying for a corresponding memory and simultaneously including a corresponding guard page; if the audio frequency exists, analyzing the sampling rate, the number of sound channels and the sampling digit, and applying for a corresponding memory and simultaneously including a corresponding guard page; and then starts decoding.

The static decoding mode can intercept memory access boundary crossing, and the dynamic decoding mode optimizes the memory more on the basis, because the decoding only needs to apply for part of the memory.

Compared with the prior art, the technical scheme of the invention has the following beneficial effects:

the method greatly reduces the requirement on the memory, and simultaneously enhances the cache capacity of the system, because the total number of the actual memory of each process is often limited by the operating system, the mode improves the hit rate of the cache, the exchange of the virtual memory and the interception of the read-write border crossing of the memory access, thereby improving the performance and the stability. And the corresponding information of the code can be dynamically updated in real time.

Detailed Description

The invention is further described below in connection with specific operations:

there must be several prerequisites to implement the functionality of the invention: 1. the CPU needs to synchronously check the use condition of the virtual page, including the authority of the page and the like; the CPU needs to transfer the process control without corresponding authority to the operating system in time; 3. the operating system can transfer control to the application program through the runtime environment; 4. the runtime environment may provide the behavior that the application needs to support exceptions; 5. the application program can apply for virtual storage areas with various authority combinations, including no read-write authority, read-only authority, read-write authority and the like; 6. the application program can dynamically change the authority of a certain part of the virtual storage area; 7. the application program has a mechanism to realize the abnormal processing mode of the initialized page access authority;

size of page: often determined by both the CPU architecture and the operating system, they set the size of each page through the memory management unit, typically 4096 pages often being written as pagesize in the case of a 32-bit operating system.

Initializing a page access authority exception handling mode:

the problem is described in detail here, and it is only required to use seh if windows is relatively simple. The linux/mac/ios/andriod adopts a signal mechanism.

The implementation mode is as follows:

first implementation

Firstly, installing a signal processing routine of the page without permission to access, and then saving and initializing corresponding registers and internal registers of the CPU. If the page has no authority to be recalled, adjusting each register according to the corresponding condition. Terminate prematurely, or re-execute an instruction. Or jump to a preset address and execute the operation downwards.

Second implementation

1. All created virtual storage areas should be shared;

2. transmitting data in a pipeline mode through a fork mode, creating a semaphore 1, defaulting to no signal, installing a signal processing routine of page no-authority access in a subprocess, judging whether the signal processing routine of page no-authority access is normal or not, if not, exiting the process, and returning to-1;

3. and the parent process always waits for the child process to quit and monitor the semaphore 1, then the parent process judges the quit code of the child process to judge whether the processing is normal or not and then judges whether the semaphore 1 has a signal within the timeout time, if so, the process is normal, and if not, the process is continued to be the process 3. If the child process data is enough, the semaphore 1 is triggered, and the parent process can normally execute the flow.

The method can replace the fork function with the vfork function, the parameters enable the parent and child processes to share the shared storage area without establishing a pipeline, and the effect is consistent and simpler.

The above is the processing flow of c + +, and different languages have different implementation modes. But the basic principle is the same, and in addition, different compilers may have a slight difference to the exception handling flow, and the following linux can use a signal capture mode. For the sake of simplicity and convenience of description, the english characters used in the following description have meanings indicated by CPU: a central processing unit; GPU: a graphics processor; YUV/RGB series and other series: the data format that exists after video decoding; PCM: the data format that exists after audio decoding; H264/H265/VP8/VP9/svac and other video encodings: the data format after video compression coding; ogg, opus, aac, mp3, etc. are data formats that exist after audio compression encoding.

In this way, there are additional aspects to note:

1. the operation is needed to be concise as much as possible in the abnormal page access authority;

2. if the page access authority is abnormal and the return fails, the corresponding parameters are not used and the initialization is needed in time;

3. the corresponding parameters and the corresponding resources should be initialized and applied for at the previous time.

Firstly, further, taking the processing flow of audio and video coding as an example:

in the process of audio and video coding, information needs to be returned timely, and coding flexibility, such as code stream control, is provided. Picture quality, and layering of video coding, scalable scalability, and other features. It can also be used to evaluate performance, encode combinations of functions, and system resource usage.

Such as logic to control the codestream: this embodiment adopts 2 ways. One is the static approach. A certain memory amount (taking the integer times of the page upwards) plus pagesize (the memory must be continuous) is applied in advance by assigning the type (I/B/P/si/sp) of the video. And then setting the last page with the permission of no reading and writing. The mode 2 is embodied in that the flexibility, the performance and the audio and video quality of the coding are more optimized according to the weight.

The first mode is a static mode:

1. and applying for the proper page number (including guard page) at one time. A plurality of memory detection points can be preset, the memory detection points are that the memory removes read-write permission, and the last page is required to have no read-write permission;

2. initializing coding parameters;

3. initializing page access authority exception handling;

4. normal encoding flow is carried out until a memory detection point or the last page is touched;

5. if the memory detection point is the memory detection point, counting system resources including CPU resources and bandwidth required by code stream expectation, and then modifying the memory pages into readable and writable pages, or setting a plurality of pages into readable and writable pages at one time, but not modifying the last page;

6. if the last page is the page, the code stream is proved to exceed the maximum allowable range, and failure is returned; the upper layer can re-encode whether to modify the corresponding encoding parameters, and the maximum allowable range is preferably divided according to the type encoding (aac/mp 3/svac/vp8/vp9/h264 avc/h264 svc/hevc/ogg/opus, etc.) of the output audio/video and the frame type (I frame/b frame/p frame/si/sp frame) of the video, and the method is simple, does not need much skill and has strong universality.

The second mode is a dynamic mode, as follows:

1. applying for a plurality of pages at one time, and presetting a plurality of memory detection points, wherein the memory detection points are about to remove read-write permission from the memory, and the last page is required to have no read-write permission;

2, initializing coding parameters;

3. initializing page access authority exception handling;

4. a normal encoding process, a memory detection point or a last page;

5. if the detection point is the memory detection point; counting system resources including CPU resources and bandwidth required by code stream expectation, and changing a coding structure (a layered structure, whether only a basic layer is needed or not), setting frame field adaptive parameters and whether to perform secondary fine coding (part) or not by adjusting corresponding coding parameters such as step length and closing or opening corresponding functions;

6. repeating step 5; under the conditions of stability and acceptable system resource loss, the method is flexibly applicable to various encoding technologies, and the encoding parameters are adjusted in time according to the system resources, so that the audio and video quality, the system resource utilization rate and the program are stable, flexible and highly consistent.

The dynamic mode can be used to quickly terminate the encoding process or adjust the corresponding encoding parameters in time. And initializing the encoder to re-encode, and if the encoding process is finished, adjusting encoding parameters in time when the memory detection point passes through the resources of the detection system. One aspect is simultaneously applied to all aspects of audio and video coding, and certain flexibility is achieved. On the other hand, the structure of the coding is changed, such as adjustment of the hierarchical structure and frame field adaptation.

The flexibility of dynamic coding also has the following points: calling a system function to reduce the CPU utilization rate under the condition of higher CPU/GPU utilization rate; under the condition of network bandwidth shortage, the quantization step length is properly enlarged, and if a layering thought is adopted, measures such as layering are reduced, and the quality of the audio and video is reduced. Under the condition of sufficient network bandwidth, the quantization step length is properly reduced, and if a layering thought is adopted, measures such as layering are added, and the quality of the audio and video is enhanced. And in the case that the CPU/GPU resources are idle, the 2 nd pass can be carried out for more detailed coding to provide audio and video quality. The upper application layer can set an appropriate detection point in the whole memory area. Therefore, the corresponding information of the coding and decoding can be dynamically counted and updated in real time at the memory detection point.

The dynamic and static coding is only directed at one-dimensional coding, multi-dimensional coding can be properly expanded, and each dimension is protected by one exception under the condition that the multiple dimensions are independent from each other, which is relatively complex. One dimension is often selected as the main one, and then in the page access authority exception processing, the operation of memory authority of all dimensions is properly adjusted or other controllable operation is performed.

Further, the processing mode of the discontinuous memory is as follows:

the above memory accesses tend to be always continuous, but rarely occur in this case. We will typically present a controlled offset position, and this is equally applicable. Now assuming that it has an access offset, by this method, the hit rate of the cache of instructions can be greatly improved. The guard page count is calculated as follows:

the guard page count is equal to the offset divided by the pagesize value, and if not an integer division, the value is added to 1.

The required memory number is applied in advance (the number is taken as an integral multiple of the page), then guard pages are added (continuous one-time application is needed, no read-write permission exists), then the following guard pages are set to have no read-write permission, and the process is the same as the static/dynamic mode.

Second, decoding process

Decoding is the reverse engineering of encoding and is the most likely to cause memory corruption, especially heap corruption. Programmers often spend a great deal of time and effort to troubleshoot such problems, and eventually often draw conclusions that the codestreams are always incorrect. This tends to be too passive and process stable, and is not desirable for project expectations. This has the possibility to actively intercept such behavior.

First, the data types are explained

Decoded data (YUV, RGB, PCM, etc type): planar follows packed.

For the PLANAr YUV format, the Y of all pixel points is stored continuously, then the U of all pixel points is stored, and then the V of all pixel points is stored.

For the YUV format of packed, Y, U, V of each pixel are stored in a continuous and crossed manner, and RGB also belongs to this type.

Data before decoding (H264, H265, VP8, VP9, ogg, opus, aac, mp3, jpg, etc.).

The first mode, static mode:

1. according to the length of source data, the integer rounded up to the page size is multiplied by 1, then a system interface is called to apply for a corresponding length at one time, the last page is set to have no read-write permission, and the first section of storage space is memorized;

2. copying a data source into a first section of storage space, and clearing the remaining data which is less than one page to be 0;

3. initializing page access authority exception handling;

4. starting code stream analysis;

4.1. if the video is the video, analyzing the width, the height and the decoded data type (YUV/RGB);

4.1.1. if the data is similar to the planer multi-dimension, generally 3-dimension, and few 4-dimension, the data can be directly used as 1-dimension in a similar packed mode, the required memory size is calculated, each dimension of the memory space and the page without the read-write permission can be respectively applied at one time, the memory space of the last page of each dimension is not provided with the read-write permission, a continuous memory with enough size and the page without the read-write permission can be applied at one time, and the third section of the memory space is memorized;

4.1.2. if 1-dimensional case: decoding to a range in the exception or signal callback, judging that the address is in the last page, and if so, indicating that the decoding fails; if the page is not set as the readable and writable right in the third section of storage space, a plurality of continuous pages can be set as readable and writable rights, but the right of the last page cannot be changed, and the previous 2 conditions are not satisfied and the return fails; the multidimensional case: multidimensional processing tends to always have 1-dimension as the primary and the rest as secondary; decoding to a range in an abnormal or signal callback, judging that the address is in the last page of the main 1-dimensional storage space in the third section, and if so, indicating that the decoding fails; if not and the address is in the third section of storage space, setting the page corresponding to each dimension as the read-write permission. The corresponding page of each dimension can also be continuously set, but the authority of the last page of each dimension cannot be changed. The former 2 cases are not satisfied, and failure is returned;

4.1.3. in the abnormal or signal callback, the system resource can be judged, for example, the CPU utilization rate is reduced by using a system call mode under the condition that the CPU resource is too high. And counting the decoding speed, increasing or decreasing the decoding threads timely, and adjusting the number of threads in the block of the GPU.

The steps 4.1.2 and 4.1.3 are repeated until the decoding source is out of bounds, or the decoding data is out of bounds, or the decoding is successful.

Mode two, dynamic mode:

dynamic approaches often require sufficient memory for decoding for non-uniform applications, but apply for a portion and then multiplex that portion. Because the decoded data are often very large, especially for high-definition and ultra-high-definition videos, the method is greatly beneficial to saving system memory resources, cache hit rate and less virtual memory exchange.

3. initializing page access authority exception handling;

4. starting code stream analysis;

4.1. calculating the length, applying for the length of a plurality of pages in one time in the third section of memory area, wherein the last page has no read-write permission, calculating the required cycle number and rounding up;

4.2. starting decoding, decoding to the range in the abnormal or signal callback, and judging whether to finish circulation;

4.2.1. if the loop is finished, directly returning to indicate that the decoding is successful;

4.2.2. if the circulation is not finished, copying the decoded data, and multiplexing a third section of memory area;

4.3. counting the decoding speed, monitoring the resource usage of the system and adjusting the resource usage of the system.

And repeating the steps of 4.2 and 4.3 until the decoding source is out of range, or the decoding data is out of range, or the decoding is successful.

The present invention is not limited to the above-described embodiments. The foregoing description of the specific embodiments is intended to describe and illustrate the technical solutions of the present invention, and the above specific embodiments are merely illustrative and not restrictive. Those skilled in the art can make many changes and modifications to the invention without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A program optimization method for high-frequency access memory of audio and video coding is characterized in that the following conditions are firstly set: the CPU synchronously checks the use condition of the virtual page; the CPU timely delivers the process control without authority to an operating system; the operating system gives control right to the application program through the runtime environment and provides the application program with abnormal supporting behaviors; an application program applies for virtual storage areas with various permission combinations and changes partial permissions of the virtual storage areas;

secondly, a section of continuous memory is applied in advance according to the type of the audio and video, and the last page of the continuous memory is set to have no read-write permission; and controlling the code stream and returning information in time in a static mode or a dynamic mode, wherein the dynamic mode is completed by multiplexing small segments of memories and adjusting coding parameters.

2. The method according to claim 1, wherein the permissions include no read-write permission, read-only permission, and read-write permission.

3. The method according to claim 1, wherein the dynamic mode comprises: when the CPU/GPU utilization rate is high, calling a corresponding function to reduce the CPU/GPU utilization rate; under the condition of network bandwidth shortage, the quantization step length is enlarged, a layering thought is adopted, layering measures are reduced, and the quality of audio and video is reduced; under the condition of sufficient network bandwidth, the step length of quantization is reduced, a layering thought is adopted, layering measures are added, and the quality of the audio and video is enhanced; and re-encoding the second time to ensure the audio and video quality under the condition that the CPU/GPU resources are idle.

4. The method according to claim 1, wherein a guard memory is included when a section of continuous memory is applied in advance according to the type of the audio and video, and then the guard memory is set to have no read-write permission.

5. A program optimization method for high-frequency access memory for audio and video decoding is characterized by firstly detecting that the following conditions are met: the CPU synchronously checks the use condition of the virtual page; the CPU timely delivers the process control without authority to an operating system; the operating system gives control right to the application program through the runtime environment and provides the application program with abnormal supporting behaviors; an application program applies for virtual storage areas with various permission combinations and changes partial permissions of the virtual storage areas;

decoding is the reverse engineering of encoding, and then code stream analysis is carried out: if the video is the video, analyzing the width, height and YUV/RGB type of the video, and applying for a corresponding memory and adding a corresponding guard page at the same time; if the audio is audio, analyzing the sampling rate, the number of channels and the sampling digit, and adding a corresponding guard page while applying for a corresponding memory; and then starts decoding.

6. The program optimization method for high-frequency access memory for audio/video decoding according to claim 5, wherein when one-time application for decoding of a part of memory, decoding is performed in a dynamic manner, and then the section of memory is multiplexed.