WO2020001063A1 - 内存管理方法以及相关装置 - Google Patents

内存管理方法以及相关装置 Download PDF

Info

Publication number
WO2020001063A1
WO2020001063A1 PCT/CN2019/077188 CN2019077188W WO2020001063A1 WO 2020001063 A1 WO2020001063 A1 WO 2020001063A1 CN 2019077188 W CN2019077188 W CN 2019077188W WO 2020001063 A1 WO2020001063 A1 WO 2020001063A1
Authority
WO
WIPO (PCT)
Prior art keywords
memory
compressed
page
frame buffer
pages
Prior art date
Application number
PCT/CN2019/077188
Other languages
English (en)
French (fr)
Inventor
周海林
曲波
曹斌
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2020001063A1 publication Critical patent/WO2020001063A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02—Addressing or allocation; Relocation
    • G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
    • G06F12/10—Address translation
    • G06F12/1009—Address translation using page tables, e.g. page table structures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02—Addressing or allocation; Relocation
    • G06F12/06—Addressing a physical block of locations, e.g. base addressing, module addressing, memory dedication
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
    • G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/60—Memory management
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding

Definitions

  • the invention relates to the field of video processing, in particular to a memory management method and a related device.
  • Double-rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDR SDRAM), also known as DDR memory
  • DDR memory Double-rate synchronous dynamic random access memory
  • the price of DDR memory is equal to or even exceeds that of the main chip. Therefore, reducing the use of DDR on the premise of the same business experience is an effective way to reduce the cost of video processing systems.
  • the frame buffer (also called frame memory) of a video decoder (VDEC) occupies a major part of the DDR memory. For example, 6-7 image frames (such as reference frames) ) Stored in the frame buffer. Therefore, in order to reduce the memory consumption in video processing, compression techniques are usually introduced in the reference frame access process, including lossless compression and lossy compression.
  • lossy compression takes advantage of the insensitivity of humans to certain frequency components in images or sound waves, and loses certain information in the data compression process. Therefore, lossy compression usually brings a large image compression rate, but Lossy compression introduces image errors. Due to the reference characteristics of the decoded reference frames, the errors are accumulated and amplified after iterating through the reference frames. Eventually, visually perceivable image damage may occur, causing picture quality (PQ). As a result, the lossy compression technique is usually not used in the reference frame access process of the decoding process.
  • Lossless compression is a more common technique used for reference frame access. It uses the statistical redundancy of data for compression. It can completely restore the original data by decompression without causing image distortion. Therefore, lossless compression technology is currently used in the decoding process. Although lossless compression does not cause data damage, the compression rate of lossless compression is not reliable. For example, 4K video also has a compression rate of only 20% or even cannot be compressed under lossless compression. So relatively speaking, the existing lossless compression technology cannot effectively save memory consumption. Therefore, how to effectively reduce the memory consumption of reference frame access during decoding becomes a problem.
  • the embodiments of the present invention provide a memory management method and related devices, which can effectively reduce the memory occupied by reference frame access during decoding.
  • an embodiment of the present invention provides a memory management method in video decoding.
  • the method includes: a video decoder decoding a code stream to obtain a first image frame, where the first image frame includes n tiles ( For example, n identical slices), each slice represents an image of a rectangular area in the first image frame, and n is an integer greater than or equal to 2; compressing the first image frame (for example, lossless compression) to obtain the first Two image frames, the second image frame includes n compressed slices; a frame buffer is allocated for the n compressed slices in memory; the n compressed slices are stored in the The memory space occupied in the frame buffer establishes a mapping relationship between a virtual page and a physical page, wherein the memory space occupied by each compressed slice includes one or more physical pages, and the n The page numbers of multiple virtual pages occupied by each compressed slice are consecutive; and the n compressed slices are stored in the frame buffer, respectively.
  • n tiles For example, n identical slices
  • each slice represents an image of a rectangular area in the first image
  • the video decoder in this embodiment of the present invention can compactly store each compressed (eg, losslessly compressed) slice in turn into a memory area with continuous virtual addresses in the frame buffer to achieve the collection of occupied memory space.
  • the page numbers of the physical pages occupied by these compressed slices may be discrete, but the page numbers of the virtual pages occupied by these compressed slices are continuous. Therefore, implementing the embodiments of the present invention can effectively reduce the memory occupation purpose of reference frame access during decoding.
  • the following two methods are designed to dynamically allocate frame memory and implement fine-grained memory allocation or memory reclamation. These two methods are respectively The memory reclamation mode and the pre-allocation mode (or on-demand allocation mode) are described below respectively.
  • a small block of memory such as 1MB or 512KB
  • a memory block is allocated each time.
  • Such a small block of memory is called a memory block.
  • a memory block pool of a decoding channel can be formed.
  • the frame buffer may be allocated to the n compressed slices in a memory block pool in memory, where the frame buffer includes M memory blocks (for example, each memory block is 1MB, Then a 15MB frame buffer requires 15 memory blocks), each memory block includes multiple physical pages, which correspond to multiple virtual pages with consecutive page numbers, but the physical pages corresponding to these virtual pages can be Are discrete (ie the page numbers of these physical pages can be discontinuous).
  • the memory space occupied by the n compressed slices is less than or equal to the memory space corresponding to the M memory blocks.
  • the unused remaining memory blocks in the frame buffer are removed and recycled to the memory block pool.
  • the remaining memory blocks can be used for subsequent frame buffer allocation and slice storage, thereby realizing memory
  • the dynamic use of space greatly improves the efficiency of memory usage.
  • the process of establishing a mapping relationship between a virtual page and a physical page for the memory space occupied by the n compressed slices in the frame buffer is as follows: finding no occupation in the memory block pool Memory blocks, each of which corresponds to a page table, which records the mapping relationship between each virtual page and the physical page in the corresponding memory block. Then, a sufficient amount of memory blocks (M memory blocks) are combined into a storage space of the size indicated by the container. For example, when the virtual memory space is 15MB, several memory blocks with a total of 15MB are selected as the physical memory required by the container. Space, for example, combining 15 1MB memory blocks.
  • the page tables corresponding to the several memory blocks with a total of 15MB can be combined into a page table of a frame buffer, so that a mapping relationship between virtual addresses and physical addresses in the frame buffer is actually established.
  • a mapping relationship between virtual addresses and physical addresses in the frame buffer is actually established.
  • the compact memory can be stored in the allocated frame buffer after lossless compression of image frames, and the physical memory space is refined to realize the centralized utilization of fragmented memory and video decoding.
  • the decoder decodes an image frame and stores it in the frame buffer, it can recycle the unused memory of the frame buffer, release the unoccupied free memory, and merge it into the next frame for decoding. Because lossless compression does not lose image quality, this way, you can use less memory budget to actually obtain more frame buffers, thereby ensuring that various performance indicators are not damaged, and under the premise of not losing image quality The purpose of saving both bandwidth and frame memory.
  • the frame buffer that can be allocated for image frames before decoding includes only virtual memory space, not physical memory space.
  • the virtual memory space includes multiple page numbers consecutively.
  • Virtual page That is to say, the page table created when the frame buffer is allocated initially only indicates one or more virtual pages, but does not indicate the mapping relationship between virtual pages and physical pages.
  • this on-demand allocation scheme It does not involve memory reclamation, but allocates virtual memory space in advance before the actual physical memory is allocated. For example, when decoding, 15 frame buffers need to be allocated, and each frame buffer occupies 15 MB of space, so a full 15 MB ⁇ 15 space is allocated in the virtual memory space.
  • each virtual page corresponding to it may not have a physical page mapped to it.
  • the page table query is performed on the page numbers of the one or more virtual pages. If the page table does not have one or more virtual pages, the page numbers and The mapping relationship between the page numbers of the physical pages, then allocate the one or more physical pages for the one or more virtual pages in the memory, thereby realizing the allocation of actual physical pages according to the actual needs of the data unit .
  • a bitmap can be queried.
  • the bitmap includes multiple bits, and each bit is used to indicate whether a physical page in the memory is Occupied, and then find one or more physical pages that are not occupied according to the bitmap. For example, each bit in the bitmap indicates the occupancy status of a physical page, where "0" indicates that the corresponding physical page is not occupied, and "1" indicates that the corresponding physical page has been occupied.
  • each bit in the bitmap indicates the occupancy status of a physical page, where "0" indicates that the corresponding physical page is not occupied, and "1" indicates that the corresponding physical page has been occupied.
  • it can find four physical pages indicated by four bits with a value of "0" according to the bitmap.
  • the process of establishing the mapping relationship between the virtual page and the physical page of the frame buffer is as follows: When it is necessary to store the compressed data unit into the frame buffer, according to one or more virtual pages corresponding to the data unit A page table query for the page number of the page, if the page number of the physical page corresponding to the page number of the one or more virtual pages cannot be queried, then the unoccupied one or more physical pages are found in the memory, and all The page number of the one or more physical pages is added to the page table so that each of the page numbers of the one or more virtual pages in the page table corresponds to a page number of a physical page, respectively. In this way, the mapping relationship between the virtual page and the physical page of the frame buffer is established.
  • the embodiment of the present invention achieves "on-demand" memory by improving the page table establishment mechanism.
  • compact memory arrangements are used to store the physical memory space allocated on demand. Image quality, so the purpose of saving both bandwidth and frame storage without losing image quality is achieved.
  • the physical page when the correspondence between the physical page and the virtual page is found by querying the page table, the physical page can be automatically supplemented to make the processing of the missing page transparent to the media frame, so that the technical solution of the embodiment of the present invention can be applied to any Formal media framework is conducive to improving applicability and user experience.
  • a plurality of compressed slices, a plurality of compressed information, and a header address information form a data unit to the frame buffer in a storage process.
  • an image frame forms a data unit for data access in units of CTU.
  • the image frame is divided into n slices of the same data length.
  • Each data unit includes, for example, 3 slices.
  • each data unit also includes a header address information.
  • the header address information indicates the start address of each data unit, and the compression information indicates the compression mode / compressed length of each slice. As shown in FIG.
  • the data unit 1 in the image frame may include header address information 1, compression information of slice 1, compression information of slice 2, compression information of slice 3, and slice 1, slice 2, and slice 3. .
  • the physical memory space is dynamically allocated according to the amount of data that needs to be stored in the frame buffer, and each compressed data unit is compactly stored in the frame buffer in order to achieve the collection of the occupied memory space and achieve The mapping relationship between each compressed data unit and the storage address is established. In this way, random access of each data unit can be achieved in accordance with the length of each original slice.
  • the unused memory space in the frame buffer can be released
  • the method further includes: The buffers respectively read the n compressed slices; decompress the n compressed slices to obtain a decoded frame of a first image frame; and further use the decoded frame as a reference frame to further decode the code stream. Decode to obtain a third image frame.
  • the decoded frame is the first image frame; if the compression process uses other compression methods (such as lossy compression), and this compression method causes some If the image is lost, the decoded frame is the first image frame that has lost some information.
  • an embodiment of the present invention provides a memory management device.
  • the device includes a decoding module, a memory management module, a compression module, and a decompression module.
  • the decoding module is configured to decode a code stream to obtain a first An image frame, the first image frame includes n slices, each slice represents an image of a region in the first image frame, n is an integer greater than or equal to 2;
  • the compression module is configured to The first image frame is compressed to obtain a second image frame, where the second image frame includes n compressed slices; and the memory management module is configured to allocate frames in the memory for the n compressed slices.
  • a buffer is further configured to establish a mapping relationship between a virtual page and a physical page for the memory space occupied by the n compressed slices in the frame buffer, where each compressed slice occupies
  • the memory space includes one or more physical pages, and the page numbers of the multiple virtual pages occupied by the n compressed slices under the mapping relationship are consecutive; and are also used to separately divide the n compressed pages Slices stored Frame buffer.
  • an embodiment of the present invention provides another apparatus for memory management.
  • the apparatus includes a processor and a memory, where the processor is configured to decode a code stream to obtain a first image frame.
  • the image frame includes n tiles, each of which represents an image of a region in the first image frame, where n is an integer greater than or equal to 2; compressing the first image frame to obtain a second image frame, the The second image frame includes n compressed slices; a frame buffer is allocated in the memory for the n compressed slices; and is further used for the n compressed slices in the frame
  • the memory space occupied in the buffer establishes a mapping relationship between virtual pages and physical pages, where the memory space occupied by each compressed slice includes one or more physical pages, and the n
  • the page numbers of multiple virtual pages occupied by the compressed slice are consecutive; and are also used to store the n compressed slices to the frame buffer, respectively.
  • processor and the memory are specifically used to implement the method described in the first aspect, and for the implementation process of the related implementation manner, reference may also be made to the description of the first aspect, and details are not described herein.
  • an embodiment of the present invention provides another non-volatile computer-readable storage medium; the computer-readable storage medium is configured to store implementation code of the method described in the second aspect.
  • the program code is executed by a computing device, the user equipment is used in the method according to the first aspect.
  • an embodiment of the present invention provides a computer program product.
  • the computer program product includes a program instruction.
  • the controller executes the method described in the first aspect.
  • the computer program product may be a software installation package.
  • the computer program product may be downloaded and executed on a controller to Implement the method described in the first aspect.
  • the embodiment of the present invention after the image frame is compressed (such as lossless compression), it can be stored in a continuous memory area of the virtual address in the frame buffer, which realizes the collection of the occupied memory area. Therefore, the embodiment of the present invention is implemented It can achieve the purpose of effectively reducing the memory occupied by reference frame access during decoding.
  • using the memory reclamation scheme provided by the embodiment of the present invention can also recover the unused memory of the frame buffer, and merge it into the next frame for decoding and use, to effectively use the fragmented memory.
  • the on-demand allocation scheme can improve the page table establishment mechanism, realize the "on-demand occupation" of memory, and greatly improve the efficiency of memory usage.
  • FIG. 1 is a schematic block diagram of a video encoding and decoding system according to an embodiment of the present invention
  • FIG. 2 is a schematic block diagram of a video encoding and decoding device according to an embodiment of the present invention
  • FIG. 3 is a schematic block diagram of a destination device in a video encoding and decoding system according to an embodiment of the present invention
  • FIG. 4 is a schematic diagram of dividing an image frame into multiple slices according to an embodiment of the present invention.
  • FIG. 6 is a schematic diagram of a random access implementation scheme provided by an embodiment of the present invention.
  • FIG. 7 is a schematic diagram of a data format of a data unit according to an embodiment of the present invention.
  • FIG. 8 is a schematic diagram of an application scenario of memory allocation according to an embodiment of the present invention.
  • FIG. 9 is a schematic diagram of an application scenario for establishing a mapping relationship between a virtual page and a physical page according to an embodiment of the present invention.
  • FIG. 10 is a schematic diagram of another application scenario of memory allocation according to an embodiment of the present invention.
  • FIG. 11 is a schematic diagram of another application scenario of performing page fault processing through an SMMU according to an embodiment of the present invention.
  • FIG. 12 is a schematic flowchart of a memory management method according to an embodiment of the present invention.
  • FIG. 13 is a schematic flowchart of another memory management method according to an embodiment of the present invention.
  • FIG. 14 is a schematic flowchart of another memory management method according to an embodiment of the present invention.
  • FIG. 1 is a schematic block diagram of a video encoding and decoding system according to an embodiment of the present invention.
  • the video codec system includes a source device 10a and a destination device 10b.
  • the source device 10a may generate encoded video data and send a code stream to the destination device 10b. Therefore, the source device 10a may be referred to as a video encoding device or a video encoding device.
  • the destination device 10b can decode the encoded video data generated by the source device 10a and perform video display, and therefore, the destination device 10b can be referred to as a video decoding device or a video decoding device.
  • the source device 10a and the destination device 10b may be examples of a video codec device or a video codec device.
  • Source device 10a and destination device 10b can include a wide range of devices, including desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, handhelds such as smartphones, televisions, cameras, and display devices , Digital media player, video game console, car computer, or the like.
  • the source device 10a can access the storage medium via a disk access or card access method.
  • the storage medium may include a variety of locally accessible data storage media, such as DDR memory, Blu-ray disc, DVD, CD-ROM, flash memory, and so on.
  • the destination device 10b can receive the encoded video data from the source device 10a via the channel 17.
  • Channel 17 may include one or more media and / or devices capable of moving the encoded video data from the source device 10a to the destination device 10b.
  • the channel 17 may include one or more communication media enabling the source device 10a to directly transmit the encoded video data to the destination device 10b in real time.
  • the source device 10a may modulate the encoded video data according to a communication standard (eg, a wireless communication protocol), and may transmit the modulated video data to the destination device 10b.
  • the one or more communication media may include wireless and / or wired communication media, such as a radio frequency (RF) spectrum or one or more physical transmission lines.
  • RF radio frequency
  • the one or more communication media may form part of a packet-based network (eg, a local area network, a wide area network, or a global network (eg, the Internet)).
  • the one or more communication media may include a router, a switch, a base station, or other devices that facilitate communication from the source device 10a to the destination device 10b.
  • the channel 17 may include a storage medium that stores the encoded video data generated by the source device 10a.
  • the destination device 10b can access the storage medium via disk access or card access.
  • the storage medium may include a variety of locally accessible data storage media, such as DDR memory, Blu-ray disc, DVD, CD-ROM, flash memory, and so on.
  • the channel 17 may include a file server or another intermediate storage device that stores the encoded video data generated by the source device 10a.
  • the destination device 10b may access the encoded video data stored at a file server or other intermediate storage device via streaming or downloading.
  • the file server may be a server type capable of storing encoded video data and transmitting the encoded video data to the destination device 10b.
  • Example file servers include a web server (eg, for a website), a file transfer protocol (FTP) server, a network attached storage (NAS) device, and a local disk drive.
  • the destination device 10b can access the encoded video data via a standard data connection (eg, an Internet connection).
  • Examples of data connection types include wireless channels (e.g., Wi-Fi connection), wired connections (e.g., DSL, cable modem, etc.), or both, suitable for accessing encoded video data stored on a file server. combination.
  • the transmission of the encoded video data from the file server may be a streaming transmission, a download transmission, or a combination of the two.
  • the technology of the embodiments of the present invention is not limited to wireless application scenarios.
  • the technology can be applied to video codecs that support multiple multimedia applications such as: air television broadcast, cable television transmission, satellite television transmission, streaming Transport video transmissions (eg, via the Internet), encoding of video data stored on a data storage medium, decoding of video data stored on a data storage medium, or other applications.
  • the video codec system may be configured to support one-way or two-way video transmission to support applications such as video streaming, video playback, video broadcasting, and / or video telephony.
  • the source device 10 a includes a video data source 11, a video encoder 12, and a DDR memory 13.
  • the source device 10 a further includes an output interface (not shown in FIG. 1).
  • the output interface may include a modulator / demodulator (modem) and / or a transmitter for transmitting the encoded code stream through the channel 17. Send to the destination device 10b.
  • modem modulator / demodulator
  • the video data source 11 may include a video capture device (for example, a video camera), a video archive containing previously captured video data, a video input interface for receiving video data from a video content provider, and / or for generating video.
  • a video capture device for example, a video camera
  • a video archive containing previously captured video data
  • a video input interface for receiving video data from a video content provider
  • / or for generating video for example, a computer graphics system of data, or a combination of the aforementioned video data sources.
  • the video encoder 12 can encode video data from a video data source 11, and the video data includes multiple image frames.
  • the video encoder 12 may store the encoded video data (for example, reference frames) in the DDR memory 13 after compression. Accordingly, the video encoder 12 may also decompress the reference frames in the DDR memory 13, Use this reference frame to continue encoding other image frames.
  • the destination device 10 b includes a video decoder 14, a video display 15, and a DDR memory 16.
  • the destination device 10b may further include an input interface (not shown), which may include a receiver and / or a modem.
  • the input interface can receive a code stream via channel 17, the code stream including the encoded video data.
  • the video decoder 14 can decode the encoded video data in the bitstream to obtain the decoded video data.
  • the video decoder 14 may store the decoded video data in the DDR memory 16 after being compressed. Accordingly, the video decoder 14 may also decompress the video data in the DDR memory 16 to facilitate decompression.
  • the obtained image frame is used as a reference frame for the video decoder 14 to continue to decode subsequent code streams.
  • the video data in the DDR memory 16 can also be decompressed, and the video data obtained after decompression can be input to the video display 15.
  • the video display 15 may be integrated with the destination device 10b, or the video display 15 may be independently deployed outside the destination device 10b. Generally, the video display 15 can perform video display according to the input video data.
  • the video display 15 may include various display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or other types of display devices.
  • the memory management method described in the embodiment of the present invention is mainly applied to a data compression / decompression process in the corresponding codec of the source device 10a and / or the destination device 10b.
  • FIG. 2 is a schematic block diagram of a video codec device 50 according to an embodiment of the present invention.
  • the video codec device 50 may be used to implement functions of a source device 10 a and / or a destination device 10 b.
  • the apparatus 50 may be, for example, a mobile terminal of a wireless communication system or a user equipment of a wired communication system. It should be understood that embodiments of the present invention may be implemented in any electronic device or device that may need to encode and decode a video image, or encode, or decode.
  • the device 50 may include a processor 56 (or controller), a memory 58, and a codec 54 for controlling the device 50, corresponding to the video encoder 12 or the destination of the source device 10 a in FIG. 1.
  • the processor 56 may be used to run an operating system (OS), and the processor 56 may be connected to a memory 58 such as a DDR memory (or DDR memory module), a Blu-ray disc, a DVD, a CD-ROM, a flash memory, and the like.
  • OS operating system
  • the memory 58 may store data of a compressed image (for example, an image frame after lossless compression), and / or may also store instructions for implementing on the processor 56.
  • the memory 58 corresponds to the DDR memory 13 of the source device 10a or the DDR memory 16 of the destination device 10b in FIG.
  • the processor 56 may be connected to a codec 54 adapted to implement encoding and decoding of video data or auxiliary encoding and decoding controlled by the processor 56.
  • the device 50 may also include a display 32 in the form of a liquid crystal display.
  • the display 32 may be any suitable display technology suitable for displaying images or video, corresponding to the video display 15 of the destination device 10b.
  • the device 50 may further include a keypad 34.
  • any suitable data or user interface mechanism may be used.
  • the user interface may be implemented as a virtual keyboard or a data entry system as part of a touch-sensitive display.
  • the device may include a microphone 36 or any suitable audio input, which may be a digital or analog signal input.
  • the apparatus 50 may further include an audio output device, which in the embodiment of the present invention may be any one of the following: a headset 38, a speaker, or an analog audio or digital audio output connection.
  • the device may also include an infrared port 42 for short-range sight communication with other devices.
  • the device 50 may also include any suitable short-range communication solution, such as a Bluetooth wireless connection or a USB / Firewire wired connection.
  • the device 50 may further include a card reader 48 and a smart card 46, such as UICC and UICC readers, for providing user information and adapted to provide authentication information for authenticating and authorizing users on the network.
  • a card reader 48 and a smart card 46 such as UICC and UICC readers, for providing user information and adapted to provide authentication information for authenticating and authorizing users on the network.
  • the apparatus 50 may further include a radio interface 52 connected to the processor 56 and adapted to generate, for example, wireless communication signals for communication with a cellular communication network, a wireless communication system, or a wireless local area network.
  • the device 50 may further include an antenna 44 connected to the radio interface 52 for transmitting radio frequency signals generated at the radio interface 52 to other devices (s) and for receiving radio frequency signals from other devices (s).
  • the device 50 may receive video data for encoding / decoding through a wireless or wired connection.
  • the memory management method described in the embodiment of the present invention is mainly applied to a data compression / decompression process in a corresponding codec of the device 50.
  • the following mainly describes the memory management scheme provided by the embodiment of the present invention from the perspective of the decoding side.
  • the memory management scheme provided by the embodiment of the present invention can also be applied to the encoding side, that is, the memory management of the encoding side
  • the solution can be implemented similarly by referring to the implementation scheme on the decoding side. This is because the decoding scheme is sometimes involved on the encoding side. Therefore, the scheme applicable to the decoding side mentioned in this embodiment can also be used, so it will not be repeated here. .
  • the destination device includes a video decoder 20, a video display 30, and a memory (including a frame buffer 22, a frame buffer Also called frame memory), such as DDR memory.
  • the video decoder 20 may be the codec 54 shown in the embodiment of FIG. 2 or include the codec 54 or a part thereof.
  • the video decoder 20 includes the decoder 21 and the memory management module 23.
  • the video decoder 20 may further include a compression module 24 and a decompression module 25.
  • the compression module 24 and the decompression module 25 may be implemented as function modules of a decoder; in yet another possible embodiment, the compression Module 24 and decompression module 25 can also be implemented independently of the decoder.
  • the frame buffer 22 is used to store video data, and can specifically store image frames compressed by the compression module 24 using a lossless compression method.
  • the memory of the destination device may be implemented with DDR, and the frame buffer 22 may be located in the DDR memory, that is, the frame buffer 22 indicates a storage area allocated for image storage in the memory.
  • the frame buffer 22 may be a direct image of a picture displayed by a video display.
  • Each storage unit of the frame buffer corresponds to one pixel on the screen, and the entire frame buffer corresponds to one frame of image.
  • the over-compression module 24 and the de-compression module 25 in this embodiment may use a lossless compression technology and a lossy compression technology, and achieve more technical effects when using the lossless compression technology.
  • This embodiment only uses the lossless compression technology as an example for description, but is not used to limit the application of the embodiment solution in the scenario of the lossy compression technology.
  • the memory management module 23 can allocate memory space (virtual memory space / physical memory space) to the frame buffer 22 through dynamic memory allocation, so as to realize effective collection and use of memory space or reclaim free memory space, thereby saving memory consumption.
  • the decoder 21 decodes an input code stream and outputs an image frame.
  • the image frames need to be compressed by the compression module 24 and then stored in the frame buffer 22 configured in the memory management module 23.
  • the image frame stored in the frame buffer 22 is used as a reference frame.
  • the reference frame is decompressed by the decompression module 25 and then applied to the decoder 21 for decoding reference.
  • the image frames stored in the frame buffer 22 are decompressed after being decompressed by the decompression module 31 and then applied to the video display 30 for video display.
  • the compression process and the decompression process correspond to each other, that is, when the compression module uses a lossless compression method to compress the image, the decompression module 25 or the decompression module 31 correspondingly The image after lossless compression is decompressed to restore the original image without causing image distortion.
  • the image frame may be divided into n identical rectangular regions along the horizontal and vertical directions, and each rectangular region becomes a tile. In this way, image data can be accessed in slices.
  • image frames can also be divided in multiple ways. For example, in H.265, image frames can be divided into multiple equal-sized tree coding units (CTUs).
  • CTU is the basic H.265 processing unit.
  • Each CTU The size can be, for example, 16X16 or 32X32 or 64X64.
  • each CTU may include multiple slices.
  • an image frame may include 8 CTUs, and at the same time, the image frame may be divided into 24 slices. In this way, each CTU may include 3 slices. . It should be noted that this example is only for explanation, not limitation.
  • memory can be divided into virtual memory space and physical memory space.
  • the physical memory space is directly related to the physical hardware of the memory module, and indicates the size of the memory space that is directly addressed according to the address line of the memory module.
  • OS operating system
  • the required memory space may be larger than the total capacity of the memory module.
  • the total capacity of the memory module is 256M, and the program
  • it is necessary to create a 2G data area so all data cannot be loaded into the physical memory of the memory module together, and some data must be placed in other media (such as hard disks), that is, the operating system will use the
  • the other media space is used as a virtual memory space, and data temporarily unnecessary by the current process is placed on other media.
  • the process needs to access that part of the data, it enters the physical memory space through scheduling.
  • the processes of the OS or other software are controlled by virtual memory that can be mapped to physical memory in order to manage the fragmented data in physical memory.
  • the virtual memory space can be understood as the sum of all the memory spaces (such as 2G) when the process of the OS or other software is running (that is, the memory space seen from the perspective of the process), and There may be a part not in the physical memory space (such as 256M), but in other media. Therefore, a mapping relationship needs to be established between the virtual memory and the physical memory, so that the software process can manage and control the operation of the virtual memory corresponding to the physical memory.
  • the operating system paging the virtual memory address space to generate a virtual page (page), and paging the physical memory address space to generate a physical page (or physical page frame, page frame).
  • the virtual page contains the corresponding
  • the physical page contains the corresponding physical address.
  • the space size of a virtual page and a physical page is the same. Therefore, the total number of virtual pages is greater than the total number of physical pages.
  • a system memory management unit (SMMU) is set.
  • the SMMU can use the processor to manage the virtual and physical memory control lines, map virtual addresses to physical addresses, and provide hardware-based memory. Access authorization and more.
  • the SMMU maintains a page table.
  • the page table can be used to record the mapping relationship between virtual pages and physical pages. More specifically, it records one between the page number of the virtual page and the page number of the physical page. A mapping relationship.
  • the image frame data in order to ensure that the memory of the image frame data does not swell after being stored in the frame buffer of the memory and is convenient for addressing, the image frame data needs to be able to be randomly accessed in the frame buffer.
  • the following first compares the differences between the prior art and the embodiments of the present invention in the implementation of the random access scheme.
  • the system fixedly allocates a frame buffer for storing image frames.
  • the frame buffer is allocated, not only the virtual memory space is allocated for the frame buffer, but also the physical memory space is allocated for the frame buffer.
  • the virtual pages of the virtual memory space correspond exactly to the physical pages of the physical memory space.
  • the image frame is divided into n slices of the same data length.
  • this solution fixes the slice coordinates to the starting point of the slice by fixing the start address of each slice.
  • the initial storage address so that the static mapping relationship between each slice and its storage address can be directly established (as shown in the upper part of Figure 5, the gray area represents the data length of slice 1, slice 2 ... slice).
  • the memory management module 23 further provided in the embodiment of the present invention adopts a strategy of dynamically allocating frame memory to implement fine-grained memory allocation in the storage space allocation of the frame buffer 22 Or the collection of free memory to maximize efficiency.
  • the memory management module 23 may allocate a frame buffer (such as the white area in the upper half of FIG. 6) in advance according to the needs of the decoder 21.
  • the frame buffer is only a piece of virtual space.
  • an image frame is accessed for data in slices, and the image frame is divided into n slices of the same data length.
  • the physical memory space is dynamically allocated according to the amount of data that needs to be stored in the frame buffer, and each compressed (such as lossless compression) slice is compactly stored in the frame buffer in order to achieve the reduction of occupied memory space.
  • Set shown in gray area in the lower half of Figure 6
  • the starting storage address of each compressed slice is no longer fixed.
  • the embodiment of the present invention re-establishes the mapping relationship between each compressed slice and the storage address. Specifically, in addition to storing each compressed slice, the embodiment of the present invention also stores header address information and compression information corresponding to each slice.
  • the header address information indicates the start address and compression information of each compressed slice.
  • the compression mode / compressed length of each compressed slice is indicated, so that the random access of each slice can be achieved by matching the original length of each slice.
  • an image frame is formed with data units in units of CTU for data access, and the image frame is divided into n slices of the same data length, and each data unit includes, for example, 3 slices
  • each data unit also includes a header address information and compression information corresponding to each slice.
  • the header address information indicates the start address of each data unit
  • the compression information indicates the compression mode / compressed length of each slice.
  • the data unit 1 in the image frame may include header address information 1, compression information of slice 1, compression information of slice 2, compression information of slice 3, and slice 1, slice 2, and slice 3. .
  • the physical memory space is dynamically allocated according to the amount of data that needs to be stored in the frame buffer, and each compressed data unit is compactly stored in the frame buffer in order to achieve the collection of the occupied memory space and achieve The mapping relationship between each compressed data unit and the storage address is established. In this way, random access of each data unit can be achieved in accordance with the length of each original slice.
  • the memory management module 23 adopts a strategy of dynamically allocating frame memory to implement fine-grained memory allocation or memory reclamation.
  • the following two methods (1) and (2) are respectively the memory reclamation mode and the pre-allocation mode, so as to realize the full utilization of memory resources.
  • a memory reclamation mode is adopted.
  • the memory management module 23's allocation of video frame memory can be divided into the following two aspects:
  • a small block of memory (such as 1MB or 512KB) is allocated at a time, which is called this A small block of memory is called a memory block.
  • a memory block pool of a decoding channel can be formed.
  • the size of the memory block pool can correspond to the minimum frame memory overhead. For example, each frame buffer needs to allocate 15MB of memory space. For 6 image frames (that is, corresponding to the 6 frame buffers shown in Figure 8), 4K video
  • the memory in the memory block pool consists of multiple discrete memory blocks.
  • each memory block corresponds to multiple virtual pages
  • the page numbers between these virtual pages in the same memory block are continuous
  • each virtual page corresponds to a physical page
  • the physical pages corresponding to these virtual pages may be discrete (that is, the page numbers of these physical pages may be discontinuous).
  • a sufficient virtual memory space (that is, only a virtual address space is allocated) can be allocated in advance for each frame buffer through the memory management module 23, and such a virtual memory space can also be called a container.
  • a container For example, if each frame buffer needs to allocate 15MB of memory space, then a 15MB container needs to be allocated for each frame buffer.
  • the "image frame" seen by the playback frame still has the previously fixed virtual address and data length, so that the technical solution of the embodiment of the present invention can be compatible with various existing playback frames.
  • the container is allocated, the relationship between the virtual address and the physical address in the frame buffer is indeterminate. This correspondence is only established when the image frame needs to be actually stored in memory.
  • the memory management module 23 when an image frame needs to be actually stored in the memory, the memory management module 23 establishes a mapping relationship between a virtual address and a physical address (that is, a mapping relationship between a container and a physical memory space).
  • each frame buffer (that is, a container with a virtual address and no physical memory space) is allocated with an actual physical memory space. Because the compression rate of lossless compression is uncertain, in a possible embodiment of the present invention, before the decoder decodes each image frame, the corresponding containers are filled with the actual physical memory space, that is, the decoding starts Before, the mapping relationship between the virtual address and the physical address of the frame buffer was established.
  • the implementation process is as follows: Find the unoccupied memory blocks in the memory block pool. These memory blocks correspond to a page table. The page table records The mapping relationship between each virtual page and the physical page in the corresponding memory block.
  • a sufficient amount of memory blocks are combined into a storage space of the size indicated by the container.
  • the virtual memory space indicated by a container is 15MB
  • several memory blocks with a total of 15MB are selected to be combined into the physical memory space required by the container.
  • the page tables corresponding to the several memory blocks with a total of 15MB can be combined into a page table of a frame buffer, so that a mapping relationship between virtual addresses and physical addresses in the frame buffer is actually established. This ensures that the frame buffer has true 15M physical memory.
  • the compressed data units of the image frame are compactly arranged in the frame buffer, and the unoccupied memory blocks in the frame buffer are removed according to the actual memory occupancy reported by the decoder, and recovered again. Memory block pool.
  • M memory blocks (memory block 1, memory block 2, ... memory block M) are found to be combined into the actual storage space, and each memory block is With corresponding page tables, these page tables are copied together to form a page table of frame buffer 0.
  • decoding mark the excess memory blocks as "free” according to the memory footprint reported by the decoder, release these free memory blocks and recycle them to the memory block pool, and the memory blocks in these memory block pools can continue to be used in subsequent Frame buffer (container), thus realizing the dynamic utilization of memory space.
  • the above process may include: the OS provides a function of allocating a frame buffer, encapsulated as "VFB_Alloc ()", and the returned virtual space does not correspond to a physical page table.
  • the memory management module manages the physical memory budget by itself. For example, a 4K decoding channel requires 90MB of memory (including 6 frame buffers), and then uses the "MMB_New ()" instruction to allocate multiple blocks with a smaller granularity (such as 1MB or 512KB). 90MB.
  • the decoder Before the decoding is started, the decoder performs the correspondence between the virtual address and the physical address of the frame buffer, finds free memory blocks, and attaches the page table corresponding to each memory block to the frame buffer page table to ensure that the current The frame buffer is true 15M physical memory to avoid page faults.
  • the memory management module removes the unoccupied memory blocks and reclaims them to the memory block pool for subsequent use according to the current memory usage of the frame buffer reported by the decoder.
  • the compact memory can be stored in the allocated frame buffer after lossless compression of image frames, and the physical memory space is refined to realize the centralized utilization of fragmented memory.
  • the decoder After decoding an image frame and storing it in the frame buffer, the memory management module can recycle the unused memory of the frame buffer, release the unoccupied free memory, and merge it into the decoding of the next frame. In this way, you can use a smaller memory budget and actually obtain a larger number of frame buffers, thereby ensuring that various performance indicators are not damaged, and achieving both the saving of bandwidth and the saving of frame memory without losing image quality. purpose.
  • the frame buffer that the memory management module 23 can allocate for image frames before decoding includes only virtual memory space, not physical memory.
  • the virtual memory space includes multiple virtual pages with consecutive page numbers.
  • the page table created when the frame buffer was initially allocated indicates the virtual page, but not the mapping relationship between the virtual page and the physical page.
  • this on-demand allocation scheme does not involve memory reclamation. Instead, the virtual memory space is allocated before the actual physical memory is allocated. For example, when decoding, 15 frame buffers need to be allocated, and each frame buffer occupies 15 MB of space, so a full 15 MB ⁇ 15 space is allocated in the virtual memory space.
  • each virtual page corresponding to it may not have a physical page mapped to it.
  • the actual physical page is allocated according to the actual demand of the data unit by way of page fault processing.
  • the process of establishing the mapping relationship between the virtual page and the physical page of the frame buffer is as follows: When it is necessary to store the compressed data unit into the frame buffer, according to one or more virtual pages corresponding to the data unit A page table query for the page number of the page, if the page number of the physical page corresponding to the page number of the one or more virtual pages cannot be queried, then the unoccupied one or more physical pages are found in the memory, and all The page number of the one or more physical pages is added to the page table so that each of the page numbers of the one or more virtual pages in the page table corresponds to a page number of a physical page, respectively. In this way, the mapping relationship between the virtual page and the physical page of the frame buffer is established. After that, the compressed data unit can be stored to the corresponding physical page.
  • the memory management module 23 may query a bitmap, and the bitmap includes multiple Each bit is used to indicate whether a physical page in memory is occupied, and then one or more physical pages that are not occupied are found according to the bitmap. As shown in FIG. 10, each bit in the bitmap indicates the occupation status of a physical page, where "0" represents that the corresponding physical page is not occupied, and "1" represents that the corresponding physical page has been occupied.
  • the compressed data unit is stored in the frame buffer 0, when you need to find 4 unused physical pages, you can find 4 physical pages indicated by 4 bits whose value is "0" according to the bitmap. .
  • "finding a bit with a value of 0" can be implemented in various ways. For example, if the number of physical pages is small, you can directly use the sequential query method to find the bits with a value of "0"; if the number of physical pages is very large, that is, the number of bits in the corresponding bitmap is huge, with 128MB of physical memory Space as an example means 32768 physical pages, so the bitmap has 32768 bits. In this case, you can use the "hierarchical labeling" method to find the "0" bit. For example, 32768 bits are divided into 256 segments, and each segment contains 128 One digit, these 128 digits form a status identifier through "multiplication".
  • the memory management module 23 may include a system memory management unit (SMMU) to implement page fault processing in the actual storage space allocation process of the frame buffer.
  • SMMU system memory management unit
  • the SMMU may include a Translation Detection Buffer (TLB) module, a page fault processing module, and an arbitration module.
  • TLB Translation Detection Buffer
  • the TLB module can load the TLB in advance and dynamically maintain the page table entries in the page table.
  • the memory management module 23 can store one or more virtual data corresponding to the data unit.
  • the virtual address of the page is sent to the TLB module through the AXI (Advanced Extensible Interface) bus.
  • AXI Advanced Extensible Interface
  • the TLB module converts the virtual address into the page number of the virtual page for page table query. If the page number cannot be queried with the page number of the virtual page, The page number of the corresponding physical page sends a page fault request to the page fault processing module.
  • the page fault processing module obtains the correspondence between the bitmap and the physical page list through the arbitration module and the AXI bus, and then uses the bitmap to find the unoccupied one or more physical pages, and obtains the physics of the one or more physical pages.
  • the address is further added to the page table of the TLB module according to the page number corresponding to the physical address, so that each of the page numbers of the one or more virtual pages in the page table corresponds to the page number of a physical page, respectively, and will be updated
  • the subsequent page table is filled into the TLB module to complete the supplementary page, and a physical page is allocated to the data unit, and the compressed data unit can be stored to the corresponding physical page subsequently.
  • the embodiment of the present invention improves the page table establishment mechanism through the memory management module 23, and realizes "on-demand" memory. After the image frames are losslessly compressed, they are stored in the on-demand physical memory space through compact memory arrangement. , To achieve the purpose of saving both bandwidth and frame storage without losing image quality.
  • the memory management module 23 can automatically supplement physical pages when a page is missing through the SMMU, so that the process of missing pages is transparent to the media frame, so that the technical solution of the embodiment of the present invention can be applied to any form of media frame, which is helpful to improve the applicability and user experience.
  • FIG. 12 is a schematic flowchart of a memory management method according to an embodiment of the present invention.
  • the method can be applied to the video decoder described in the embodiment of FIG. 3.
  • the video decoder includes a decoder and a compression module.
  • the decompression module, the memory management module, the compression module and the decompression module can be implemented as function modules of the decoder, or the compression module and the decompression module can also be implemented independently of the decoder.
  • the method includes but is not limited to the following steps:
  • Step S101 the decoder decodes the code stream to obtain a first image frame, where the first image frame includes n tiles of the same size, each of which represents an image of a rectangular region in the first image frame , N ⁇ 2.
  • Step S102 The compression module compresses the first image frame (such as lossless compression) to obtain a second image frame, where the second image frame includes n compressed slices. It is understandable that due to the image data of each slice Each has a difference, so after each slice undergoes lossless compression, the compression ratio also has a difference, and the data length of each compressed slice formed also has a difference.
  • Step S103 The memory management module stores the n compressed slices in a frame buffer of the memory, and the n compressed slices correspond to physical pages occupied by the frame buffer. Virtual pages have consecutive page numbers.
  • the frame buffer of the memory is configured and managed by a memory management module.
  • the memory management module allocates a frame buffer for the n compressed slices in the memory, and establishes a virtual page for the memory space occupied by the n compressed slices in the frame buffer. Mapping relationship with physical pages, wherein the memory space occupied by each compressed slice includes one or more physical pages, and a plurality of virtual pages occupied by the n compressed slices under the mapping relationship The page numbers of the pages are continuous. After that, the memory management module stores the n compressed slices into the frame buffer, respectively.
  • the video decoder can compactly store the compressed (such as lossless compression) slices into the frame buffer in order to achieve the collection of the occupied memory space.
  • the physical pages occupied by these compressed slices The page numbers can be discrete, but the page numbers of the virtual pages occupied by these compressed slices are continuous.
  • the embodiment of the present invention may also re-establish the mapping relationship between each compressed slice and the storage address.
  • the embodiment of the present invention also stores header address information and compression information corresponding to each slice.
  • the header address information indicates the start address and compression information of each compressed slice.
  • the compression mode / compressed length of each compressed slice is indicated, so that the random access of each slice can be achieved by matching the original length of each slice.
  • the image frame may also perform data access according to the form of a data unit
  • the data unit may have various implementation forms in different application scenarios, for example, the data unit is a tree coding unit (Coding, Tree Unit, CTU), Coding Tree Block (CTB), Coding Block (CB), Coding Unit (CU), Prediction Block (PB), macro block, etc.
  • the n compressed slices may be divided into multiple groups, each group is used to form a data unit, each group includes a different compressed slice, and each of the The compressed slices each correspond to one piece of compression information, and the compressed information indicates the storage space occupied by the corresponding compressed slice; each group corresponds to one header address information.
  • each of the plurality of groups, the corresponding header address information of the group, and the compression information corresponding to the group constitute a data unit, and the memory management module stores the multiple data units thus constituted respectively.
  • the plurality of data units have consecutive page numbers in virtual pages corresponding to the physical pages occupied by the frame buffer.
  • the memory management module further includes: The buffers respectively read the n compressed slices, and a decompression module decompresses the n compressed slices to obtain a decoded frame of the first image frame, and then, the decoder uses the decoded frame as a reference The frame further decodes the code stream to obtain a third image frame.
  • the decoded frame is the first image frame; if the compression process of the above step S102 is another compression method (if there is lossy compression), And this compression method causes part of the image loss, then the decoded frame is the first image frame that lost part of the information.
  • the embodiment of the present invention can realize that after image frames are losslessly compressed, they are stored in a continuous memory area of virtual addresses in the frame buffer, and the collection of occupied memory areas is achieved. Therefore, the implementation of the embodiment of the present invention can achieve The purpose of saving both bandwidth and frame memory under the premise of loss of image quality.
  • FIG. 13 is a schematic flowchart of another memory management method according to an embodiment of the present invention.
  • the method can be applied to the video decoder described in the embodiment of FIG. 3.
  • the video decoder includes a decoder. , Compression module, decompression module, memory management module.
  • This method uses a memory reclamation mode, including but not limited to the following steps:
  • Step S201 The decoder decodes the code stream to obtain a first image frame, where the first image frame includes n pieces of the same size, and each piece represents an image of a rectangular area in the first image frame, n ⁇ 2.
  • Step S202 The compression module performs lossless compression on the first image frame to obtain a second image frame, where the second image frame includes n compressed slices.
  • Step S203 The memory management module allocates multiple unused fine-grained memory blocks (for example, 1MB or 512KB) in the memory.
  • Each fine-grained memory block includes multiple virtual pages with consecutive page numbers.
  • a page maps a physical page. That is, the page numbers between these virtual pages in the same memory block are continuous, and each virtual page corresponds to a physical page, but the physical pages corresponding to these virtual pages may be discrete.
  • Step S204 The memory management module combines the plurality of fine-grained memory blocks into the frame buffer, and a sum of physical memory spaces corresponding to the plurality of memory blocks is M, and 0 ⁇ M ⁇ the first image frame The amount of data.
  • each frame buffer needs to allocate 15MB of memory space, several memory blocks can be combined into a 15MB frame buffer for insurance purposes.
  • the current lossless compression can be estimated in advance, Compression ratio, then you can also appropriately reduce the memory space allocated according to the estimated compression ratio. For example, it is estimated that the compression rate is not less than 20%, then several memory blocks can be combined into a 12MB frame buffer.
  • a page table corresponding to each memory block may be determined, and a page table corresponding to each memory block is respectively used to indicate a mapping relationship between each virtual page and a physical page in the memory block, and then each memory block is correspondingly
  • the page tables are combined into a page table of the frame buffer.
  • Step S205 The memory management module stores the n compressed slices in a frame buffer, respectively, and the n compressed slices have continuous virtual pages corresponding to the physical pages occupied by the frame buffer. Page number, to realize the continuity of the virtual memory page number, which is convenient for the software to manage and maintain. For details, reference may also be made to the description of step S103 in the embodiment of FIG. 11, and details are not described herein again.
  • Step S206 The memory management module releases a memory block in the frame buffer that is not occupied by the multiple data units. Specifically, after decoding, the memory management module marks the excess memory blocks as "free" according to the reported memory occupancy, releases these free memory blocks for recycling, and these recovered memory blocks can continue to be applied to subsequent frame buffers. , So as to achieve the dynamic use of memory space. For example, if 15MB of physical memory space is allocated for the frame buffer, but after lossless compression compactly stores image frames in the frame buffer in turn, the actual memory space occupied is 10MB, then the unoccupied 5MB of physical memory space can be released Corresponding memory block, apply this 5MB physical memory space recycling to the next frame buffer.
  • the compact memory can be stored in the allocated frame buffer after lossless compression of image frames, and the physical memory space is refined to realize the centralized utilization of fragmented memory.
  • the decoder After decoding an image frame and storing it in the frame buffer, the memory management module can recycle the unused memory of the frame buffer, release the unoccupied free memory, and merge it into the decoding of the next frame. In this way, you can use a smaller memory budget and actually obtain a larger number of frame buffers, thereby ensuring that various performance indicators are not damaged, and achieving both the saving of bandwidth and the saving of frame memory without losing image quality. purpose.
  • FIG. 14 is a schematic flowchart of another memory management method according to an embodiment of the present invention.
  • the method can be applied to the video decoder described in the embodiment of FIG. 3.
  • the video decoder includes a decoder.
  • Compression module, decompression module, and memory management module the memory management module includes the SMMU.
  • This method uses a pre-allocation mode, including but not limited to the following steps:
  • Step S301 The decoder decodes the code stream to obtain a first image frame, where the first image frame includes n tiles of the same size, each of which represents an image of a rectangular region in the first image frame, n ⁇ 2.
  • Step S302 The compression module performs lossless compression on the first image frame to obtain a second image frame, where the second image frame includes n compressed slices.
  • Step S303 The memory management module allocates a virtual memory space for the first image frame in advance.
  • the pre-allocated virtual memory space includes a plurality of virtual pages with consecutive page numbers to implement the continuous virtual memory page numbers, which is convenient for software to manage and maintain.
  • the space size of the virtual memory space is K, and 0 ⁇ K ⁇ the data amount of the first image frame.
  • each frame buffer occupies 15MB space, and a full 15MB ⁇ 15 space is allocated in the virtual memory space. But for each of the 15 frame buffers, each virtual page corresponding to it may not have a physical page mapped to it.
  • the actual physical page is allocated according to the actual demand of the data unit by way of page fault processing.
  • Step S304 The memory management module sends the page number of one or more virtual pages corresponding to each compressed slice in the second image frame to the SMMU for page table query, and if the query is not available, The page fault processing is started when the page number of the page corresponds to the page number of the physical page.
  • Step S305 If the page number of the physical page corresponding to the page number of the one or more virtual pages cannot be queried, the memory management module allocates corresponding physical pages to the one or more virtual pages.
  • the process may be Including: obtaining a bitmap, the bitmap includes multiple bits, each bit is used to indicate whether a physical page in the memory is occupied, one or more unused physical pages can be found according to the bitmap, and Increasing the page number of the one or more physical pages to the page table, so that each of the page numbers of the one or more virtual pages in the page table corresponds to a page number of a physical page, The number of pages of the physical page is not less than the number of pages of the virtual page.
  • Step S306 The memory management module separately stores each compressed slice to the corresponding physical page.
  • Step S307 The memory management module releases virtual memory space in the frame buffer that is not occupied by the plurality of compressed slices. For example, if 15MB of virtual memory space is allocated for the frame buffer, but after lossless compression compactly stores image frames in the frame buffer in turn, the actual physical memory space occupied is 10MB, then the unoccupied 5MB of physical memory can be released.
  • the virtual page corresponding to the space for example, the page number of the virtual page corresponding to the 5MB physical memory space is deleted from the page table.
  • the embodiment of the present invention achieves "on-demand" of memory by improving the page table establishment mechanism.
  • the difference from the embodiment of Fig. 13 lies in that it does not require a recovery operation. It is stored in the physical memory space allocated on demand, which realizes the purpose of saving both bandwidth and frame storage without losing image quality.
  • the memory management module 23 can automatically supplement physical pages when a page is missing through the SMMU, so that the process of missing pages is transparent to the media frame, so that the technical solution of the embodiment of the present invention can be applied to any form of media frame, which is beneficial to improving applicability And user experience.
  • the embodiments of the present invention mainly use the application scenario of lossless compression as an example to describe the technical solution of the present invention.
  • the actual application of the embodiments of the present invention is not limited to the application of lossless compression.
  • Scenario, in other possible application scenarios (such as lossy compression) the technical solution of the embodiment of the present invention may also be applied to achieve the collective utilization of memory, thereby reducing the memory occupied by reference frame access during decoding, here No more detailed description.
  • the computer program product includes one or more computer instructions.
  • the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.
  • the computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium.
  • the computer instructions may be from a network site, computer, server, or data center.
  • the computer-readable storage medium may be any available medium that can be accessed by a computer, and may also be a data storage device such as a server, a data center, or the like that includes one or more available media integration.
  • the usable medium may be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape, etc.), an optical medium (such as a DVD, etc.), or a semiconductor medium (such as a solid state drive), and so on.
  • the codec 54 in FIG. 2 is an independent hardware device and is located outside the processor 56.
  • the codec 54 may exist in the form of software, that is, a computer program product as previously mentioned, and run on the processor 56.
  • the codec 54 is now a software device, including software code instructions.
  • the video decoder 20 may be implemented in software or hardware or a combination thereof.
  • any module or unit in the video decoder 20 may be implemented in software or hardware.
  • the decoder 21, the memory management module 23, the compression module 24, and the decompression module 25 are all hardware.
  • the decoder 21, the memory management module 23, the compression module 24, and the decompression module 25 are all software. At this time, they are all modules formed by software codes running on the processor 56.
  • a certain module when a certain module is implemented in hardware, it may include a large number of transistors, computing devices, or logic circuits, and may specifically exist in the form of an integrated circuit.
  • the frame buffer 22 in FIG. 3 is located in the memory 58 of FIG. 2.
  • the memory 58 may specifically include one or more memories.
  • the memory 58 includes a plurality of different types of memory, including volatile memory and non-volatile memory.
  • a non-volatile memory is used to store software code, is executed by the processor 56, and drives the work of the processor 56.
  • a volatile memory may be used to store data, such as temporarily storing data temporarily, forming the memory mentioned in the previous embodiment and including the frame buffer 22.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Multimedia (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)

Abstract

本申请提供了内存管理方法以及相关装置，该方法包括：对码流进行解码得到第一图像帧，第一图像帧包括n个片；对第一图像帧进行压缩得到第二图像帧，第二图像帧包括n个经压缩后的片；在内存中为n个经压缩后的片分配帧缓冲区，并为n个经压缩后的片所占用的内存空间建立虚拟页面与物理页面的映射关系，每个经压缩后的片所占用的内存空间包括一个或多个物理页面，n个经压缩后的片所占用的多个虚拟页面的页号连续；最后，将n个经压缩后的片存储到帧缓冲区。实施本申请方案能够有效降低解码时的参考帧存取对内存的占用。

Description

内存管理方法以及相关装置
本申请要求于2018年06月28日提交中国国家知识产权局、申请号为2018106947906申请名称为“内存管理方法以及相关装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本发明涉及视频处理领域，尤其涉及内存管理方法以及相关装置。
背景技术
双倍速率同步动态随机存储器(Double Data Rate SDRAM，DDR SDRAM)又称DDR内存，是视频处理系统的重要构成部分，通常情况下DDR内存与主芯片价格平齐甚至超过主芯片。所以，在相同业务体验的前提下减少DDR的使用，是降低视频处理系统成本的有效途径。
在媒体架构中，视频解码器(Video Decoder，VDEC)的帧缓冲区(frame buffer，也可称为帧存)占据了DDR内存的主要部分，例如将6～7帧的图像帧(如参考帧)存储在帧缓冲区。因此，为了降低视频处理中内存的占用，通常在参考帧存取过程中引入压缩技术，包括无损压缩和有损压缩这两种技术。
其中，有损压缩利用了人类对图像或声波中的某些频率成分不敏感的特性，在数据压缩过程中损失掉一定的信息，所以有损压缩通常能带来较大的图像压缩率，但是，有损压缩会引入图像的误差，由于解码的参考帧具有参考特性，误差经过参考帧迭代之后不断累积放大，最终有可能呈现视觉可觉察的图像损伤，造成图像质量(Picture Quality，PQ)的下降，所以，在解码环节的参考帧存取过程中通常不采用有损压缩技术。
无损压缩技术是更为常见的应用于参考帧存取的技术，其利用数据的统计冗余进行压缩，可通过在解压缩完全恢复原始数据而不引起图像失真。所以，目前在解码环节通常采用无损压缩技术。虽然无损压缩不会造成数据损伤，但是无损压缩的压缩率并不可靠，例如4K视频在无损压缩下也存在压缩率仅为20％甚至无法压缩的情形。所以相对而言，现有的无损压缩技术不能有效节省内存占用。因此，如何有效降低解码时的参考帧存取对内存的占用就成为一个问题。
发明内容
本发明实施例提供了内存管理方法以及相关装置，能够有效降低解码时的参考帧存取对内存的占用。
第一方面，本发明实施例提供了一种视频解码中的内存管理方法，该方法包括：视频解码器对码流进行解码得到第一图像帧，所述第一图像帧包括n个片tile(比如n个相同的片)，每个片表示所述第一图像帧中的一个矩形区域的图像，n是大于等于2的整数；对所述第一图像帧进行压缩(例如无损压缩)得到第二图像帧，所述第二图像帧包括n个经压缩后的片；在内存中为所述n个经压缩后的片分配帧缓冲区；为所述n个经压缩后的片在 所述帧缓冲区中所占用的内存空间建立虚拟页面与物理页面的映射关系，其中，每个经压缩后的片所占用的内存空间包括一个或多个物理页面，在所述映射关系下所述n个经压缩后的片所占用的多个虚拟页面的页号连续；分别将所述n个经压缩后的片存储到所述帧缓冲区。
可以看到，本发明实施例中视频解码器可将各个经过压缩(例如无损压缩)后的片依次紧凑地存入帧缓冲区中虚拟地址连续的内存区域，实现所占用内存空间的归集，这些经过压缩后的片所占用的物理页面的页号可以是离散的，但是这些经过压缩后的片所占用的虚拟页面的页号是连续的。所以，实施本发明实施例能够有效降低解码时的参考帧存取对内存的占用目的。
基于第一方面，在本发明可能的实施方式中，为了实现内存资源充分利用，设计了以下两种方式来实现动态分配帧存，实现精细粒度的内存分配或内存回收，这两种方式分别为内存回收模式和预分配模式(或称按需分配模式)，下面分别描述这两种方式。
(1)在采用内存回收模式的内存分配实现方式中，在进行物理内存分配时，每次分配一个小块的内存(例如1MB或512KB)，称这样一个小块的内存称为内存块，多次分配后可形成一个解码通道的内存块池。内存块池的大小可对应帧存开销的最小值，例如每个帧缓冲区需要分配15MB的内存空间，针对6帧图像帧，4K视频的解码通道大约需要90MB(即15MB×6＝90MB)。在解码过程中，可在内存中的内存块池中为所述n个经压缩后的片分配所述帧缓冲区，所述帧缓冲区包括M个内存块(例如每个内存块为1MB，那么一个15MB的帧缓冲区需要15个内存块)，每个内存块包括多个物理页面，所述多个物理页面对应于多个页号连续的虚拟页面，但这些虚拟页面对应的物理页面可以是离散的(即这些物理页面的页号可以是不连续的)。所述n个经压缩后的片所占用的内存空间小于等于所述M个内存块对应的内存空间。在n个经压缩后的片需要被实际存入内存的时候，建立虚拟地址与物理地址的映射关系，之后，再将n个经压缩后的片紧凑排列地存入该帧缓冲区的内存块。最后，根据实际内存占用量，将帧缓冲区中未被占用的剩余内存块摘除，重新回收到内存块池，所述剩余内存块可用于后续的帧缓冲区分配和片存储，从而实现了内存空间的动态利用，大大提升了内存的使用效率。
其中，本发明实施例为所述n个经压缩后的片在所述帧缓冲区中所占用的内存空间建立虚拟页面与物理页面的映射关系的过程如下：寻找到内存块池中的无占用的内存块，这些内存块分别对应一个页表，该页表记录了对应的内存块中的各个虚拟页面与物理页面的映射关系。然后，将足量的内存块(M个内存块)组合成容器所指示大小的存储空间，例如虚拟内存空间为15MB时，选择总量为15MB的若干个内存块组合为容器所需的物理内存空间，例如将15个1MB的内存块进行组合。具体的，可将所述总量为15MB的若干个内存块对应的各个页表组合成帧缓冲区的页表，这样，就实际建立起了帧缓冲区中虚拟地址与物理地址的映射关系，从而确保该帧缓冲区拥有真正的15M物理内存
可以看到，本发明实施例通过可将图像帧无损压缩后紧凑内存排布存入所分配的帧缓冲区，通过对物理内存空间进行精细化分配，实现对碎片内存的归集利用，视频解码器解码完一帧图像帧并存入帧缓冲区后，可将帧缓冲区没有用完的内存回收出来，释放未被占用的空闲内存，合并到下一帧的解码使用中。由于无损压缩不损失图像质量，这样，就能 用较少的内存预算，实际获得较多的帧缓冲区个数，从而确保各项性能指标并不受损，实现了不损失图像质量的前提下既节省带宽又节省帧存的目的。
(2)在采用预分配模式的内存分配实现方式中，可在解码前为图像帧分配的帧缓冲区只包括虚拟内存空间，而不包括物理内存空间，该虚拟内存空间包括多个页号连续的虚拟页面。也就是说，一开始分配帧缓冲区时建立的页表只指示了一个或多个虚拟页面，而没有指示虚拟页面与物理页面的映射关系，与前一种实现方案不同，本按需分配方案不涉及内存回收而是在分配实际物理内存之前提前分配虚拟内存空间。例如解码时，需要分配15个帧缓冲区，每个帧缓冲区占15MB空间，则在虚拟内存空间中足额分配15MB×15的空间。但对于这15个帧缓冲区中的每一个帧缓冲区，其对应的每个虚拟页面都可能没有物理页面与其进行映射。在需要将图像帧的数据单元存入内存时，再分别对所述一个或多个虚拟页面的页号进行页表查询；若查询到页表中不具有一个或多个虚拟页面的页号与物理页面的页号的映射关系，则在所述内存中为所述一个或多个虚拟页面分配所述一个或多个物理页面，从而实现了按照该数据单元的实际需求来分配实际的物理页面。
其中，当需要在内存中寻找未被占用的一个或多个物理页面时，可查询位图(bit map)，位图包括多个位，每个位分别用于指示内存中的一个物理页面是否被占用，然后根据位图来寻找到未被占用的一个或多个物理页面。例如，位图中的各个位分别指示了一个物理页面的占用状态，其中“0”代表对应的物理页面未被占用，“1”代表了对应的物理页面已被占用。在经压缩后的数据单元存入帧缓冲区过程中，当需要寻找未被占用的4个物理页面时，可根据位图寻找4个值为“0”的位所指示的4个物理页面。
本发明实施例中，建立帧缓冲区的虚拟页面与物理页面的映射关系的过程如下：在需要将经压缩后的数据单元存入帧缓冲区时，根据数据单元对应的一个或多个虚拟页面的页号进行页表查询，若查询不到与所述一个或多个虚拟页面的页号对应的物理页面的页号，则在内存中寻找未被占用的一个或多个物理页面，将所述一个或多个物理页面的页号增加到页表，以使得页表中所述一个或多个虚拟页面的页号中的每一个分别对应一个物理页面的页号。这样，就建立帧缓冲区的虚拟页面与物理页面的映射关系。
可以看到，本发明实施例通过改进页表的建立机制，实现了内存“按需占用”，图像帧经过无损压缩后，通过紧凑内存排布存入按需分配的物理内存空间，由于不损失图像质量，所以实现了不损失图像质量的前提下既节省带宽又节省帧存的目的。另外，本发明实施例中在查询页表发现缺少物理页面与虚拟页面的对应关系时，能够自动补充物理页面，使缺页处理过程对媒体框架透明，使得本发明实施例技术方案可以适用于任何形态的媒体框架，有利于提升应用性和用户体验。
基于第一方面，在可能的实施方式中，在存储过程可将多个经压缩后的片、多个压缩信息、和一个头地址信息组成一个数据单元的形式存储到所述帧缓冲区。例如，图像帧以CTU为单位形成数据单元进行数据存取，图像帧划分成n个相同数据长度的片，每个数据单元例如包括3个片，此外，每个数据单元还包括一个头地址信息以及每个片所对应的压缩信息，头地址信息指示了各个数据单元的起始地址，压缩信息指示了各个片的压缩模式/压缩后的长度。如图7所示，图像帧中的数据单元1可包括头地址信息1、片1的压缩信息1、片2的压缩信息2、片3的压缩信息3、以及片1、片2、片3。在图像压缩过程根据 需要存入帧缓冲区的数据量来动态分配物理内存空间，并且将各个经过压缩后的数据单元依次紧凑地存入帧缓冲区，实现所占用内存空间的归集，且实现建立了各个经过压缩后的数据单元与存储地址之间的映射关系。这样，配合原有的各片的长度就可以实现各个数据单元的随机存取。在经压缩后的图像完全存入帧缓冲区后，便可释放帧缓冲区中未占用的内存空间
基于第一方面，在可能的实施方式中，在分别将所述n个经压缩后的片存储到内存的帧缓冲区后，在对后续图像进行解码的过程中，还包括：从所述帧缓冲区分别读取所述n个经压缩后的片；对所述n个经压缩后的片做解压得到第一图像帧的解码帧；以所述解码帧作为参考帧进一步对所述码流进行解码得到第三图像帧。需要说明的是，如果上述压缩过程采用是无损压缩，那么所述解码帧就是所述第一图像帧；如果压缩过程采用是其他的压缩方式(如有损压缩)，且该压缩方式造成了部分图像损失，那么解码帧为损失部分信息的第一图像帧。
第二方面，本发明实施例提供了一种内存管理的装置，所述装置包括解码模块、内存管理模块、压缩模块和解压缩模块，其中：所述解码模块，用于对码流进行解码得到第一图像帧，所述第一图像帧包括n个片，每个片表示所述第一图像帧中的一个区域的图像，n是大于等于2的整数；所述压缩模块，用于对所述第一图像帧进行压缩得到第二图像帧，所述第二图像帧包括n个经压缩后的片；所述内存管理模块，用于在内存中为所述n个经压缩后的片分配帧缓冲区；还用于为所述n个经压缩后的片在所述帧缓冲区中所占用的内存空间建立虚拟页面与物理页面的映射关系，其中，每个经压缩后的片所占用的内存空间包括一个或多个物理页面，在所述映射关系下所述n个经压缩后的片所占用的多个虚拟页面的页号连续；还用于分别将所述n个经压缩后的片存储到所述帧缓冲区。
需要说明的是，上述功能模块具体用于实现第一方面所描述的方法，相关实施方式的实现过程还可参考第一方面的描述，这里不再赘述。
第三方面，本发明实施例提供了又一种内存管理的装置，所述装置包括处理器和存储器，其中：所述处理器用于，对码流进行解码得到第一图像帧，所述第一图像帧包括n个片tile，每个片表示所述第一图像帧中的一个区域的图像，n是大于等于2的整数；对所述第一图像帧进行压缩得到第二图像帧，所述第二图像帧包括n个经压缩后的片；在所述存储器中为所述n个经压缩后的片分配帧缓冲区；还用于为所述n个经压缩后的片在所述帧缓冲区中所占用的内存空间建立虚拟页面与物理页面的映射关系，其中，每个经压缩后的片所占用的内存空间包括一个或多个物理页面，在所述映射关系下所述n个经压缩后的片所占用的多个虚拟页面的页号连续；还用于分别将所述n个经压缩后的片存储到所述帧缓冲区。
需要说明的是，所述处理器和存储器具体用于实现第一方面所描述的方法，相关实施方式的实现过程还可参考第一方面的描述，这里不再赘述。
第四方面，本发明实施例提供了又一种非易失性计算机可读存储媒介；所述计算机可读存储媒介用于存储第二方面所述方法的实现代码。所述程序代码被计算设备执行时，所述用户设备用于第一方面所述方法。
第五方面，本发明实施例提供了一种计算机程序产品；该计算机程序产品包括程序指 令，当该计算机程序产品被计算设备执行时，该控制器执行前述第一方面所述方法。该计算机程序产品可以为一个软件安装包，在需要使用前述第一方面的任一种可能的设计提供的方法的情况下，可以下载该计算机程序产品并在控制器上执行该计算机程序产品，以实现第一方面所述方法。
可以看到，本发明实施例能够实现图像帧经过压缩(如无损压缩)后，存入帧缓冲区中虚拟地址连续的内存区域，实现了所占用内存区域的归集，所以实施本发明实施例能够达到有效降低解码时的参考帧存取对内存的占用目的。另外，采用本发明实施例提供的内存回收方案还能将帧缓冲区没有用完的内存回收出来，合并到下一帧的解码使用，实现对碎片内存的有效利用；采用本发明实施例提供的按需分配方案能够改进页表的建立机制，实现了内存“按需占用”，极大提升了内存的使用效率。
附图说明
为了更清楚地说明本发明实施例或背景技术中的技术方案，下面将对本发明实施例或背景技术中所需要使用的附图进行说明。
图1是本发明实施例提供的一种视频编解码系统的示意性框图；
图2是本发明实施例提供的一种视频编解码装置的示意性框图；
图3是本发明实施例提供的一种视频编解码系统中的目的地装置的示意性框图；
图4是本发明实施例提供的一种将图像帧划分为多个片的示意图；
图5是现有技术提供的一种随机存取的实现方案示意图；
图6是本发明实施例提供的一种随机存取的实现方案示意图；
图7是本发明实施例提供的一种数据单元的数据格式示意图；
图8是本发明实施例提供的一种内存分配的应用场景示意图；
图9是本发明实施例提供的一种建立虚拟页面和物理页面映射关系的应用场景示意图；
图10是本发明实施例提供的又一种内存分配的应用场景示意图；
图11是本发明实施例提供的又一种通过SMMU进行缺页处理的应用场景示意图；
图12是本发明实施例提供的一种内存管理方法的流程示意图；
图13是本发明实施例提供的又一种内存管理方法的流程示意图；
图14是本发明实施例提供的又一种内存管理方法的流程示意图。
具体实施方式
下面结合本发明实施例中的附图对本发明实施例进行描述。本发明的实施方式部分使用的术语仅用于对本发明的具体实施例进行解释，而非旨在限定本发明。
下面首先介绍本发明实施例涉及的系统架构。参见图1，图1是本发明实施例提供的一种视频编解码系统的示意性框图。如图1所示，视频编解码系统包含源装置10a及目的地装置10b。源装置10a可产生经编码视频数据并发送码流至目的地装置10b，因此，源装置10a可被称作视频编码装置或视频编码设备。目的地装置10b可解码由源装置10a产生的经编码视频数据并进行视频显示，因此，目的地装置10b可被称作视频解码装置或视频 解码设备。源装置10a及目的地装置10b可为视频编解码装置或视频编解码设备的实例。源装置10a及目的地装置10b可包括广泛范围的装置，包含台式计算机、移动计算装置、笔记本(例如，膝上型)计算机、平板计算机、机顶盒、智能手机等手持机、电视、相机、显示装置、数字媒体播放器、视频游戏控制台、车载计算机，或其类似者。
在一实例中，源装置10a可经由磁盘存取或卡存取方式来存取存储媒介。存储媒介可包含多种本地存取式数据存储媒介，例如DDR内存、蓝光光盘、DVD、CD-ROM、快闪存储器，等等。
目的地装置10b可经由信道17接收来自源装置10a的编码后的视频数据。信道17可包括能够将经编码视频数据从源装置10a移动到目的地装置10b的一个或多个媒介及/或装置。在一个实例中，信道17可包括使源装置10a能够实时地将编码后的视频数据直接传输到目的地装置10b的一个或多个通信媒介。在此实例中，源装置10a可根据通信标准(例如，无线通信协议)来调制编码后的视频数据，且可将调制后的视频数据传输到目的地装置10b。所述一个或多个通信媒介可包含无线及/或有线通信媒介，例如射频(RF)频谱或一根或多根物理传输线。所述一个或多个通信媒介可形成基于包的网络(例如，局域网、广域网或全球网络(例如，因特网))的部分。所述一个或多个通信媒介可包含路由器、交换器、基站，或促进从源装置10a到目的地装置10b的通信的其它设备。
在另一实例中，信道17可包含存储由源装置10a产生的编码后的视频数据的存储媒介。在此实例中，目的地装置10b可经由磁盘存取或卡存取来存取存储媒介。存储媒介可包含多种本地存取式数据存储媒介，例如DDR内存、蓝光光盘、DVD、CD-ROM、快闪存储器，等等。
在另一实例中，信道17可包含文件服务器或存储由源装置10a产生的编码后的视频数据的另一中间存储装置。在此实例中，目的地装置10b可经由流式传输或下载来存取存储于文件服务器或其它中间存储装置处的编码后的视频数据。文件服务器可以是能够存储编码后的视频数据且将所述编码后的视频数据传输到目的地装置10b的服务器类型。实例文件服务器包含web服务器(例如，用于网站)、文件传送协议(FTP)服务器、网络附加存储(NAS)装置，及本地磁盘驱动器。目的地装置10b可经由标准数据连接(例如，因特网连接)来存取编码后的视频数据。数据连接的实例类型包含适合于存取存储于文件服务器上的编码后的视频数据的无线信道(例如，Wi-Fi连接)、有线连接(例如，DSL、缆线调制解调器等)，或两者的组合。编码后的视频数据从文件服务器的传输可为流式传输、下载传输或两者的组合。
本发明实施例的技术不限于无线应用场景，示例性的，可将所述技术应用于支持以下应用等多种多媒体应用的视频编解码：空中电视广播、有线电视发射、卫星电视发射、流式传输视频发射(例如，经由因特网)、存储于数据存储媒介上的视频数据的编码、存储于数据存储媒介上的视频数据的解码，或其它应用。在一些实例中，视频编解码系统可经配置以支持单向或双向视频传输，以支持例如视频流式传输、视频播放、视频广播及/或视频电话等应用。
在图1的实例中，源装置10a包含视频数据源11、视频编码器12及DDR内存13。此外，在一些实例中，源装置10a还包括输出接口(图1未示)，输出接口可包含调制器/解 调器(调制解调器)及/或发射器，用于将编码的码流通过信道17发送至目的地装置10b。
其中，视频数据源11可包含视频俘获装置(例如，视频相机)、含有先前俘获的视频数据的视频存档、用以从视频内容提供者接收视频数据的视频输入接口，及/或用于产生视频数据的计算机图形系统，或上述视频数据源的组合。
视频编码器12可编码来自视频数据源11的视频数据，视频数据包括多帧图像帧。在一些实例中，视频编码器12可将编码后的视频数据(例如参考帧)经过压缩后存储于DDR内存13中，相应的，视频编码器12还可解压缩DDR内存13中的参考帧，利用该参考帧来继续编码其他的图像帧。
在图1的实例中，目的地装置10b包含视频解码器14、视频显示器15和DDR内存16。在一些实例中，目的地装置10b还可包括输入接口(图未示出)、输入接口可包含接收器及/或调制解调器。输入接口可经由信道17接收码流，该码流包括编码后的视频数据。
其中，视频解码器14可解码码流中的经编码后的视频数据，得到解码后的视频数据。在一些实例中，视频解码器14可将解码后的视频数据经过压缩后存储于DDR内存16中，相应的，视频解码器14还可解压缩DDR内存16中的视频数据，以便于将解压缩后得到的图像帧作为参考帧来供视频解码器14继续解码后续的码流。另外，当需要进行视频显示时，也可以解压缩DDR内存16中的视频数据，解压缩后得到的视频数据可输入至视频显示器15。
视频显示器15可与目的地装置10b整合在一起，或者，视频显示器15可独立部署在目的地装置10b的外部。一般来说，视频显示器15可根据所输入的视频数据进行视频显示。视频显示器15可包括多种显示装置，例如液晶显示器(LCD)、等离子体显示器、有机发光二极管(OLED)显示器或其它类型的显示装置。
本发明实施例所描述的内存管理方法主要应用于源装置10a和/或目的地装置10b相应的编解码中的数据压缩/解压缩过程。
参见图2，图2是本发明实施例提供的一种视频编解码装置50的示意性框图，所述视频编解码装置50可用于实现源装置10a和/或目的地装置10b的功能。装置50可以例如是无线通信系统的移动终端或者有线通信系统的用户设备。应理解，可以在可能需要对视频图像进行编码和解码，或者编码，或者解码的任何电子设备或者装置内实施本发明的实施例。
如图2所示，装置50可以包括用于控制装置50的处理器56(或者控制器)、存储器58和编码解码器54，对应于图1中的源装置10a的视频编码器12或目的地装置10b的视频解码器14。处理器56可用于运行操作系统(OS)，处理器56可以连接到存储器58，该存储器58例如为DDR内存(或称DDR内存条)、蓝光光盘、DVD、CD-ROM、快闪存储器等等存储媒介，在本发明具体实施例中，存储器58可以存储经压缩图像的数据(例如经过无损压缩后的图像帧)，和/或也可以存储用于在处理器56上实施的指令，该存储器58对应于图1中的源装置10a的DDR内存13或目的地装置10b的DDR内存16。处理器56可连接到适合于实现视频数据的编码和解码或者由处理器56控制的辅助编码和解码的编码解码器54。
装置50还可以包括形式为液晶显示器的显示器32，显示器32可以是适合于显示图像 或者视频的任何适当的显示器技术，对应于目的地装置10b的视频显示器15。装置50还可以包括小键盘34。在本发明的其它实施例中，可以运用任何适当的数据或者用户接口机制。例如，可以实施用户接口为虚拟键盘或者数据录入系统作为触敏显示器的一部分。装置可以包括麦克风36或者任何适当的音频输入，该音频输入可以是数字或者模拟信号输入。装置50还可以包括如下音频输出设备，该音频输出设备在本发明的实施例中可以是以下各项中的任何一项：耳机38、扬声器或者模拟音频或者数字音频输出连接。装置还可以包括用于与其它设备的近程视线通信的红外线端口42。在其它实施例中，装置50还可以包括任何适当的近程通信解决方案，比如蓝牙无线连接或者USB/火线有线连接。
可选的，装置50还可以包括用于提供用户信息并且适合于提供用于在网络认证和授权用户的认证信息的读卡器48和智能卡46，例如UICC和UICC读取器。
装置50还可以包括无线电接口52，该无线电接口52连接到处理器56并且适合于生成例如用于与蜂窝通信网络、无线通信系统或者无线局域网通信的无线通信信号。装置50还可以包括天线44，该天线44连接到无线电接口52用于向其它(多个)装置发送在无线电接口52生成的射频信号并且用于从其它(多个)装置接收射频信号。在本发明的一些实施例中，装置50可以通过无线或者有线连接接收视频数据进行编码/解码。
本发明实施例所描述的内存管理方法主要应用于装置50相应的编解码中的数据压缩/解压缩过程。下文主要从解码侧的角度来具体描述本发明实施例所提供的内存管理方案，需要理解的是，本发明实施例所提供的内存管理方案也可以应用于编码侧，即关于编码侧的内存管理方案可参考解码侧的实现方案进行类似实施，这是因为在编码侧有时也会涉及到解码方案，因此也可以使用本实施例提到的适用于解码侧的方案，所以后文将不再赘述。
参见图3，图3示出了一种视频编解码系统中的目的地装置的具体逻辑结构，该目的地装置包括视频解码器20、视频显示器30和内存(包括帧缓冲区22，帧缓冲区又可称为帧存)，如DDR内存。其中，视频解码器20在硬件实现上可为图2实施例所示的编码解码器54、或包括编码解码器54或是其一部分，视频解码器20包括解码器21和内存管理模块23。另外，视频解码器20还可包括压缩模块24和解压缩模块25，在一可能实施例中，压缩模块24和解压缩模块25可以作为解码器的功能模块来实现；在又一可能实施例中，压缩模块24和解压缩模块25也可以独立于解码器来实现。
其中，帧缓冲区(frame buffer)22用于实现存储视频数据，具体的可存储经过压缩模块24采用无损压缩方式压缩的图像帧。具体实现中，目的地装置的内存可采用DDR来实现，帧缓冲区22可位于DDR内存中，也就是说，帧缓冲区22指示了内存中所分配的用于图像存储的存储区域。具体实现中，帧缓冲区22可以是视频显示器所显示画面的一个直接映象，帧缓冲区的每一存储单元对应屏幕上的一个像素，整个帧缓冲区对应一帧图像。根据应用场景的需要，帧缓冲区22的数量可以有多个，每个帧缓冲区22可用于动态存储一个用于作为后续解码参考的参考帧或者用于视频显示的图像帧。
需要说明的是，本实施例的过压缩模块24和解压缩模块25可以使用无损压缩技术和有损压缩技术，并在使用无损压缩技术时达到更加的技术效果。本实施例仅以无损压缩技术为例作说明，但不用于限定实施例方案在的有损压缩技术场景下的应用。
内存管理模块23可通过动态的内存分配来为帧缓冲区22分配内存空间(虚拟内存空 间/物理内存空间)，实现有效归集利用内存空间或回收空闲内存空间，从而节省内存的消耗。
具体实施例中，解码器21对输入的码流进行解码，输出图像帧。为节省带宽，图像帧需经过压缩模块24压缩之后，再存入内存管理模块23中所配置的帧缓冲区22。在一可能实施例中，存入帧缓冲区22的图像帧作为参考帧，在后续解码过程中该参考帧经过解压缩模块25进行解压缩之后应用到解码器21中进行解码参考；在又一可能的实施例中，存入帧缓冲区22的图像帧经过解压缩模块31之后进行解压缩之后应用到视频显示器30中进行视频显示。可以理解的，本发明实施例中，压缩过程和解压缩过程之间互相对应，也就是说，当压缩模块采用无损压缩方式对图像进行压缩，那么解压缩模块25或解压缩模块31就是相应地对经无损压缩后的图像进行解压缩，恢复原先的图像且不引起图像失真。
本发明实施例中，可将图像帧沿着水平和垂直方向分割成n个相同的矩形区域，每个矩形区域成为一个片(tile)。这样，可以以片为单元进行图像数据的存取。此外，图像帧还可以有多种分割方式，比如在H.265中，图像帧可被分割为多个大小相等的树形编码单元(CTU)，CTU是H.265基本处理单元，每个CTU的大小例如可以是16X16或32X32或64X64等等。这样，每个CTU可能会包括多个片。举例来说，如图4所示的一种图像帧分割方式示例中，一个图像帧可包括8个CTU，同时，该图像帧可分割成24个片，这样，每个CTU可包括3个片。需要说明的是，本示例仅仅用于解释而非限定。
下面介绍本发明实施例涉及的虚拟内存空间和物理内存空间。在计算科学中，内存可分为虚拟内存空间和物理内存空间。物理内存空间直接与内存条的物理硬件相关，表示根据内存条的地址线直接进行寻址的内存空间大小。示例性地，处理器的操作系统(OS)运行的一个程序(进程)时，其所需的内存空间是有可能大于内存条的总容量，举例来说内存条的总容量是256M，而程序却需要创建一个2G的数据区，那么就无法将所有数据都能一起加载到内存条的物理内存中，有一部分数据要放到其他媒介(比如硬盘)中，也就是说，操作系统就会借助其他媒介空间作为虚拟的内存空间，把当前进程暂时不需要的数据放在其他媒介上。待进程需要访问那部分数据时，再通过调度进入物理内存空间。或者，OS或其他软件的进程为了管理在物理内存中碎片化的数据，通过可映射到物理内存的虚拟内存来掌控。尽管物理内存的地址是碎片化、和非连续的，但进程所观察到的虚拟内存仍然是连续的，便于进程进行处理。以上仅仅是举例，总体来说，虚拟内存空间可以理解为OS或其他软件的进程运行时所有内存空间(如2G)的总和(即可理解为在进程的角度所看到的内存空间)，并且可能有一部分不在物理内存空间(如256M)中，而是在其他媒介中。所以，虚拟内存和物理内存之间需要建立映射关系，使得软件进程能管理和掌控与物理内存对应的虚拟内存的操作。
在计算科学的内存分页机制中，操作系统对虚拟内存地址空间进行分页产生虚拟页面(page)，对物理内存地址空间进行分页产生物理页面(或称物理页帧，page frame)，虚拟页面包含相应的虚拟地址，物理页面包含相应的物理地址，一个虚拟页面和一个物理页面的空间大小是一样的，所以，虚拟页面的总数要大于物理页面的总数。在处理器中，设置有系统内存管理单元(System Memory Management Unit，SMMU)，SMMU可用处理器来管理虚拟内存和物理内存的控制线路，负责将虚拟地址映射为物理地址，以及提供硬件机 制的内存访问授权等等。具体的，SMMU维护有一个页表(page table)，页表可用于记录虚拟页面与物理页面的映射关系，更确切的说，是记录虚拟页面的页号与物理页面的页号之间的一一映射关系。
本发明实施例中，为了保证图像帧数据存入内存的帧缓冲区后内存不膨胀以及寻址方便，图像帧数据需要能够在帧缓冲区中随机存取。为了更好理解本发明实施例的技术思想，下面首先比较现有技术与本发明实施例在随机存取方案的实现上的差异。
参见图5，图5为一种随机存取的实现方案。在该方案中，系统固定分配用于存储图像帧的帧缓冲区，分配帧缓冲区的时候不仅为该帧缓冲区分配了虚拟内存空间，还为该帧缓冲区分配了物理内存空间，并将虚拟内存空间的虚拟页面和物理内存空间的物理页面完全对应起来。具体的，图像帧划分成n个相同数据长度的片，在不采用无损压缩技术对图像进行压缩时，这种方案通过固定每个片的起始地址，将片的坐标固定映射为片的起始存储地址，这样就能直接建立各个片与其存储地址的静态映射关系(如图5上半部分示出灰色区域分别表示片1、片2…片的数据长度)。在采用无损压缩后，不同的片被压缩的程度各有差异(如图5下半部分示出灰色区域分别表示片1、片2…片n压缩后的数据长度)，但是各个数据单元的起始存储地址依然固定不变，所以在帧缓冲区存取图像帧时能够实现随机存取。然而，通过无损压缩后空闲出来的内存(如图4下半部分示出空白区域表示空闲出来的内存)也很难被利用到其他帧存中，造成浪费。
为了解决现有技术中无损压缩方式形成的内存浪费，本发明实施例进一步提供的内存管理模块23采用动态分配帧存的策略，在对帧缓冲区22的存储空间分配中实现精细粒度的内存分配或空闲内存的回收，实现效率最大化。
内存管理模块23可根据解码器21的需要预先分配帧缓冲区(如图6上半部分的白色区域)，该帧缓冲区只是一片虚拟空间。参见图6，在本发明一种实施例中，图像帧以片为单位进行数据存取，图像帧划分成n个相同数据长度的片。在图像压缩过程根据需要存入帧缓冲区的数据量来动态分配物理内存空间，并且将各个经过压缩(例如无损压缩)后的片依次紧凑地存入帧缓冲区，实现所占用内存空间的归集(如图6下半部分示出灰色区域)，各个经过压缩后的片的起始存储地址不再固定。为了实现随机存储，本发明实施例重新建立各个经过压缩后的片与存储地址之间的映射关系。具体的，除了存储各个经过压缩后的片之外，本发明实施例还存储了各个片对应的头地址信息以及压缩信息，头地址信息指示了各个经过压缩后的片的起始地址，压缩信息指示了各个经过压缩后的片的压缩模式/压缩后的长度，这样配合原有的各片的长度就可以实现各个片的随机存取。在经压缩后的图像完全存入帧缓冲区后，便可释放帧缓冲区中未占用的内存空间(如图6下半部分示出空白区域)。
参见图7，在本发明实施例一种具体实现中，图像帧以CTU为单位形成数据单元进行数据存取，图像帧划分成n个相同数据长度的片，每个数据单元例如包括3个片，此外，每个数据单元还包括一个头地址信息以及每个片所对应的压缩信息，头地址信息指示了各个数据单元的起始地址，压缩信息指示了各个片的压缩模式/压缩后的长度。如图7所示，图像帧中的数据单元1可包括头地址信息1、片1的压缩信息1、片2的压缩信息2、片3的压缩信息3、以及片1、片2、片3。在图像压缩过程根据需要存入帧缓冲区的数据量来 动态分配物理内存空间，并且将各个经过压缩后的数据单元依次紧凑地存入帧缓冲区，实现所占用内存空间的归集，且实现建立了各个经过压缩后的数据单元与存储地址之间的映射关系。这样，配合原有的各片的长度就可以实现各个数据单元的随机存取。在经压缩后的图像完全存入帧缓冲区后，便可释放帧缓冲区中未占用的内存空间
下面描述本发明实施例中，内存管理模块23采用动态分配帧存的策略，实现精细粒度的内存分配或内存回收的几种实现方式。如下面(1)和(2)两种方式分别为内存回收模式和预分配模式，以实现内存资源充分利用。
(1)一种精细粒度的内存分配实现方式中，采用内存回收模式，对操作系统而言，内存管理模块23对视频帧存的分配行为可分为如下两方面：
一方面是通过内存管理模块23或操作系统OS中的视频固件(Video Firmwire，VFMW)进行物理内存分配，如图8所示，每次分配一个小块的内存(例如1MB或512KB)，称这样一个小块的内存称为内存块，多次分配后可形成一个解码通道的内存块池。内存块池的大小可对应帧存开销的最小值，例如每个帧缓冲区需要分配15MB的内存空间，针对6帧图像帧(即对应于图8所示的6个帧缓冲区)，4K视频的解码通道大约需要90MB(即15MB×6＝90MB)。内存块池中的内存由多个离散的内存块构成。每个内存块内的虚拟地址是连续的，也就是说，每个内存块对应多个虚拟页面，同一内存块中的这些虚拟页面之间的页号连续，每个虚拟页面对应于一个物理页面，但这些虚拟页面对应的物理页面可以是离散的(即这些物理页面的页号可以是不连续的)。
另一方面，可通过内存管理模块23预先为每个帧缓冲区分配足够的虚拟内存空间(即只分配虚拟地址空间)，这样的虚拟内存空间又可称为容器。例如如果每个帧缓冲区需要分配15MB的内存空间，那么需要为每个帧缓冲区分配15MB的容器。通过所分配的容器，播放框架看到的“图像帧”依然具有原先固定的虚拟地址与数据长度，从而本发明实施例的技术方案能够兼容于现有的各种播放框架。在容器分配时，帧缓冲区中的虚拟地址与物理地址的对关系是不确定的，这种对应关系在图像帧需要被实际存入内存的时候才建立起来。
下面进一步描述本发明实施例中，在图像帧需要被实际存入内存的时候，内存管理模块23建立虚拟地址与物理地址的映射关系(即容器与物理内存空间的映射关系)的实施方式。
本发明实施例中，可在VFMW拥有内存块池的前提下，在解码之前为每个帧缓冲区(即空有虚拟地址而无物理内存空间的容器)分配实际的物理内存空间。由于无损压缩的压缩率具有不确定性，所以在本发明可能实施例中，在解码器解码每帧图像帧之前，都将对应的各个容器注满实际的物理内存空间，也就是说，解码启动之前，建立帧缓冲区的虚拟地址与物理地址之间的映射关系，其实现过程如下：寻找到内存块池中的无占用的内存块，这些内存块分别对应一个页表，该页表记录了对应的内存块中的各个虚拟页面与物理页面的映射关系。然后，将足量的内存块组合成容器所指示大小的存储空间，例如一个容器所指示的虚拟内存空间为15MB时，选择总量为15MB的若干个内存块组合为容器所需的物理内存空间，例如将15个1MB的内存块进行组合。具体的，可将所述总量为15MB的若干个内存块对应的各个页表组合成帧缓冲区的页表，这样，就实际建立起了帧缓冲区中虚 拟地址与物理地址的映射关系，从而确保该帧缓冲区拥有真正的15M物理内存。解码结束后，将图像帧各个经压缩后的数据单元紧凑排列地存入该帧缓冲区，根据解码器上报的实际内存占用量，将帧缓冲区中未被占用的内存块摘除，重新回收到内存块池。
例如，参见图9，在对帧缓冲区0的实际存储空间分配中，寻找到M个内存块(内存块1、内存块2…内存块M)来组合成实际存储空间，每个内存块都具有对应的页表，将这些页表拷贝在一起组合为帧缓冲区0的页表。解码结束后，根据解码器上报的内存占用量，将多余的内存块标记为“空闲”，释放这些空闲的内存块回收到内存块池，这些内存块池中的内存块可继续应用于后续的帧缓冲区(容器)，从而实现了内存空间的动态利用。
在一种具体应用场景中，上述过程可包括：OS提供分配帧缓冲区的功能，封装成“VFB_Alloc()”，其返回的虚拟空间没有物理页表对应。内存管理模块自行管理物理内存预算，比如4K解码通道需要90MB内存(包括6个帧缓冲区)，则通过“MMB_New()”指令以较小粒度(比如1MB或512KB)分配多块，总容量为90MB。在解码启动之前，解码器执行帧缓冲区的虚拟地址与物理地址之间的对应，寻找空闲的内存块，并将各个内存块对应的页表挂接到帧缓冲区页表，确保当前针对的帧缓冲区是真正的15M物理内存，避免出现缺页。解码结束后，内存管理模块根据解码器上报的当前的帧缓冲区的内存占用量，将无占用的内存块摘除，重新回收到内存块池，以便后续使用。
可以看到，本发明实施例通过可将图像帧无损压缩后紧凑内存排布存入所分配的帧缓冲区，通过对物理内存空间进行精细化分配，实现对碎片内存的归集利用，解码器解码完一帧图像帧并存入帧缓冲区后，内存管理模块可将帧缓冲区没有用完的内存回收出来，释放未被占用的空闲内存，合并到下一帧的解码使用中。这样，就能用较少的内存预算，实际获得较多的帧缓冲区个数，从而确保各项性能指标并不受损，实现了不损失图像质量的前提下既节省带宽又节省帧存的目的。
(2)在又一种内存分配实现方式中，采用预分配模式或按需分配方式，内存管理模块23可在解码前为图像帧分配的帧缓冲区只包括虚拟内存空间，而不包括物理内存空间，该虚拟内存空间包括多个页号连续的虚拟页面。也就是说，一开始分配帧缓冲区时建立的页表只指示了虚拟页面，而没有指示虚拟页面与物理页面的映射关系，与前一种实现方案不同，本按需分配方案不涉及内存回收而是在分配实际物理内存之前提前分配虚拟内存空间。例如解码时，需要分配15个帧缓冲区，每个帧缓冲区占15MB空间，则在虚拟内存空间中足额分配15MB×15的空间。但对于这15个帧缓冲区中的每一个帧缓冲区，其对应的每个虚拟页面都可能没有物理页面与其进行映射。在需要将图像帧的数据单元存入内存时，通过缺页处理的方式按照该数据单元的实际需求来分配实际的物理页面。
本发明实施例中，建立帧缓冲区的虚拟页面与物理页面的映射关系的过程如下：在需要将经压缩后的数据单元存入帧缓冲区时，根据数据单元对应的一个或多个虚拟页面的页号进行页表查询，若查询不到与所述一个或多个虚拟页面的页号对应的物理页面的页号，则在内存中寻找未被占用的一个或多个物理页面，将所述一个或多个物理页面的页号增加到页表，以使得页表中所述一个或多个虚拟页面的页号中的每一个分别对应一个物理页面的页号。这样，就建立帧缓冲区的虚拟页面与物理页面的映射关系。之后，就可将经压缩后的数据单元存储到所述对应的物理页面。
在一种需要进行缺页处理的具体应用场景中，当需要在内存中寻找未被占用的一个或多个物理页面时，内存管理模块23可查询位图(bit map)，位图包括多个位，每个位分别用于指示内存中的一个物理页面是否被占用，然后根据位图来寻找到未被占用的一个或多个物理页面。如图10所示，位图中的各个位分别指示了一个物理页面的占用状态，其中“0”代表对应的物理页面未被占用，“1”代表了对应的物理页面已被占用。在经压缩后的数据单元存入帧缓冲区0过程中，当需要寻找未被占用的4个物理页面时，可根据位图寻找4个值为“0”的位所指示的4个物理页面。需要说明的是，在上述处理流程中，“寻找值为0的位”可以有多种多样的实现方式。比如，如果物理页面数量较少，可以直接采用依顺序查询的方式来寻找到值为“0”的位；如果物理页面数量非常多，即对应位图中的位的数目巨大，以128MB物理内存空间为例意味着32768个物理页，因此位图有32768个位，这种情况下可以通过“分级标示”的方法来查找“0”位，比如将32768个位分成256段，每段包含128个位，这128个位通过“相互乘积”来形成一个状态标识，如果某一段的状态标识为“0”，说明这一段就存在“0”位，如果某一段的状态标识为“1”，说明这一段就不存在“0”位。这样就能快速实现“寻找值为0的位”。
在本发明实施例的具体实现中，内存管理模块23可包括系统内存管理单元(SMMU)来实现帧缓冲区的实际存储空间分配过程中的缺页处理。参见图11，SMMU可包括转换检测缓冲区(Translation Lookaside Buffer，TLB)模块、缺页处理模块和仲裁模块。TLB模块可预先将TLB载入，动态维护页表中的页表项，在需要将经压缩后的数据单元存入帧缓冲区时，内存管理模块23可将数据单元对应的一个或多个虚拟页面的虚拟地址通过AXI(Advanced extensible Interface,高级可扩展接口)总线送入TLB模块，TLB模块将虚拟地址转化成虚拟页面的页号进行页表查询，如果查询不到与该虚拟页面的页号对应的物理页面的页号，则向缺页处理模块发送缺页请求。缺页处理模块通过仲裁模块和AXI总线来获得位图和物理页列表的对应关系，进而通过位图来寻找未被占用的一个或多个物理页面，获得所述一个或多个物理页面的物理地址，进而根据物理地址对应的页号增加到TLB模块的页表，以使得页表中所述一个或多个虚拟页面的页号中的每一个分别对应一个物理页面的页号，进而将更新后的页表填充到TLB模块，从而完成补页，为数据单元分配了物理页面，后续就可将经压缩后的数据单元存储到所述对应的物理页面。
可以看到，本发明实施例通过内存管理模块23改进页表的建立机制，实现了内存“按需占用”，图像帧经过无损压缩后，通过紧凑内存排布存入按需分配的物理内存空间，实现了不损失图像质量的前提下既节省带宽又节省帧存的目的。另外，内存管理模块23通过SMMU可以实现缺页时自动补充物理页面，使缺页处理过程对媒体框架透明，使得本发明实施例技术方案可以适用于任何形态的媒体框架，有利于提升应用性和用户体验。
基于上述系统架构和装置，下面进一步描述本发明实施例涉及的内存管理方法。
参见图12，图12示出了本发明实施例涉及的一种内存管理方法的流程示意图，该方法可应用于图3实施例所描述的视频解码器，该视频解码器包括解码器、压缩模块、解压缩模块、内存管理模块，压缩模块和解压缩模块可以作为解码器的功能模块来实现，或者，压缩模块和解压缩模块也可以独立于解码器来实现。该方法包括但不限于如下步骤：
步骤S101：解码器对码流进行解码得到第一图像帧，所述第一图像帧包括n个大小相同的片(tile)，每个片表示所述第一图像帧中的一个矩形区域的图像，n≥2。
步骤S102：压缩模块对所述第一图像帧进行压缩(如无损压缩)得到第二图像帧，所述第二图像帧包括n个经压缩后的片，可以理解的，由于各个片的图像数据各有差异，所以各个片经过无损压缩后，压缩率也各有差异，形成的各个经压缩后的片的数据长度也各有差异。
步骤S103：内存管理模块分别将所述n个经压缩后的片存储到内存的帧缓冲区(frame buffer)，所述n个经压缩后的片在所述帧缓冲区占用的物理页面所对应的虚拟页面具有连续的页号。其中，所述内存的帧缓冲区由内存管理模块进行配置和管理。
具体的，内存管理模块在内存中为所述n个经压缩后的片分配帧缓冲区，并为所述n个经压缩后的片在所述帧缓冲区中所占用的内存空间建立虚拟页面与物理页面的映射关系，其中，每个经压缩后的片所占用的内存空间包括一个或多个物理页面，在所述映射关系下所述n个经压缩后的片所占用的多个虚拟页面的页号连续，之后，内存管理模块分别将所述n个经压缩后的片存储到所述帧缓冲区。
也就是说，视频解码器可将各个经过压缩(例如无损压缩)后的片依次紧凑地存入帧缓冲区，实现所占用内存空间的归集，这些经过压缩后的片所占用的物理页面的页号可以是离散的，但是这些经过压缩后的片所占用的虚拟页面的页号是连续的。为了实现随机存储，本发明实施例还可重新建立各个经过压缩后的片与存储地址之间的映射关系。具体的，除了存储各个经过压缩后的片之外，本发明实施例还存储了各个片对应的头地址信息以及压缩信息，头地址信息指示了各个经过压缩后的片的起始地址，压缩信息指示了各个经过压缩后的片的压缩模式/压缩后的长度，这样配合原有的各片的长度就可以实现各个片的随机存取。具体实施情况还可参考图6实施例的描述。
另外，在可能的实施例中，图像帧还可根据数据单元的形式进行数据存取，所述数据单元在不同的应用场景中可以有多种多样的实现形式，例如数据单元为树形编码单元(Coding Tree Unit，CTU)、树形编码块(Coding Tree Block，CTB)、编码块(Coding Block，CB)、编码单元(Coding Unit，CU)、预测块(Prediction Block，PB)、宏块等等中的一种。具体的，可以将所述n个经压缩后的片划分成多个组，每个组用于形成一个数据单元，每一组包括不同的经压缩后的片，每一组中的每个经压缩后的片分别对应于一个压缩信息，所述压缩信息指示了对应的经压缩后的片所占用的存储空间；每一组对应一个头地址信息。也就是说，所述多个组中的每一组、以及该组对应的头地址信息、该组对应的压缩信息构成一个数据单元，进而内存管理模块将这样构成的多个数据单元分别存储到所述帧缓冲区，所述多个数据单元在所述帧缓冲区占用的物理页面所对应的虚拟页面具有连续的页号。具体实施情况还可参考图7实施例的描述。
需要说明的是，内存管理模块分别将所述n个经压缩后的片存储到内存的帧缓冲区后，在解码器对后续图像进行解码的过程中，还包括：内存管理模块从所述帧缓冲区分别读取所述n个经压缩后的片，解压缩模块对所述n个经压缩后的片做解压得到第一图像帧的解码帧，然后，解码器以所述解码帧作为参考帧进一步对所述码流进行解码得到第三图像帧。需要说明的是，如果上述步骤S102的压缩过程采用是无损压缩，那么所述解码帧就是所述 第一图像帧；如果上述步骤S102的压缩过程采用是其他的压缩方式(如有损压缩)，且该压缩方式造成了部分图像损失，那么解码帧为损失部分信息的第一图像帧。
可以看到，本发明实施例能够实现图像帧经过无损压缩后，存入帧缓冲区中虚拟地址连续的内存区域，实现了所占用内存区域的归集，所以实施本发明实施例能够实现了不损失图像质量的前提下既节省带宽又节省帧存的目的。
参见图13，图13示出了本发明实施例涉及的又一种内存管理方法的流程示意图，该方法可应用于图3实施例所描述的视频解码器，同样，该视频解码器包括解码器、压缩模块、解压缩模块、内存管理模块。该方法采用内存回收模式，包括但不限于如下步骤：
步骤S201：解码器对码流进行解码得到第一图像帧，所述第一图像帧包括n个大小相同的片，每个片表示所述第一图像帧中的一个矩形区域的图像，n≥2。
步骤S202：压缩模块对所述第一图像帧进行无损压缩得到第二图像帧，所述第二图像帧包括n个经压缩后的片。
步骤S203：内存管理模块在所述内存中分配未被占用的多个细粒度的内存块(例如1MB或512KB)，每个细粒度的内存块包括多个页号连续的虚拟页面，每个虚拟页面映射一个物理页面。也就是说，同一内存块中的这些虚拟页面之间的页号连续，每个虚拟页面对应于一个物理页面，但这些虚拟页面对应的物理页面可以是离散的。具体实施过程可参考图8实施例的描述，这里不再赘述。
步骤S204：内存管理模块将所述多个细粒度的内存块组合成所述帧缓冲区，所述多个内存块对应的物理内存空间之和为M，0≤M≤所述第一图像帧的数据量。
例如如果每个帧缓冲区需要分配15MB的内存空间，那么为了保险起见可将若干个内存块组合成15MB的帧缓冲区，当然，在可能的实施例中，如果能预先估计当前的无损压缩的压缩率，那么也可以根据预估的压缩率适当降低所需要分配的内存空间。比如预估到压缩率不低于20％，那么可将若干个内存块组合成12MB的帧缓冲区。
具体实现中，可确定各个内存块对应的页表，每个内存块对应的页表分别用于指示该内存块中的各个虚拟页面与物理页面的映射关系，然后将所述各个内存块对应的页表组合成所述帧缓冲区的页表。
本步骤的具体实施过程还可参考图8实施例的描述，这里不再赘述。
步骤S205：内存管理模块分别将所述n个经压缩后的片存储到帧缓冲区，所述n个经压缩后的片在所述帧缓冲区占用的物理页面所对应的虚拟页面具有连续的页号，实现虚拟内存页号的连续，便于软件做管理和维护。具体还可参考图11实施例步骤S103的描述，这里不再赘述。
步骤S206：内存管理模块释放所述帧缓冲区中未被所述多个数据单元占用的内存块。具体的，解码结束后，内存管理模块根据上报的内存占用量，将多余的内存块标记为“空闲”，释放这些空闲的内存块回收，这些回收的内存块可继续应用于后续的帧缓冲区，从而实现了内存空间的动态利用。比如，如果为帧缓冲区分配了15MB的物理内存空间，但是经过无损压缩在帧缓冲区依次紧凑地存入图像帧后，占用的实际内存空间为10MB，那么可释放无占用的5MB物理内存空间对应的内存块，将这5MB物理内存空间回收应用到下 一个帧缓冲区。
可以看到，本发明实施例通过可将图像帧无损压缩后紧凑内存排布存入所分配的帧缓冲区，通过对物理内存空间进行精细化分配，实现对碎片内存的归集利用，解码器解码完一帧图像帧并存入帧缓冲区后，内存管理模块可将帧缓冲区没有用完的内存回收出来，释放未被占用的空闲内存，合并到下一帧的解码使用中。这样，就能用较少的内存预算，实际获得较多的帧缓冲区个数，从而确保各项性能指标并不受损，实现了不损失图像质量的前提下既节省带宽又节省帧存的目的。
参见图14，图14示出了本发明实施例涉及的又一种内存管理方法的流程示意图，该方法可应用于图3实施例所描述的视频解码器，同样，该视频解码器包括解码器、压缩模块、解压缩模块、内存管理模块，该内存管理模块包含SMMU。该方法采用预分配模式，包括但不限于如下步骤：
步骤S301：解码器对码流进行解码得到第一图像帧，所述第一图像帧包括n个大小相同的片tile，每个片表示所述第一图像帧中的一个矩形区域的图像，n≥2.
步骤S302：压缩模块对所述第一图像帧进行无损压缩得到第二图像帧，所述第二图像帧包括n个经压缩后的片。
步骤S303：内存管理模块预先为所述第一图像帧分配虚拟内存空间，所述预先分配的虚拟内存空间包括多个页号连续的虚拟页面实现虚拟内存页号的连续，便于软件做管理和维护，所述虚拟内存空间的空间大小为K，0≤K≤所述第一图像帧的数据量。
例如解码时需要分配15个帧缓冲区，每个帧缓冲区占15MB空间，则在虚拟内存空间中足额分配15MB×15的空间。但对于这15个帧缓冲区中的每一个帧缓冲区，其对应的每个虚拟页面都可能都没有物理页面与其进行映射。在需要将图像帧的数据单元存入内存时，通过缺页处理的方式按照该数据单元的实际需求来分配实际的物理页面。
步骤S304：内存管理模块分别将所述第二图像帧中的每个经压缩后的片对应的一个或多个虚拟页面的页号送入SMMU中进行页表查询，并在查询不到与虚拟页面的页号对应的物理页面的页号时启动缺页处理，具体实施方式可参考图11实施例的描述，这里不再赘述。
步骤S305：若查询不到与所述一个或多个虚拟页面的页号对应的物理页面的页号，则内存管理模块为所述一个或多个虚拟页面分配对应的物理页面。
具体实施例中，若查询不到与所述一个或多个虚拟页面的页号对应的物理页面的页号，则在所述内存中寻找未被占用的一个或多个物理页面，该过程可包括：获取位图，所述位图包括多个位，每个位分别用于指示内存中的一个物理页面是否被占用，可根据位图寻找到未被占用的一个或多个物理页面，然后将所述一个或多个物理页面的页号增加到所述页表，以使得所述页表中所述一个或多个虚拟页面的页号中的每一个分别对应一个物理页面的页号，该物理页面的页面数不少于该虚拟页面的页面数。具体实施方式可参考图10实施例和图11实施例的描述，这里不再赘述。
步骤S306：内存管理模块分别将所述每个经压缩后的片存储到所述对应的物理页面。
步骤S307：内存管理模块释放所述帧缓冲区中未被所述多个经压缩后的片占用的虚拟内存空间。比如，如果为帧缓冲区分配了15MB的虚拟内存空间，但是经过无损压缩在帧 缓冲区依次紧凑地存入图像帧后，占用的实际物理内存空间为10MB，那么可释放无占用的5MB物理内存空间对应的虚拟页面，比如讲这5MB物理内存空间对应的虚拟页面的页号从页表中删除。
可以看到，本发明实施例通过改进页表的建立机制，实现了内存“按需占用”，与图13实施例的区别在于其不需要回收操作，图像帧经过无损压缩后，通过紧凑内存排布存入按需分配的物理内存空间，实现了不损失图像质量的前提下既节省带宽又节省帧存的目的。另外，内存管理模块23通过SMMU可以实现缺页时自动补充物理页面，使缺页处理过程对媒体框架透明，使得本发明实施例的技术方案可以适用于任何形态的媒体框架，有利于提升应用性和用户体验。
需要说明的是，上文中本发明实施例主要以无损压缩的应用场景为例来描述本发明的技术方案，但是，需要理解的是，本发明实施例的实际应用并不局限于无损压缩的应用场景，在其他可能的应用场景中(例如有损压缩)，也可能会应用本发明实施例的技术方案来实现内存的归集利用，从而降低解码时的参考帧存取对内存的占用，这里不再具体描述。
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者任意组合来实现。当使用软件实现时，可以全部或者部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机指令，在计算机上加载和执行所述计算机程序指令时，全部或部分地产生按照本发明实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络或其他可编程装置。所述计算机指令可存储在计算机可读存储媒介中，或者从一个计算机可读存储媒介向另一个计算机可读存储媒介传输，例如，所述计算机指令可以从一个网络站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线)或无线(例如红外、微波等)方式向另一个网络站点、计算机、服务器或数据中心进行传输。所述计算机可读存储媒介可以是计算机能够存取的任何可用媒介，也可以是包含一个或多个可用媒介集成的服务器、数据中心等数据存储设备。所述可用媒介可以是磁性媒介(例如软盘、硬盘、磁带等)、光媒介(例如DVD等)、或者半导体媒介(例如固态硬盘)等等。
在图2中的编码解码器54是一个独立的硬件装置，位于处理器56之外。可替换地，编码解码器54可以软件，即之前提到的计算机程序产品的形式存在，并运行于处理器56之上。此时编码解码器54是一个软件装置，包括软件代码指令。具体参见图3，视频解码器20可以以软件或硬件或其结合的方式实现，例如，视频解码器20内的任一个模块或单元可以由软件或硬件实现。在一个示例中，解码器21、内存管理模块23、压缩模块24和解压缩模块25均是硬件。在另一个示例中，解码器21、内存管理模块23、压缩模块24和解压缩模块25均是软件，此时他们全部是运行在处理器56上的软件代码形成的模块。在一个实例中，当某个模块以硬件实现，可包括大量晶体管、运算器件、或逻辑电路等，具体可以集成电路形式存在。
图3中的帧缓冲区22位于图2的存储器58中。存储器58可具体包括一个或多个存储器。例如，存储器58包括多个不同类型的存储器，包括易失性存储器和非易失性存储器。例如，一种非易失性存储器用于存储软件代码，由处理器56执行，并驱动处理器56的工作。一种易失性存储器可用于存储数据，例如短暂地临时性存储数据，形成之前实施例提 到的内存并包括所述帧缓冲区22。
在上述实施例中，对各个实施例的描述各有侧重，某个实施例中没有详述的部分，可以参见其他实施例的相关描述。

Claims (25)

  1. 一种视频解码中的内存管理方法，其特征在于，包括：
    对码流进行解码得到第一图像帧，所述第一图像帧包括n个片tile，每个片表示所述第一图像帧中的一个区域的图像，n是大于等于2的整数；
    对所述第一图像帧进行压缩得到第二图像帧，所述第二图像帧包括n个经压缩后的片；
    在内存中为所述n个经压缩后的片分配帧缓冲区frame buffer；
    为所述n个经压缩后的片在所述帧缓冲区中所占用的内存空间建立虚拟页面与物理页面的映射关系，其中，每个经压缩后的片所占用的内存空间包括一个或多个物理页面，在所述映射关系下所述n个经压缩后的片所占用的多个虚拟页面的页号连续；
    分别将所述n个经压缩后的片存储到所述帧缓冲区。
  2. 根据权利要求1所述的方法，其特征在于，所述在内存中为所述n个经压缩后的片分配帧缓冲区，包括：
    在内存中的内存块池中为所述n个经压缩后的片分配所述帧缓冲区，所述帧缓冲区包括M个内存块，每个内存块包括多个物理页面，所述多个物理页面对应于多个页号连续的虚拟页面，所述n个经压缩后的片所占用的内存空间小于等于所述M个内存块对应的内存空间；
    所述方法还包括：将所述帧缓冲区中未被占用的剩余内存块回收至所述内存块池，所述剩余内存块用于后续的帧缓冲区分配和片存储。
  3. 根据权利要求2所述的方法，其特征在于，所述为所述n个经压缩后的片在所述帧缓冲区中所占用的内存空间建立虚拟页面与物理页面的映射关系，包括：
    确定所述M个内存块中的各个内存块所对应的页表，每个内存块对应的页表包括该内存块中的各个虚拟页面与物理页面的映射关系；
    将所述M个内存块对应的页表组合成所述n个经压缩后的片所占用的内存空间的页表。
  4. 根据权利要求1所述的方法，其特征在于，
    所述在内存中为所述n个经压缩后的片分配所述帧缓冲区，包括：
    为所述n个经压缩后的片分配虚拟内存空间，所述虚拟内存空间包括多个页号连续的虚拟页面；所述n个经压缩后的片中的每个经压缩后的片对应于一个或多个虚拟页面；
    分别对所述一个或多个虚拟页面的页号进行页表page table查询；
    若查询到页表中不具有所述一个或多个虚拟页面的页号与物理页面的页号的映射关系，则在所述内存中为所述一个或多个虚拟页面分配所述一个或多个物理页面。
  5. 根据权利要求4所述的方法，其特征在于，在所述内存中所述为所述一个或多个虚拟页面分配所述一个或多个物理页面之前，还包括：
    获取位图bit map，所述位图包括多个位，各个位分别用于指示内存中的各个物理页面是否已被占用；
    根据所述位图寻找到内存中未被占用的所述一个或多个物理页面。
  6. 根据权利要求4或5所述的方法，其特征在于，所述为所述n个经压缩后的片在所述帧缓冲区中所占用的内存空间建立虚拟页面与物理页面的映射关系，包括：
    将所述一个或多个物理页面的页号添加至所述页表，以使得所述页表包括所述一个或多个虚拟页面与所述一个或多个物理页面之间的映射关系。
  7. 根据权利要求1-6任一项所述的方法，其特征在于，
    所述分别将所述n个经压缩后的片存储到所述帧缓冲区，包括：
    以多个经压缩后的片、多个压缩信息、和一个头地址信息组成一个数据单元的形式存储到所述帧缓冲区；每个经压缩后的片对应一个压缩信息，所述压缩信息用于指示对应的经压缩后的片所占用的内存空间大小；所述头地址信息用于指示所述数据单元的起始地址。
  8. 根据权利要求1-7任一项所述的方法，其特征在于，所述分别将所述n个经压缩后的片存储到所述帧缓冲区之后，所述方法还包括：
    从所述帧缓冲区分别读取所述n个经压缩后的片；
    对所述n个经压缩后的片做解压得到第一图像帧的解码帧；
    以所述解码帧作为参考帧进一步对所述码流进行解码得到第三图像帧。
  9. 一种内存管理的装置，其特征在于，所述装置包括解码模块、内存管理模块和压缩模块，其中：
    所述解码模块，用于对码流进行解码得到第一图像帧，所述第一图像帧包括n个片tile，每个片表示所述第一图像帧中的一个区域的图像，n是大于等于2的整数；
    所述压缩模块，用于对所述第一图像帧进行压缩得到第二图像帧，所述第二图像帧包括n个经压缩后的片；
    所述内存管理模块，用于在内存中为所述n个经压缩后的片分配帧缓冲区frame buffer；还用于为所述n个经压缩后的片在所述帧缓冲区中所占用的内存空间建立虚拟页面与物理页面的映射关系，其中，每个经压缩后的片所占用的内存空间包括一个或多个物理页面，在所述映射关系下所述n个经压缩后的片所占用的多个虚拟页面的页号连续；还用于分别将所述n个经压缩后的片存储到所述帧缓冲区。
  10. 根据权利要求9所述的装置，其特征在于，所述内存管理模块具体用于：在内存中的内存块池中为所述n个经压缩后的片分配所述帧缓冲区，所述帧缓冲区包括M个内存块，每个内存块包括多个物理页面，所述多个物理页面对应于多个页号连续的虚拟页面，所述n个经压缩后的片所占用的内存空间小于等于所述M个内存块对应的内存空间；
    所述内存管理模块还用于：将所述帧缓冲区中未被占用的剩余内存块回收至所述内存 块池，所述剩余内存块用于后续的帧缓冲区分配和片存储。
  11. 根据权利要求10所述的装置，其特征在于，所述内存管理模块具体用于：
    确定所述M个内存块中的各个内存块所对应的页表，每个内存块对应的页表包括该内存块中的各个虚拟页面与物理页面的映射关系；
    将所述M个内存块对应的页表组合成所述n个经压缩后的片所占用的内存空间的页表。
  12. 根据权利要求9所述的装置，其特征在于，所述内存管理模块具体用于：
    为所述n个经压缩后的片分配虚拟内存空间，所述虚拟内存空间包括多个页号连续的虚拟页面；所述n个经压缩后的片中的每个经压缩后的片对应于一个或多个虚拟页面；
    分别对所述一个或多个虚拟页面的页号进行页表page table查询；
    若查询到页表中不具有所述一个或多个虚拟页面的页号与物理页面的页号的映射关系，则在所述内存中为所述一个或多个虚拟页面分配所述一个或多个物理页面。
  13. 根据权利要求12所述的装置，其特征在于，在所述内存中所述为所述一个或多个虚拟页面分配所述一个或多个物理页面之前，所述内存管理模块还用于：
    获取位图bit map，所述位图包括多个位，各个位分别用于指示内存中的各个物理页面是否已被占用；
    根据所述位图寻找到内存中未被占用的所述一个或多个物理页面。
  14. 根据权利要求12或13所述的装置，其特征在于，所述内存管理模块具体用于：
    将所述一个或多个物理页面的页号添加至所述页表，以使得所述页表包括所述一个或多个虚拟页面与所述一个或多个物理页面之间的映射关系。
  15. 根据权利要求9-14任一项所述的装置，其特征在于，所述内存管理模块具体用于：
    以多个经压缩后的片、多个压缩信息、和一个头地址信息组成一个数据单元的形式存储到所述帧缓冲区；每个经压缩后的片对应一个压缩信息，所述压缩信息用于指示对应的经压缩后的片所占用的内存空间大小；所述头地址信息用于指示所述数据单元的起始地址。
  16. 根据权利要求9-15任一项所述的方法，其特征在于，所述装置还包括解压缩模块；
    所述内存管理模块用于，在分别将所述n个经压缩后的片存储到所述帧缓冲区之后，从所述帧缓冲区分别读取所述n个经压缩后的片；
    所述解压缩模块用于，对所述n个经压缩后的片做解压得到第一图像帧的解码帧；
    所述解码模块还用于，以所述解码帧作为参考帧进一步对所述码流进行解码得到第三图像帧。
  17. 一种内存管理的装置，其特征在于，所述装置包括处理器和存储器，其中：
    所述处理器用于，对码流进行解码得到第一图像帧，所述第一图像帧包括n个片tile，每个片表示所述第一图像帧中的一个区域的图像，n是大于等于2的整数；
    对所述第一图像帧进行压缩得到第二图像帧，所述第二图像帧包括n个经压缩后的片；
    在所述存储器中为所述n个经压缩后的片分配帧缓冲区frame buffer；还用于为所述n个经压缩后的片在所述帧缓冲区中所占用的内存空间建立虚拟页面与物理页面的映射关系，其中，每个经压缩后的片所占用的内存空间包括一个或多个物理页面，在所述映射关系下所述n个经压缩后的片所占用的多个虚拟页面的页号连续；还用于分别将所述n个经压缩后的片存储到所述帧缓冲区。
  18. 根据权利要求17所述的装置，其特征在于，所述处理器具体用于：在所述存储器中的内存块池中为所述n个经压缩后的片分配所述帧缓冲区，所述帧缓冲区包括M个内存块，每个内存块包括多个物理页面，所述多个物理页面对应于多个页号连续的虚拟页面，所述n个经压缩后的片所占用的内存空间小于等于所述M个内存块对应的内存空间；
    所述处理器还用于：将所述帧缓冲区中未被占用的剩余内存块回收至所述内存块池，所述剩余内存块用于后续的帧缓冲区分配和片存储。
  19. 根据权利要求18所述的装置，其特征在于，所述处理器具体用于：
    确定所述M个内存块中的各个内存块所对应的页表，每个内存块对应的页表包括该内存块中的各个虚拟页面与物理页面的映射关系；
    将所述M个内存块对应的页表组合成所述n个经压缩后的片所占用的内存空间的页表。
  20. 根据权利要求17所述的装置，其特征在于，所述处理器具体用于：
    为所述n个经压缩后的片分配虚拟内存空间，所述虚拟内存空间包括多个页号连续的虚拟页面；所述n个经压缩后的片中的每个经压缩后的片对应于一个或多个虚拟页面；
    分别对所述一个或多个虚拟页面的页号进行页表page table查询；
    若查询到页表中不具有所述一个或多个虚拟页面的页号与物理页面的页号的映射关系，则在所述存储器中为所述一个或多个虚拟页面分配所述一个或多个物理页面。
  21. 根据权利要求20所述的装置，其特征在于，在所述存储器中所述为所述一个或多个虚拟页面分配所述一个或多个物理页面之前，所述处理器还用于：
    获取位图bit map，所述位图包括多个位，各个位分别用于指示所述存储器中的各个物理页面是否已被占用；
    根据所述位图寻找到存储器中未被占用的所述一个或多个物理页面。
  22. 根据权利要求20或21所述的装置，其特征在于，所述处理器具体用于：
    将所述一个或多个物理页面的页号添加至所述页表，以使得所述页表包括所述一个或多个虚拟页面与所述一个或多个物理页面之间的映射关系。
  23. 根据权利要求17-22任一项所述的装置，其特征在于，所述处理器具体用于：
    以多个经压缩后的片、多个压缩信息、和一个头地址信息组成一个数据单元的形式存储到所述帧缓冲区；每个经压缩后的片对应一个压缩信息，所述压缩信息用于指示对应的经压缩后的片所占用的内存空间大小；所述头地址信息用于指示所述数据单元的起始地址。
  24. 根据权利要求17-23任一项所述的方法，其特征在于，所述处理器还用于：
    在分别将所述n个经压缩后的片存储到所述帧缓冲区之后，从所述帧缓冲区分别读取所述n个经压缩后的片；
    对所述n个经压缩后的片做解压得到第一图像帧的解码帧；
    以所述解码帧作为参考帧进一步对所述码流进行解码得到第三图像帧。
  25. 一种非易失的计算机可读存储媒介，其特征在于，所述计算机可读存储媒介用于存储程序代码，所述程序代码被计算设备执行时，所述计算设备用于执行权利要求1-8任一项所述的方法。
PCT/CN2019/077188 2018-06-28 2019-03-06 内存管理方法以及相关装置 WO2020001063A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810694790.6 2018-06-28
CN201810694790.6A CN110659225A (zh) 2018-06-28 2018-06-28 内存管理方法以及相关装置

Publications (1)

Publication Number Publication Date
WO2020001063A1 true WO2020001063A1 (zh) 2020-01-02

Family

ID=68984693

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/077188 WO2020001063A1 (zh) 2018-06-28 2019-03-06 内存管理方法以及相关装置

Country Status (2)

Country Link
CN (1) CN110659225A (zh)
WO (1) WO2020001063A1 (zh)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111787330B (zh) * 2020-06-16 2022-03-08 眸芯科技（上海）有限公司 支持解码压缩帧缓存自适应分配的编码方法及应用
CN113835872A (zh) * 2020-06-24 2021-12-24 北京小米移动软件有限公司 一种用于减少内存开销的数据处理方法、装置及存储介质
CN112738525B (zh) * 2020-12-11 2023-06-27 深圳万兴软件有限公司 视频处理方法、装置及计算机可读存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1424855A (zh) * 2001-12-13 2003-06-18 联发科技股份有限公司 视讯解码的存储器存取方法
CN1497449A (zh) * 2002-09-30 2004-05-19 国际商业机器公司 连续物理存储器分配的方法和系统
US20050265459A1 (en) * 2004-06-01 2005-12-01 Bhattacharjya Anoop K Fixed budget frame buffer compression using block-adaptive spatio-temporal dispersed dither
CN102243615A (zh) * 2011-03-21 2011-11-16 广东志成冠军集团有限公司 一种支持音视频帧存储的内存缓冲管理机制
CN106162192A (zh) * 2016-07-06 2016-11-23 华为技术有限公司 处理视频数据的方法和装置

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8477852B2 (en) * 2007-06-20 2013-07-02 Nvidia Corporation Uniform video decoding and display
US8824560B2 (en) * 2010-07-07 2014-09-02 Netzyn, Inc. Virtual frame buffer system and method
EP3204859B1 (en) * 2014-10-07 2020-12-02 Google LLC Methods and systems for cache lines de-duplication
US9946660B2 (en) * 2016-07-29 2018-04-17 Hewlett Packard Enterprise Development Lp Memory space management
CN107995494B (zh) * 2017-12-12 2019-11-22 珠海全志科技股份有限公司 视频图像数据的压缩方法与解压方法、计算机装置、计算机可读存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1424855A (zh) * 2001-12-13 2003-06-18 联发科技股份有限公司 视讯解码的存储器存取方法
CN1497449A (zh) * 2002-09-30 2004-05-19 国际商业机器公司 连续物理存储器分配的方法和系统
US20050265459A1 (en) * 2004-06-01 2005-12-01 Bhattacharjya Anoop K Fixed budget frame buffer compression using block-adaptive spatio-temporal dispersed dither
CN102243615A (zh) * 2011-03-21 2011-11-16 广东志成冠军集团有限公司 一种支持音视频帧存储的内存缓冲管理机制
CN106162192A (zh) * 2016-07-06 2016-11-23 华为技术有限公司 处理视频数据的方法和装置

Also Published As

Publication number Publication date
CN110659225A (zh) 2020-01-07

Similar Documents

Publication Publication Date Title
WO2020001063A1 (zh) 内存管理方法以及相关装置
US20140086309A1 (en) Method and device for encoding and decoding an image
US8477852B2 (en) Uniform video decoding and display
KR102543444B1 (ko) 영상 부호화 장치
US10509681B2 (en) Electronic apparatus for effective resource management and method thereof
EP3849188A1 (en) Point cloud coding method and encoder
US9335946B2 (en) Method and apparatus for managing memory
US20140219361A1 (en) Image data encoding for access by raster and by macroblock
KR20130085389A (ko) 서브영역별로 엔트로피 부호화의 병렬 처리가 가능한 비디오 부호화 방법 및 장치, 서브영역별로 엔트로피 복호화의 병렬 처리가 가능한 비디오 복호화 방법 및 장치
US11483561B2 (en) Transform method in picture block encoding, inverse transform method in picture block decoding, and apparatus
RU2656727C1 (ru) Поверхности управления сжатием, поддерживаемые виртуальной памятью
CN106162192A (zh) 处理视频数据的方法和装置
CN103686077A (zh) 应用于3g无线网络实时音视频数据传输的双缓冲方法
US10694178B2 (en) Video processing device for encoding frames constituting hierarchical structure
US20220007037A1 (en) Point cloud encoding method and apparatus, point cloud decoding method and apparatus, and storage medium
US10674160B2 (en) Parallel video encoding device and encoder configured to operate in parallel with another encoder
US11470321B2 (en) Processing media using neural networks
CN113301290B (zh) 视频数据处理方法及视频会议终端
US11030480B2 (en) Electronic device for high-speed compression processing of feature map of CNN utilizing system and controlling method thereof
CN116700943A (zh) 一种视频播放系统、方法以及电子设备
CN101847394B (zh) 用以解码及显示视讯文件的存储器映射方法及装置
US20080189491A1 (en) Fusion memory device and method
JP2003177958A (ja) 特殊メモリデバイス
TWI512675B (zh) 影像處理裝置與方法
TWI565303B (zh) 影像處理系統及影像處理方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19825853

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19825853

Country of ref document: EP

Kind code of ref document: A1