WO2023078228A1 - 媒体文件内存管理方法、装置、计算机设备和可读介质 - Google Patents

媒体文件内存管理方法、装置、计算机设备和可读介质 Download PDF

Info

Publication number
WO2023078228A1
WO2023078228A1 PCT/CN2022/128907 CN2022128907W WO2023078228A1 WO 2023078228 A1 WO2023078228 A1 WO 2023078228A1 CN 2022128907 W CN2022128907 W CN 2022128907W WO 2023078228 A1 WO2023078228 A1 WO 2023078228A1
Authority
WO
WIPO (PCT)
Prior art keywords
memory
memory page
media file
pointer
media
Prior art date
Application number
PCT/CN2022/128907
Other languages
English (en)
French (fr)
Inventor
骆俊
王日红
张晓渠
Original Assignee
中兴通讯股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中兴通讯股份有限公司 filed Critical 中兴通讯股份有限公司
Publication of WO2023078228A1 publication Critical patent/WO2023078228A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02—Addressing or allocation; Relocation
    • G06F12/06—Addressing a physical block of locations, e.g. base addressing, module addressing, memory dedication
    • G06F12/0646—Configuration or reconfiguration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02—Addressing or allocation; Relocation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02—Addressing or allocation; Relocation
    • G06F12/0223—User address space allocation, e.g. contiguous or non contiguous base addressing
    • G06F12/023—Free address space management
    • G06F12/0253—Garbage collection, i.e. reclamation of unreferenced memory
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02—Addressing or allocation; Relocation
    • G06F12/06—Addressing a physical block of locations, e.g. base addressing, module addressing, memory dedication
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
    • G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Definitions

  • Embodiments of the present disclosure relate to but are not limited to the field of computer technology, and specifically relate to a media file memory management method, device, computer equipment and readable medium.
  • CDN Content Delivery Network, Content Distribution Network
  • CDN can send video resources from the source station to the nearest service node around the customer, shortening the physical service distance, so that customers can access video resources faster.
  • a traditional CDN server supports high-traffic media services
  • its memory management adopts the LRU (Least recently used) structure, and its memory management structure is presented as a linked list.
  • the CDN server obtains the media data, it reads the media data into the LRU linked list.
  • the CDN server starts to reclaim memory from the end of the LRU linked list, thereby realizing memory management.
  • the media files are generally large, about several gigabytes, causing the LRU list to be too long.
  • the disclosure provides a media file memory management method, device, computer equipment and readable medium.
  • an embodiment of the present disclosure provides a method for media file memory management, the method comprising: in response to obtaining a corresponding media file according to a media file playback request, generating a ring-shaped memory page structure for the media file; wherein , the memory page structure includes a push pointer, a recovery pointer and a plurality of memory pages, the push pointer and the recovery pointer point to different memory pages, and the memory page is mounted with the media data of the media file, each The memory page forms a ring and can flow according to a preset flow direction; and during the flow of the memory page, push the media data mounted on the memory page pointed to by the push pointer, and send the media data to the memory page pointed to by the recovery pointer The mounted media data is replaced in the memory page with unmounted media data in the media file until the last media data in the media file is mounted in the memory structure.
  • an embodiment of the present disclosure also provides a media file memory management device, including a memory page structure generation module, a media data push module, and a memory page recycling module, wherein the memory page structure generation module is configured to respond to Obtain the corresponding media file according to the media file playback request, and generate a ring-shaped memory page structure for the media file; wherein, the memory page structure includes a push pointer, a recovery pointer and a plurality of memory pages, and the push pointer and the memory page structure The recovery pointers point to different memory pages, the memory pages are mounted with the media data of the media files, and each of the memory pages forms a ring and can flow according to a preset flow direction; the media data push module is configured , during the memory page flow process, push the media data mounted on the memory page pointed to by the push pointer; and the memory page recycling module is configured to, during the memory page flow process, In the memory page pointed to by the reclamation pointer, the mounted media data is replaced with unmounted media data in the media file until the
  • an embodiment of the present disclosure further provides a computer device, including: at least one processor; a storage device, on which at least one program is stored; when the at least one program is executed by the at least one processor, the The at least one processor implements the media file memory management method as described above.
  • an embodiment of the present disclosure further provides a computer-readable medium on which a computer program is stored, wherein, when the program is executed, the above-mentioned method for managing media file memory is implemented.
  • FIG. 1 is a schematic diagram of an operating system architecture of an embodiment of the present disclosure
  • FIG. 2 is a schematic flow diagram of a method for managing media file memory provided by an embodiment of the present disclosure
  • FIG. 3 is a schematic diagram of a circular memory page structure provided by an embodiment of the present disclosure.
  • FIG. 4 is a schematic flowchart of a method for managing media file memory provided by another embodiment of the present disclosure.
  • FIG. 5 is a schematic flowchart of generating a ring-shaped memory page structure provided by an embodiment of the present disclosure
  • FIG. 6 is a schematic flowchart of a method for managing media file memory provided by another embodiment of the present disclosure.
  • FIG. 7 is a schematic structural diagram of a media file memory management device provided by an embodiment of the present disclosure.
  • FIG. 8 is a schematic structural diagram of a media file memory management device provided by another embodiment of the present disclosure.
  • Fig. 9 is a schematic structural diagram of a media file memory management device provided by another embodiment of the present disclosure.
  • Embodiments described herein may be described with reference to plan views and/or cross-sectional views by way of idealized schematic illustrations of the present disclosure. Accordingly, the example illustrations may be modified according to manufacturing techniques and/or tolerances. Therefore, the embodiments are not limited to the ones shown in the drawings but include modifications of configurations formed based on manufacturing processes. Accordingly, the regions illustrated in the figures have schematic properties, and the shapes of the regions shown in the figures illustrate the specific shapes of the regions of the elements, but are not intended to be limiting.
  • An embodiment of the present disclosure provides a media file memory management method.
  • the media file memory management method is applied to a media file memory management device.
  • the media file memory management device can be a module in a CDN server, and the CDN server runs a Linux/Unix operating system. After the media file memory management device is loaded into the operating system, the memory management service can be provided for the media service.
  • the embodiment of the disclosure supports xfs (log file system), ext (Extended file system, extended file system) and fuse (Filesystem in Userspace, user space file system), and the service of the application layer has no influence on the content of the present invention.
  • Fig. 1 is the schematic diagram of the operating system architecture of the embodiment of the present disclosure, as shown in Fig. 1, the media file memory management device belongs to the Linux/Unix class kernel module in the operating system, runs in the Linux/Unix class kernel space, and is responsible for assisting the operating system to Memory page management improves system performance.
  • the media file memory management device runs on the xfs file system and under the virtual file system vfs, and realizes the media file memory management function by replacing some functions in the sendfile (zero copy) process of xfs.
  • the media file memory management method includes the following steps:
  • Step 21 in response to obtaining the corresponding media file according to the media file playback request, generate a ring-shaped memory page structure for the media file; wherein, the memory page structure includes a push pointer, a recovery pointer and a plurality of memory pages, and the push pointer and The recovery pointers point to different memory pages, and the media data of the media files are mounted on the memory pages, and each memory page forms a ring and can flow according to a preset flow direction.
  • the client sends a media file play request to the CDN server, and the CDN server obtains the corresponding media file according to the media file play request.
  • the media file memory management device After acquiring the media file, the media file memory management device generates a circular memory page structure for the media file.
  • FIG. 3 is a schematic diagram of a ring-shaped memory page structure provided by an embodiment of the present disclosure.
  • the ring-shaped memory page structure includes multiple memory pages, and different memory pages mount different media data in media files. Each memory page is connected end to end to form a ring.
  • the memory page structure also includes a push pointer and a recovery pointer, and the push pointer and the recovery pointer point to different memory pages.
  • the media data mounted on the memory page pointed to by the push pointer is the media data pushed to the client, and is also the media data that the user is watching.
  • the memory page pointed to by the recycling pointer is the currently recycled memory page, and the memory page can be recycled and reused by replacing the media data mounted on the memory page.
  • the memory pages in the memory page structure can circulate to form a data flow, and the flow direction (as shown by the arrow in Figure 3) can be preset. Once the ring-shaped memory page structure is generated, the flow direction of the memory page is fixed.
  • Step 22 during the memory page flow process, push the media data mounted on the memory page pointed to by the push pointer, and replace the mounted media data with the unmounted media data in the memory page pointed to by the recovery pointer.
  • the loaded media data until the last media data in the media file is mounted in the memory structure.
  • the media data mounted on the corresponding memory page is pushed to the client that initiates the media file playback request to realize push streaming service, so that the user can watch the media data; since the user watches the media file from front to back, the embodiment of the present disclosure regards the playback of a media file as a data stream flowing from front to back.
  • the streaming service is to send each media data of the media file to the user in sequence from front to back.
  • the corresponding memory page is recycled and reused, that is, the media data mounted in the memory page is deleted (the deleted media data is the media data that has been pushed), and the unused The mounted media data is mounted on this memory page.
  • the pushed memory page After the media data mounted on the memory page pointed to by the push pointer is pushed, the pushed memory page will flow according to the flow direction of the memory page.
  • the media data that has not been mounted in the media file replaces the media data that has been pushed in the memory page that has been pushed, so as to realize the recycling and reuse of the memory page, and the memory page that remounts the new media data continues to follow the memory page
  • the flow direction of the page flows, and when the flow reaches the position of the push pointer, the media data mounted on it is pushed, and this cycle goes back and forth, thereby forming a circular data flow.
  • the media file memory management method includes: in response to obtaining the corresponding media file according to the media file playback request, generating a ring-shaped memory page structure for the media file;
  • the memory page structure includes a push pointer, Recycling pointers and multiple memory pages, push pointers and recycling pointers point to different memory pages, the memory pages are loaded with media data of media files, each memory page forms a ring and can flow according to the preset flow direction;
  • the memory page flows In the process, push the media data mounted on the memory page pointed to by the push pointer, and replace the mounted media data with the unmounted media data in the media file in the memory page pointed to by the recovery pointer until the The last media data in the media file is mounted in the memory structure;
  • the embodiment of the present disclosure can improve the memory usage efficiency of the CDN server in the high-bandwidth and high-traffic scenario, greatly reduce the consumption of the operating system and the time delay caused by memory reclamation, and utilize 2 -5M of memory can provide several gigabytes of media file play
  • the media file service has a characteristic.
  • these memory pages can be directly used to mount the media data that the user is about to watch, which can bypass the complicated memory management process in the operating system, and can greatly reduce the time spent by media services in obtaining memory pages, thereby Improve the performance of media file serving.
  • embodiments of the present disclosure provide a ring-shaped memory page structure, which can carry a streaming service of a media file. After that, the media data that has not been pushed can be mounted again. Through the flow of media data stream and memory page, smaller memory can be used to realize the playback of larger media files.
  • the LRU memory structure has a relatively long linked list, and the operating system takes a long time to traverse the linked list, resulting in high CPU usage and high memory usage.
  • a memory page structure is generated for a media file to mount the media data stream.
  • the memory page required by the circular media data stream is smaller, only about 2-5M.
  • the embodiment of the present disclosure considers that the media data that the user has watched will not be watched again with a high probability, and a ring-shaped memory page structure is designed to form a ring-shaped media data stream, which can quickly mount the media data that has been pushed.
  • the data memory page quickly mounts the media data to be pushed, so only about 2-5M of memory can be used to play several gigabytes of media files, which greatly reduces the memory usage and the consumption of the operating system and CPU.
  • the media file memory management method further includes the following step: during the process of generating the ring-shaped memory page structure for the media file, setting the position of the recycling pointer according to the size of the free memory of the operating system.
  • the position of the push pointer is the position of the memory page to which the media data being pushed belongs (that is, the position of the memory page to which the media data that the user is watching) belongs, and the position of the recovery pointer is the position of the memory page that starts recycling.
  • the memory page structure includes a first area A and a second area B.
  • the flow direction is from the recovery pointer to the push pointer.
  • area B the flow direction is from the push pointer to the recycling pointer.
  • the media data mounted on the memory page in the first area A is the media data to be pushed
  • the media data mounted on the memory page in the second area B is the media data that has been pushed.
  • the position of the recycling pointer can be relative to the pushed Adjust the position of the pointer.
  • setting the position of the recycling pointer according to the size of the free memory of the operating system includes the following steps: setting the recycling pointer in the first area in response to the free memory of the operating system being greater than a preset first threshold; If the free memory is less than or equal to the preset first threshold, the recovery pointer is set in the second area.
  • the recycling pointer when the free memory of the operating system is large, the recycling pointer is set relatively far away from the push pointer along the flow direction, so that there are more memory pages that have pushed media data mounted between the recycling pointer and the pushing pointer (that is, the first The second area B is larger than the first area A), and these memory pages will be loaded with new media data after a long time, thereby reducing the number of memory page recycling and reuse as a whole and improving the performance of the CDN server.
  • the recycling pointer is set relatively close to the pushing pointer along the flow direction, so that there are fewer memory pages to mount the pushed media data between the recycling pointer and the pushing pointer (that is, the second area B is smaller than In the first area A), these memory pages will mount new media data in a short time, thereby increasing the number of memory page recycling and reuse as a whole.
  • CDN Server performance will be reduced.
  • the operating system performs memory management based on memory pages, and does not particularly distinguish the types and uses of data mounted on the memory pages. This is because the operating system's memory management is not user- or service-specific. Therefore, the memory management of the operating system must be universal in order to adapt to various complex memory management scenarios.
  • this general memory management method will show insufficient performance when dealing with media services with large traffic and large files. This is because the memory pages for mounting media files are distributed on the LRU list, which will cause the LRU list to be very long. When reclaiming memory pages, it is necessary to traverse the LRU linked list, resulting in more CPU time-consuming, resulting in insufficient performance of the CDN server.
  • the embodiment of the present disclosure provides a separate memory management service for a large-flow media service by establishing a memory pool.
  • the media file memory management method may also include the following steps:
  • Step 20 for the media file, apply for the first memory page from the operating system, and establish a memory pool according to the first memory page.
  • this step can be performed in the initialization phase, that is, a memory pool of media files is established in advance.
  • the following initialization operations can also be performed: determine the size of the reserved memory according to the current performance of the CDN server, replace some functions such as sendfile, generate a debugging interface file in the proc directory of the operating system, initialize the flow management, initialize the memory pool, Create memory management and stream management kernel threads, etc.
  • the memory pool After completing the initialization of the memory pool, apply for a certain size of memory (ie the first memory page) to the operating system. After the memory application is successful, the memory pool is formally formed, and the memory management of the media file is started.
  • said generating the ring-shaped memory page structure for the media file includes: generating the ring-shaped memory page structure for the media file at least according to the first memory page in the memory pool. That is to say, the memory pages in the memory page structure are preferentially obtained from the memory pool of the corresponding media file.
  • the embodiment of the present disclosure adopts the memory pool technology, and pre-applies to the operating system for a certain number of first memory pages to form a memory pool, and can obtain independent memory from the operating system of the CDN server to provide a separate service for the delivery of media files.
  • a memory pool is established for each media file, and the memory pages required to generate a ring-shaped memory page structure are preferentially obtained from the memory pool. In this way, the memory pages of the media file can be collectively managed together, instead of being dispersed and managed by the operating system.
  • these memory pages can be quickly operated without being distributed to the operating system and processed by the memory management device of the operating system, which avoids complicated memory management processes and does not need to frequently apply for memory pages from the operating system, which can reduce Time-consuming memory management, thereby improving the performance of media file memory services.
  • said at least generating a ring-shaped memory page structure for the media file according to the first memory page in the memory pool includes the following steps:
  • Step 211 in response to insufficient memory resources in the memory pool, apply for a second memory page from the operating system.
  • Step 212 Generate a circular memory page structure for the media file according to the first memory page and the second memory page in the memory pool.
  • the amount of memory allocated to the media file is not fixed. During the process of memory management, if the memory resource in the memory pool is insufficient, the memory can be allocated again by the system.
  • the media file memory management method may also include the following steps: not recording the occupation of the first memory page.
  • the occupation of the first memory page means that once the first memory page is successfully applied to the operating system, the first memory page will be occupied to establish a memory pool, and the first memory page allocated by the operating system for the media file memory service will be immediately occupy.
  • the operating system will record the occupied memory pages, that is, add 1 to the count of the occupied memory pages, and the memory recycling program kwsap of the operating system will only recycle the memory pages whose count is 0.
  • the occupation of the first memory page is actually realized, and the occupied first memory page does not increase.
  • count count that is, the count count of the first memory page will not be incremented by 1. Therefore, the first memory page is completely transparent to the operating system, and the operating system will not perceive the occupation of the first memory page at all, will not affect the normal recovery of memory by the operating system through kswap, and will not affect the performance of the operating system.
  • the operating system when the operating system runs out of memory resources and triggers kswap to recycle memory resources, it can recycle the first memory page in the memory pool for use by other programs that need memory resources, thereby improving multi-service co-creation and complex scenarios.
  • the performance stability of the next CDN server The embodiments of the present disclosure can dynamically adjust the memory size of the managed media files, enhance the flexibility of media file memory management, and solve the problem of high memory usage of the CDN server without causing a major impact on the operating system.
  • described media file memory management method can also comprise the following steps:
  • Step 21' releasing the first memory page in the memory pool to the operating system in response to the satisfaction of the preset memory page recycling condition.
  • the embodiments of the present disclosure can actively release memory resources to the operating system, so as to avoid excessive memory usage of the CDN server caused by excessive memory resource usage.
  • the embodiments of the present disclosure can support xfs, ext, and fuse file systems at the same time, and can flexibly set memory page recycling conditions for different types of file systems.
  • the ext and fuse file systems are systems used to manage files in the operating system.
  • the fuse file system refers to the file system that is completely implemented in the user mode, which avoids the overhead of switching the user mode and the system mode back and forth on the server.
  • the ext file system is Linux /Unix-like standard extended file system.
  • the memory page recycling condition includes at least one of the following: a first recycling condition for a memory page corresponding to an xfs file system, a second recycling condition for a memory page corresponding to an ext file system, and a memory page corresponding to a fuse file system
  • the third recovery condition correspondsly, the satisfaction of the preset memory page recycling condition includes: at least one of the first memory page recycling condition, the memory page second recycling condition, and the memory page third recycling condition is met.
  • the embodiments of the present disclosure can actively release memory to the operating system according to preset memory page recycling conditions, reduce the memory usage of the CDN server, and thereby improve the overall performance of the CDN server.
  • the first recovery condition of the memory page corresponding to the xfs file system may include: the remaining memory of the operating system is greater than the preset minimum memory and less than the user-defined memory waterline value; the memory corresponding to the ext file system
  • the second recycling condition of the page may include: the memory recycling time set periodically arrives.
  • K can reclaim the memory page at 4 o'clock in the morning every day;
  • the third recycling condition of the memory page corresponding to the fuse file system may include: The media data has not been viewed for more than a preset period of time. For example, for the fuse file system, memory pages that have not been viewed for one hour are reclaimed.
  • the embodiments of the present disclosure set up a memory page structure recovery mechanism, that is, recover all memory pages in the memory page structure. Therefore, in some embodiments, the media file memory management method may further include the following step: releasing the memory pages in the memory page structure to the memory pool in response to the completion of pushing all the media data in the media file. In this step, the memory page structure may be periodically scanned to determine whether all media data in the corresponding media file has been pushed. If all the media data has been pushed, each memory page in the memory page structure may be recycled to the memory pool.
  • the CDN server can provide multiple media files and generate a ring-shaped memory page structure for each media file.
  • the embodiment of the present disclosure can periodically scan each memory page structure to realize the monitoring and monitoring of the memory page structure. Recycle.
  • the CDN server in the embodiment of the present disclosure can be applicable to RTSP (Real Time Streaming Protocol, real-time streaming protocol) on-demand business scenarios, HLS (HTTP Live Streaming, HTTP-based adaptive code rate streaming media transmission protocol) on-demand business scenarios, DASH (Dynamic Adaptive Streaming over HTTP, dynamic adaptive streaming based on HTTP) on-demand business scenarios, HPD (http Progressive Download, progressive download) on-demand business scenarios.
  • RTSP Real Time Streaming Protocol, real-time streaming protocol
  • HLS HTTP Live Streaming, HTTP-based adaptive code rate streaming media transmission protocol
  • DASH Dynamic Adaptive Streaming over HTTP, dynamic adaptive streaming based on HTTP
  • HPD http Progressive Download, progressive download
  • the media file memory management method provided by the embodiment.
  • the application scenarios of the embodiments of the present disclosure are not limited to the above scenarios, and all application scenarios related to the media contract delivery mechanism of the CDN server are applicable.
  • the embodiment of the present disclosure provides a solution for memory management of media files, which adopts the idea of memory pool and data flow to independently manage the memory of media files.
  • independent memory can be obtained from the operating system of the CDN server to provide a separate service for the delivery of media files.
  • the memory pages used to mount data streams can be obtained from the memory pool. If the memory resources in the memory pool are insufficient, you can apply to the operating system. In addition, the memory resources in the memory pool can also be released to the operating system to avoid Occupying too many memory resources causes the memory usage of the CDN server to be too high.
  • a ring-shaped memory page structure can be provided for each media file.
  • the memory page of the watched media data can be directly used as the memory page to mount the media data to be played.
  • Form a ring-shaped flowable media data stream In this way, files with a size of several gigabytes can only be played with a very small memory of 2-5M, which can reduce the consumption of operating system memory management and CPU usage in high-traffic scenarios, thereby improving the overall performance of the CDN server.
  • the media file memory management method in the embodiment of the present disclosure realizes the performance improvement of the CDN server by independently maintaining the memory of the media file, adopting a memory pool, a ring-shaped memory page structure and a data flow mechanism.
  • a circular memory page structure is generated for the requested media file, and the media data is mounted on the memory page in the memory page structure, which is formed by the circular flow of the memory page data flow.
  • the memory pages in the memory pool can be obtained from the operating system, or they can be actively released and recycled to the operating system.
  • the memory pages applied to the operating system will no longer increase the count count when they enter the memory pool queue, so that the kswap of the operating system can normally reclaim these memory pages, that is to say, the media file memory management device in the embodiment of the present disclosure no longer reserves Memory, if the operating system needs to reclaim memory, it can be reclaimed through kwap normally.
  • the implementation basis of the embodiment scheme of the present disclosure is to replace some functions in the sendfile process of the xfs file system.
  • the operation set and functions of the xfs file system can be replaced, so all memory pages requested by calling the sendfile service are taken from the memory pool. Obtained, so as to realize the management function of the memory page of the media file.
  • the embodiment of the present disclosure further supports ext and fuse file systems, and can actively release memory to the operating system.
  • the embodiment of the disclosure also provides a media file memory management device, as shown in Figure 7, the media file memory management device includes a memory page structure generation module 101, a media data push module 102 and a memory page recycling Module 103.
  • the memory page structure generating module 101 is configured to, in response to obtaining the corresponding media file according to the media file play request, generate a ring-shaped memory page structure for the media file; wherein, the memory page structure includes a push pointer and a recovery pointer and a plurality of memory pages, the push pointer and the recovery pointer point to different memory pages, the memory pages are mounted with the media data of the media file, and each of the memory pages forms a ring and can be configured according to a preset flow direction flow.
  • the media data pushing module 102 is configured to, during the memory page flow process, push the media data mounted on the memory page pointed to by the pushing pointer.
  • the memory page recycling module 103 is configured to, during the memory page flow process, replace the mounted media data in the memory page pointed to by the recycling pointer with unmounted media data in the media file, Until the last media data in the media file is mounted in the memory structure.
  • the memory page structure generating module 101 is further configured to, during the process of generating the ring-shaped memory page structure for the media file, set the position of the recovery pointer according to the size of the free memory of the operating system.
  • the memory page structure includes a first area and a second area, in the first area, the flow direction is from the recovery pointer to the push pointer, in the second area In the second area, the flow direction is the direction from the pushing pointer to the recycling pointer.
  • the memory page structure generating module 101 is further configured to, in response to the free memory of the operating system being greater than a preset first threshold, set the recovery pointer in the first area; in response to the free memory of the operating system being less than or equal to A first threshold is preset, and the recovery pointer is set in the second area.
  • the media file memory management device further includes a memory pool management module 104, and the memory pool management module 104 is configured to generate a ring for the media file in the memory page structure generation module 101.
  • the memory pool management module 104 is configured to generate a ring for the media file in the memory page structure generation module 101.
  • the memory page structure for the media file, apply for a first memory page from the operating system, and establish a memory pool according to the first memory page.
  • the memory page structure generation module 101 is configured to generate a circular memory page structure for the media file at least according to the first memory page in the memory pool.
  • the memory page structure generation module 101 is configured to, in response to insufficient memory resources in the memory pool, apply for a second memory page from the operating system; according to the first memory page in the memory pool and The second memory page generates a circular memory page structure for the media file.
  • the memory pool management module 104 is further configured to not record the occupancy of the first memory page after the memory pool is established according to the first memory page.
  • the memory pool management module 104 is further configured to: after the memory pool is established according to the first memory page, in response to a preset memory page recovery condition being met, the memory page in the memory pool A page of memory is released to the operating system.
  • the memory page recycling conditions include at least one of the following: the first memory page recycling condition corresponding to the xfs file system, the second memory page recycling condition corresponding to the ext file system, and the memory page corresponding to the fuse file system The third recycling condition of the memory page;
  • Satisfying the preset memory page recovery condition includes: at least one of the memory page first recovery condition, the memory page second recovery condition and the memory page third memory page recovery condition is satisfied.
  • the media file memory management device further includes a stream management module 105, and the stream management module 105 is configured to, in response to the completion of pushing all media data in the media file, push all the media data The memory pages in the memory page structure are released to the memory pool.
  • An embodiment of the present disclosure also provides a computer device, the computer device includes: one or more processors and a storage device; wherein, one or more programs are stored on the storage device, when the one or more programs are executed by the one or more When one or more processors execute, the one or more processors implement the media file memory management method provided in the foregoing embodiments.
  • An embodiment of the present disclosure also provides a computer-readable medium on which a computer program is stored, wherein when the computer program is executed, the method for managing the memory of a media file as provided in the foregoing embodiments is implemented.
  • the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be composed of several physical components. Components cooperate to execute.
  • Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit .
  • Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media).
  • computer storage media includes both volatile and nonvolatile media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. permanent, removable and non-removable media.
  • Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cartridges, tape, magnetic disk storage or other magnetic storage devices, or can Any other medium used to store desired information and which can be accessed by a computer.
  • communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media .

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

本公开提供一种媒体文件内存管理方法，所述方法包括：响应于根据媒体文件播放请求获取到对应的媒体文件，为媒体文件生成环状的内存页结构；该内存页结构包括推送指针、回收指针和多个内存页面，推送指针和回收指针指向不同的内存页面，内存页面挂载有媒体文件的媒体数据，各内存页面形成环状且能够按照预设的流动方向流动；在内存页面流动过程中，推送所述推送指针指向的内存页面所挂载的媒体数据，并在回收指针指向的内存页面中将所挂载的媒体数据替换为媒体文件中未被挂载的媒体数据，直至将媒体文件中最后的媒体数据挂载在内存结构中；可以提升解决CDN服务器性能不足的问题。本公开还提供一种媒体文件内存管理装置、计算机设备和可读介质。

Description

媒体文件内存管理方法、装置、计算机设备和可读介质
相关公开的交叉引用
本公开要求在2021年11月4日提交国家知识产权局、公开号为CN202111300555.4、发明名称为“媒体文件内存管理方法、装置、计算机设备和可读介质”的中国专利公开的优先权，这些公开的全部内容通过引用结合在本公开中。
技术领域
本公开的实施例涉及但不限于计算机技术领域，具体涉及一种媒体文件内存管理方法、装置、计算机设备和可读介质。
背景技术
随着互联网的快速发展，人们在视频方面花费的时间越来越多，短视频的兴起改变了人们的生活和娱乐方式。随着媒体服务的带宽越来越大，为了避免观众观看视频时候产生卡顿现象，越来越多的视频服务厂商开通了CDN(Content Delivery Network，内容分发网络)加速服务。
CDN能够将视频资源从源站分级发送到客户周围最近的服务节点，缩短了物理上的服务距离，从而使得客户能够更快的访问到视频资源。传统的CDN服务器支持大流量的媒体服务的时候，其内存管理采用LRU(Least recently used，最近最少使用)结构，其内存管理结构呈现为一个链表。CDN服务器获取媒体数据之后，将媒体数据读入到该LRU链表中，对于长时间没有访问过的媒体数据，CDN服务器从该LRU链表的末尾开始回收内存，从而实现内存的管理。
对于承载大流量媒体服务的CDN服务器来说，媒体文件一般较大，为数G左右，造成LRU链表过长，CDN服务器遍历LRU链表耗时较大， 造成CDN服务器性能不足的问题，以及内存回收不及时引起的CPU冲高、内存使用率过高的问题。
发明内容
本公开提供一种媒体文件内存管理方法、装置、计算机设备和可读介质。
第一方面，本公开实施例提供一种媒体文件内存管理方法，所述方法包括：响应于根据媒体文件播放请求获取到对应的媒体文件，为所述媒体文件生成环状的内存页结构；其中，所述内存页结构包括推送指针、回收指针和多个内存页面，所述推送指针和所述回收指针指向不同的内存页面，所述内存页面挂载有所述媒体文件的媒体数据，各所述内存页面形成环状且能够按照预设的流动方向流动；以及在所述内存页面流动过程中，推送所述推送指针指向的内存页面所挂载的媒体数据，并在所述回收指针指向的内存页面中将所挂载的媒体数据替换为所述媒体文件中未被挂载的媒体数据，直至将所述媒体文件中最后的媒体数据挂载在所述内存结构中。
又一方面，本公开实施例还提供一种媒体文件内存管理装置，包括内存页结构生成模块、媒体数据推送模块和内存页面回收模块，其中，所述内存页结构生成模块被配置成，响应于根据媒体文件播放请求获取到对应的媒体文件，为所述媒体文件生成环状的内存页结构；其中，所述内存页结构包括推送指针、回收指针和多个内存页面，所述推送指针和所述回收指针指向不同的内存页面，所述内存页面挂载有所述媒体文件的媒体数据，各所述内存页面形成环状且能够按照预设的流动方向流动；所述媒体数据推送模被配置成，在所述内存页面流动过程中，推送所述推送指针指向的内存页面所挂载的媒体数据；并且所述内存页面回收模块被配置成，在所述内存页面流动过程中，在所述回收指针指向的内存页面中将所挂载的媒体数据替换为所述媒体文件中未被挂载的媒体数据，直至将所述媒体文件中最后的媒体数据挂载在所述内存结构中。
又一方面，本公开实施例还提供一种计算机设备，包括：至少一个处理器；存储装置，其上存储有至少一个程序；当所述至少一个程序被所述至少一个处理器执行时，使得所述至少一个处理器实现如前所述的媒体文件内存管理方法。
又一方面，本公开实施例还提供一种计算机可读介质，其上存储有计算机程序，其中，所述程序被执行时实现如前所述的媒体文件内存管理方法。
附图说明
图1为本公开实施例的操作系统架构示意图；
图2为本公开实施例提供的媒体文件内存管理方法流程示意图；
图3为本公开实施例提供的环状的内存页结构示意图；
图4为本公开另一个实施例提供的媒体文件内存管理方法流程示意图；
图5为本公开实施例提供的生成环状的内存页结构的流程示意图；
图6为本公开另一个实施例提供的媒体文件内存管理方法流程示意图；
图7为本公开实施例提供的媒体文件内存管理装置的结构示意图；
图8为本公开另一个实施例提供的媒体文件内存管理装置的结构示意图；
图9为本公开另一个实施例提供的媒体文件内存管理装置的结构示意图。
具体实施方式
在下文中将参考附图更充分地描述示例实施例，但是所述示例实施例可以以不同形式来体现且不应当被解释为限于本文阐述的实施例。反之，提供这些实施例的目的在于使本公开透彻和完整，并将使本领域技术人员充分理解本公开的范围。
如本文所使用的，术语“和/或”包括一个或多个相关列举条目的任何和所有组合。
本文所使用的术语仅用于描述特定实施例，且不意欲限制本公开。如本文所使用的，单数形式“一个”和“该”也意欲包括复数形式，除非上下文另外清楚指出。还将理解的是，当本说明书中使用术语“包括”和/或“由……制成”时，指定存在所述特征、整体、步骤、操作、元件和/或组件，但不排除存在或添加一个或多个其他特征、整体、步骤、操作、元件、组件和/或其群组。
本文所述实施例可借助本公开的理想示意图而参考平面图和/或截面图进行描述。因此，可根据制造技术和/或容限来修改示例图示。因此，实施例不限于附图中所示的实施例，而是包括基于制造工艺而形成的配置的修改。因此，附图中例示的区具有示意性属性，并且图中所示区的形状例示了元件的区的具体形状，但并不旨在是限制性的。
除非另外限定，否则本文所用的所有术语(包括技术和科学术语)的含义与本领域普通技术人员通常理解的含义相同。还将理解，诸如那些在常用字典中限定的那些术语应当被解释为具有与其在相关技术以及本公开的背景下的含义一致的含义，且将不解释为具有理想化或过度形式上的含义，除非本文明确如此限定。
本公开实施例提供一种媒体文件内存管理方法，所述媒体文件内存管理方法应用于媒体文件内存管理装置，媒体文件内存管理装置可以为CDN服务器中的模块，CDN服务器运行Linux/Unix操作系统，将该媒体文件内存管理装置装载入操作系统之后，可以为媒体服务提供内存管理服务。本公开实施例支持xfs(日志文件系统)、ext(Extended file system，延伸文件系统)和fuse(Filesystem in Userspace，用户空间文件系统)，应用层的服务对于本发明的内容没有影响。
图1为本公开实施例的操作系统架构示意图，如图1所示，媒体文件内存管理装置在操作系统中属于Linux/Unix类内核模块，运行在Linux/Unix类内核空间，负责协助操作系统对内存页面的管理，提高系统性能。媒体文件内存管理装置运行于xfs文件系统之上，虚拟文件系统vfs 之下，通过替换xfs的sendfile(零拷贝)流程中的部分函数实现媒体文件内存管理功能。
如图2所示，本公开实施例提供的媒体文件内存管理方法包括以下步骤：
步骤21，响应于根据媒体文件播放请求获取到对应的媒体文件，为媒体文件生成环状的内存页结构；其中，所述内存页结构包括推送指针、回收指针和多个内存页面，推送指针和回收指针指向不同的内存页面，内存页面挂载有媒体文件的媒体数据，各内存页面形成环状且能够按照预设的流动方向流动。
在本步骤中，客户端向CDN服务器发送媒体文件播放请求，CDN服务器根据该媒体文件播放请求获取相应的媒体文件。在获取到媒体文件之后，媒体文件内存管理装置针对该媒体文件生成环状的内存页结构。
图3为本公开实施例提供的环状的内存页结构示意图，如图3所示，该环状的内存页结构包括多个内存页面，不同的内存页面挂载媒体文件中不同的媒体数据，各个内存页面首尾相接形成环状。内存页结构还包括推送指针和回收指针，推送指针和回收指针指向不同的内存页面。推送指针指向的内存页面所挂载的媒体数据为向客户端推送的媒体数据，也是用户正在观看的媒体数据。回收指针指向的内存页面为当前回收再利用的内存页面，通过替换该内存页面上挂载的媒体数据可以实现内存页面的回收再利用。所述内存页结构中的内存页面可以循环流动，形成数据流，流动方向(如图3中箭头所示)可以预先设置，一旦生成环状的内存页结构，内存页面的流动方向固定不变。
步骤22，在内存页面流动过程中，推送所述推送指针指向的内存页面所挂载的媒体数据，并在回收指针指向的内存页面中将所挂载的媒体数据替换为媒体文件中未被挂载的媒体数据，直至将媒体文件中最后的媒体数据挂载在内存结构中。
在本步骤中，在内存页结构中内存页面流动过程中，一方面，在推送指针的位置，将相应的内存页面上挂载的媒体数据推送给发起媒体文件播 放请求的客户端，实现推流服务，以便用户观看该媒体数据；由于用户在观看媒体文件的时候是从前往后观看的，本公开实施例将一个媒体文件的播放视为一个数据流从前往后流动。推流服务便是按照从前往后的顺序，依次向用户发送媒体文件的各个媒体数据。另一方面，在回收指针的位置，对相应的内存页面进行回收再利用，即将该内存页面中所挂载的媒体数据删除(该删除的媒体数据为已推送的媒体数据)，并将未被挂载的媒体数据挂载在该内存页面上。
在推送完成推送指针所指的内存页面上所挂载的媒体数据之后，该推送完成的内存页面会根据内存页面的流动方向流动，当该推送完成的内存页面流动到回收指针的位置时，用媒体文件中未被挂载的媒体数据替换该推送完成的内存页面中已被推送的媒体数据，从而实现对内存页面的回收和重新利用，重新挂载了新的媒体数据的内存页面继续按照内存页面的流动方向流动，在流动到推送指针的位置时，其上挂载的媒体数据被推送，这样循环往复，从而形成环状的数据流。
本公开实施例提供的媒体文件内存管理方法，所述方法包括：响应于根据媒体文件播放请求获取到对应的媒体文件，为媒体文件生成环状的内存页结构；该内存页结构包括推送指针、回收指针和多个内存页面，推送指针和回收指针指向不同的内存页面，内存页面挂载有媒体文件的媒体数据，各内存页面形成环状且能够按照预设的流动方向流动；在内存页面流动过程中，推送所述推送指针指向的内存页面所挂载的媒体数据，并在回收指针指向的内存页面中将所挂载的媒体数据替换为媒体文件中未被挂载的媒体数据，直至将媒体文件中最后的媒体数据挂载在内存结构中；本公开实施例可以提升高带宽大流量场景下CDN服务器的内存使用效率，大大减少内存回收对操作系统的消耗以及造成的时延，利用2-5M的内存即可提供数G大小的媒体文件播放服务，从而解决CDN服务器面对大流量媒体服务时性能不足的问题。
媒体文件服务有一个特点，用户从前往后观看媒体文件，其中已经看完的媒体数据大概率不会再次进行观看，因此那些挂载已推送的媒体数据的内存页面没有必要再进入操作系统自带的内存管理装置中，这些内存页 面可以直接用来挂载用户即将观看的媒体数据，这样可以绕过操作系统中复杂的内存管理流程，可以大幅减少媒体服务在获取内存页面上的耗时，从而提升媒体文件服务的性能。基于媒体文件服务的上述特点，本公开实施例提供一种环状的内存页结构，该内存页结构能够承载一个媒体文件的推流服务，挂载已推送的媒体数据的内存页面在到达回收指针的位置之后可以再次挂载尚未推送的媒体数据，通过媒体数据流和内存页面的流动，可以使用较小的内存即可实现较大媒体文件的播放。
对于大流量的媒体文件服务来说，在相关技术中，LRU内存结构具有一个比较长的链表，操作系统在遍历该链表的时候耗时较长，导致CPU冲高和内存使用率过高。在本公开实施例中，为一个媒体文件生成一个内存页结构以挂载媒体数据流，相比于LRU链表，该环状的媒体数据流所需的内存页面较小，只有2-5M左右。本公开实施例考虑到用户观看完的媒体数据大概率不会再次重复观看的特点，设计了环状的内存页结构以便形成环状的媒体数据流，能够快速将挂载了已推送完成的媒体数据的内存页面快速挂载上即将推送的媒体数据，因此只需要2-5M左右的内存大小便可播放数G大小的媒体文件，大大降低了内存的使用率和对操作系统以及CPU的消耗。
在一些实施例中，所述媒体文件内存管理方法还包括以下步骤：在为媒体文件生成环状的内存页结构的过程中，根据操作系统空闲内存的大小设置回收指针的位置。推送指针的位置是正在推送的媒体数据所属内存页面的位置(即用户正在观看的媒体数据所属内存页面的位置)，而回收指针的位置则是开始回收再利用的内存页面的位置。
在一些实施例中，如图3所示，所述内存页结构包括第一区域A和第二区域B，在第一区域A内，流动方向为从回收指针到推送指针的方向，在第二区域B内，流动方向为从推送指针到回收指针的方向。第一区域A中内存页面所挂载的媒体数据为即将被推送的媒体数据，第二区域B中内存页面所挂载的媒体数据为已推送完成的媒体数据，回收指针的位置可以相对于推送指针的位置进行调整。
相应的，所述根据操作系统空闲内存的大小设置所述回收指针的位置， 包括以下步骤：响应于操作系统空闲内存大于预设第一阈值，将回收指针设置在第一区域；响应于操作系统空闲内存小于或等于预设第一阈值，将回收指针设置在第二区域。也就是说，在操作系统空闲内存较大的情况下，将回收指针沿流动方向相对远离推送指针设置，这样，回收指针与推送指针之间挂载已推送媒体数据的内存页面较多(即第二区域B大于第一区域A)，这些内存页面会在较长时间后才会挂载上新的媒体数据，从而从整体上减少内存页面回收再利用的次数，提高CDN服务器的性能。在操作系统空闲内存较小的情况下，将回收指针沿流动方向相对接近推送指针设置，这样，回收指针与推送指针之间挂载已推送媒体数据的内存页面较少(即第二区域B小于第一区域A)，这些内存页面会在短时间内挂载上新的媒体数据，从而从整体上增加内存页面回收再利用的次数，相应的，相对于操作系统空闲内存较大的情况，CDN服务器的性能会有所下降。
相关技术中，操作系统是基于内存页面进行内存管理，并不会特别区分该内存页面上挂载数据的类型和用途。这是因为操作系统的内存管理并不针对用户或者特定服务。因此，操作系统的内存管理必须具有通用性，才能适应于各种复杂的内存管理场景。但是这种通用的内存管理方式在处理大流量大文件的媒体服务的时候则会显得性能不足，这是因为挂载媒体文件的内存页面分布在LRU链表上会造成该LRU链表很长，操作系统在回收内存页面的时候需要遍历该LRU链表，造成CPU耗时较多，从而导致CDN服务器性能不足的问题。
为解决上述问题，本公开实施例通过建立内存池，为大流量媒体服务提供单独的内存管理服务。
因此，在一些实施例中，如图4所示，在为媒体文件生成环状的内存页结构(即步骤21)之前，所述媒体文件内存管理方法还可以包括以下步骤：
步骤20，针对媒体文件，向操作系统申请第一内存页面，并根据第一内存页面建立内存池。
需要说明的是，本步骤可以在初始化阶段执行，即预先建立媒体文件的内存池。在初始化阶段，还可以执行以下初始化操作：根据CDN服务 器当前的性能确定预留内存的大小，替换sendfile等部分函数，在操作系统的proc目录下产生调试接口文件，流管理初始化，内存池初始化，创建内存管理和流管理内核线程等。
在完成内存池的初始化之后，向操作系统申请一定大小的内存(即第一内存页面)，内存申请成功之后就正式形成了内存池，开始进行媒体文件的内存管理。
相应的，所述为媒体文件生成环状的内存页结构(即步骤21)，包括：至少根据内存池中的第一内存页面为媒体文件生成环状的内存页结构。也就是说，内存页结构中的内存页面优先从相应的媒体文件的内存池中获取。
本公开实施例采用内存池技术，预先向操作系统申请一定数量的第一内存页面组成内存池，可以从CDN服务器的操作系统中获取独立的内存来为媒体文件的发包提供单独的服务。为每个媒体文件建立一个内存池，生成环状的内存页结构所需的内存页面优先从该内存池中获取。这样，该媒体文件的内存页面可以集中在一起统一管理，而不是分散开来由操作系统进行管理。因此，可以快速对这些内存页面进行操作，而不用分散到操作系统当中由操作系统自带的内存管理装置进行处理，避免了复杂的内存管理流程，无需频繁从操作系统中申请内存页面，可以减少内存管理的耗时，从而提升媒体文件内存服务的性能。
在一些实施例中，如图5所示，所述至少根据内存池中的第一内存页面为媒体文件生成环状的内存页结构，包括以下步骤：
步骤211，响应于内存池内的内存资源不足，向操作系统申请第二内存页面。
步骤212，根据内存池中的第一内存页面和第二内存页面，为媒体文件生成环状的内存页结构。
在本公开实施例中，分配给媒体文件的内存并不是固定数量不变的，在进行内存管理的过程中，若内存池中的内存资源不足，可以再次通过系统分配来获取内存。
在一些实施例中，在根据第一内存页面建立内存池之后，所述媒体文 件内存管理方法还可以包括以下步骤：不记录对第一内存页面的占用。
对第一内存页面的占用是指，一旦向操作系统成功申请第一内存页面，该第一内存页面就会被占用以建立内存池，操作系统为媒体文件内存服务分配的第一内存页面随即被占用。
在相关技术中，操作系统会对被占用的内存页面进行记录，即对被占用的内存页面的count计数加1，操作系统的内存回收程序kwsap只会回收count计数为0的内存页面。
在本公开实施例中，即使向操作系统成功申请第一内存页面，并根据第一内存页面建立内存池，真正实现了对第一内存页面的占用，也不增加所占用的第一内存页面的count计数，即不会对第一内存页面的count计数进行加1操作。因此，第一内存页面对于操作系统而言是完全透明的，操作系统完全不会感知到对第一内存页面的占用，不会影响操作系统正常通过kswap回收内存，不会影响操作系统的性能。也就是说，操作系统在内存资源不足，触发kswap进行内存资源回收的时候，能够将内存池中的第一内存页面回收，以供其他需要内存资源的程序使用,从而提升多服务合设复杂场景下CDN服务器的性能稳定性。本公开实施例可动态调节所管理的媒体文件的内存大小，增强了媒体文件内存管理的弹性，在解决CDN服务器内存使用率过高的问题的同时也不会对操作系统造成较大影响。
在一些实施例中，如图6所示，在根据第一内存页面建立内存池(即步骤20)之后，所述媒体文件内存管理方法还可以包括以下步骤：
步骤21’，响应于预设的内存页面回收条件满足，将内存池中的第一内存页面释放给操作系统。
本公开实施例可以主动将内存资源释放给操作系统，避免占用过多内存资源造成CDN服务器的内存使用率过高。
本公开实施例可以同时支持xfs、ext、fuse文件系统，并能够针对不同类型的文件系统，灵活设置内存页面回收条件。ext和fuse文件系统是操作系统中用来管理文件的系统，其中，fuse文件系统是指完全在用户态实现的文件系统，避免了服务器来回切换用户态和系统态的开销，ext文 件系统是Linux/Unix类标准拓展文件系统。
在一些实施例中，内存页面回收条件包括以下至少之一：与xfs文件系统对应的内存页面第一回收条件、与ext文件系统对应的内存页面第二回收条件、与fuse文件系统对应的内存页面第三回收条件。相应的，所述预设的内存页面回收条件满足包括：内存页面第一回收条件、内存页面第二回收条件和内存页面第三回收条件中的至少一个满足。
在多服务合设等复杂场景下，本公开实施例可以根据预设的内存页面回收条件主动向操作系统释放内存，减少CDN服务器的内存使用率，从而提升CDN服务器的整体性能。
在一些实施例中，与xfs文件系统对应的内存页面第一回收条件可以包括：操作系统的剩余内存大于预设的最小内存，且小于自定义的内存水线值；与ext文件系统对应的内存页面第二回收条件可以包括：周期设置的内存回收时间到达，示例性的，针对ext文件系统，K可以每天凌晨4点回收内存页面；与fuse文件系统对应的内存页面第三回收条件可以包括：媒体数据超过预设时长未被观看，示例性的，针对fuse文件系统，回收一小时未观看的内存页面。
针对相关技术中存在的内存回收不及时导致的CDN服务器CPU冲高、内存使用率过高的问题，本公开实施例设置了内存页结构回收机制，即回收内存页结构中的全部内存页面。因此，在一些实施例中，所述媒体文件内存管理方法还可以包括以下步骤：响应于媒体文件中的全部媒体数据推送完成，将内存页结构中的内存页面释放到内存池中。在本步骤中，可以周期扫描内存页结构，判断相应的媒体文件中全部的媒体数据是否推送完成，若全部推送完成，则可以将该内存页结构中的各个内存页面回收至内存池中。
需要说明的是，CDN服务器可以提供多个媒体文件，针对每个媒体文件均生成一个环状的内存页结构，本公开实施例可以针对每个内存页结构周期扫描，实现内存页结构的监控和回收。
本公开实施例中的CDN服务器可以适用于RTSP(Real Time Streaming  Protocol，实时流传输协议)点播业务场景、HLS(HTTP Live Streaming，基于HTTP的自适应码率流媒体传输协议)点播业务场景、DASH(Dynamic Adaptive Streaming over HTTP，基于HTTP的动态自适应流)点播业务场景、HPD(http Progressive Download，渐进式下载)点播业务场景。RTSP点播业务、HLS点播业务、DASH点播业务、HPD点播业务，其区别只是在于应用层所使用的协议和媒体文件的格式不同，其底层均是涉及媒体文件发包的服务，因此均可适用本公开实施例提供的媒体文件内存管理方法。本公开实施例的应用场景不限于以上场景，涉及CDN服务器媒体发包机制的应用场景都可以适用。
针对大流量情况下的CDN服务器性能不足问题，本公开实施例提供一种针对媒体文件进行内存管理的方案，采用内存池与数据流的思想对媒体文件进行独立的内存管理。
通过内存池的形式，可以从CDN服务器的操作系统中获取独立的内存来为媒体文件的发包提供单独的服务。用于挂载数据流的内存页面可以从内该存池中获取，若该内存池的内存资源不足，可以再向操作系统申请，另外，内存池中的内存资源也可以释放给操作系统，避免占用过多内存资源造成CDN服务器的内存使用率过高。
通过数据流的形式，可以为每一个媒体文件提供一个环状的内存页结构，当用户播放媒体文件时，已观看的媒体数据的内存页面可直接作为挂载即将播放的媒体数据的内存页面，形成一个环状的可流动的媒体数据流。这样可以使播放数G大小的文件只占用2-5M这样极小的内存，在大流量场景下可以减少对操作系统内存管理方面的消耗，减少CPU的使用率，从而提升CDN服务器整体性能。
本公开实施例的媒体文件内存管理方法，通过独立维护媒体文件的内存，采用内存池、环状的内存页结构和数据流机制，实现CDN服务器性能的提升。在获取了用户的媒体文件播放请求之后，为所请求的媒体文件生成一个环状的内存页结构，并将媒体数据挂载在该内存页结构中的内存页面上，通过内存页面的循环流动形成数据流。内存池中的内存页面可以从操作系统获取，也可以主动向操作系统释放、回收。向操作系统申请的 内存页面在进入内存池队列的时候不再增加count计数，从而使得操作系统的kswap可以正常回收这些内存页面，也就是说，本公开实施例的媒体文件内存管理装置不再保留内存，如果操作系统需要回收内存，就可以正常通过kwap进行回收。
本公开实施例方案的实现基础是替换xfs文件系统sendfile流程中的部分函数，在初始化阶段可以替换xfs文件系统的操作集和函数，因此所有通过调用sendfile的服务申请的内存页面都从内存池中获得，从而实现媒体文件的内存页面的管理功能。另外本公开实施例还拓展支持ext和fuse文件系统，可以主动的向操作系统释放内存。
基于相同的技术构思，本公开实施例还提供一种媒体文件内存管理装置，如图7所示，所述媒体文件内存管理装置包括内存页结构生成模块101、媒体数据推送模块102和内存页面回收模块103。
内存页结构生成模块101被配置成，响应于根据媒体文件播放请求获取到对应的媒体文件，为所述媒体文件生成环状的内存页结构；其中，所述内存页结构包括推送指针、回收指针和多个内存页面，所述推送指针和所述回收指针指向不同的内存页面，所述内存页面挂载有所述媒体文件的媒体数据，各所述内存页面形成环状且能够按照预设的流动方向流动。
媒体数据推送模102被配置成，在所述内存页面流动过程中，推送所述推送指针指向的内存页面所挂载的媒体数据。
内存页面回收模块103被配置成，在所述内存页面流动过程中，在所述回收指针指向的内存页面中将所挂载的媒体数据替换为所述媒体文件中未被挂载的媒体数据，直至将所述媒体文件中最后的媒体数据挂载在所述内存结构中。
在一些实施例中，内存页结构生成模块101还被配置成，在为所述媒体文件生成环状的内存页结构的过程中，根据操作系统空闲内存的大小设置所述回收指针的位置。
在一些实施例中，所述内存页结构包括第一区域和第二区域，在所述 第一区域内，所述流动方向为从所述回收指针到所述推送指针的方向，在所述第二区域内，所述流动方向为从所述推送指针到所述回收指针的方向。
内存页结构生成模块101还被配置成，响应于所述操作系统空闲内存大于预设第一阈值，将所述回收指针设置在所述第一区域；响应于所述操作系统空闲内存小于或等于预设第一阈值，将所述回收指针设置在所述第二区域。
在一些实施例中，如图8所示，所述媒体文件内存管理装置还包括内存池管理模块104，内存池管理模块104被配置成，在内存页结构生成模块101为所述媒体文件生成环状的内存页结构之前，针对所述媒体文件，向操作系统申请第一内存页面，并根据所述第一内存页面建立内存池。
内存页结构生成模块101被配置成，至少根据所述内存池中的所述第一内存页面为所述媒体文件生成环状的内存页结构。
在一些实施例中，内存页结构生成模块101被配置成，响应于所述内存池内的内存资源不足，向所述操作系统申请第二内存页面；根据所述内存池中的第一内存页面和所述第二内存页面，为所述媒体文件生成环状的内存页结构。
在一些实施例中，内存池管理模块104还被配置成，在根据所述第一内存页面建立内存池之后，不记录对所述第一内存页面的占用。
在一些实施例中，内存池管理模块104还被配置成，在根据所述第一内存页面建立内存池之后，响应于预设的内存页面回收条件满足，将所述内存池中的所述第一内存页面释放给所述操作系统。
在一些实施例中，所述内存页面回收条件包括以下至少之一：与xfs文件系统对应的内存页面第一回收条件、与ext文件系统对应的内存页面第二回收条件、与fuse文件系统对应的内存页面第三回收条件；
所述预设的内存页面回收条件满足，包括：所述内存页面第一回收条件、所述内存页面第二回收条件和所述内存页面第三回收条件中的至少一个满足。
在一些实施例中，如图9所示，所述媒体文件内存管理装置还包括流 管理模块105，流管理模块105被配置成，响应于所述媒体文件中的全部媒体数据推送完成，将所述内存页结构中的内存页面释放到所述内存池中。
本公开实施例还提供了一种计算机设备，该计算机设备包括：一个或多个处理器以及存储装置；其中，存储装置上存储有一个或多个程序，当上述一个或多个程序被上述一个或多个处理器执行时，使得上述一个或多个处理器实现如前述各实施例所提供的媒体文件内存管理方法。
本公开实施例还提供了一种计算机可读介质，其上存储有计算机程序，其中，该计算机程序被执行时实现如前述各实施例所提供的媒体文件内存管理方法。
本领域普通技术人员可以理解，上文中所公开方法中的全部或某些步骤、装置中的功能模块/单元可以被实施为软件、固件、硬件及其适当的组合。在硬件实施方式中，在以上描述中提及的功能模块/单元之间的划分不一定对应于物理组件的划分；例如，一个物理组件可以具有多个功能，或者一个功能或步骤可以由若干物理组件合作执行。某些物理组件或所有物理组件可以被实施为由处理器，如中央处理器、数字信号处理器或微处理器执行的软件，或者被实施为硬件，或者被实施为集成电路，如专用集成电路。这样的软件可以分布在计算机可读介质上，计算机可读介质可以包括计算机存储介质(或非暂时性介质)和通信介质(或暂时性介质)。如本领域普通技术人员公知的，术语计算机存储介质包括在用于存储信息(诸如计算机可读指令、数据结构、程序模块或其他数据)的任何方法或技术中实施的易失性和非易失性、可移除和不可移除介质。计算机存储介质包括但不限于RAM、ROM、EEPROM、闪存或其他存储器技术、CD-ROM、数字多功能盘(DVD)或其他光盘存储、磁盒、磁带、磁盘存储或其他磁存储装置、或者可以用于存储期望的信息并且可以被计算机访问的任何其他的介质。此外，本领域普通技术人员公知的是，通信介质通常包含计算机可读指令、数据结构、程序模块或者诸如载波或其他传输机制之类的调制数据信号中的其他数据，并且可包括任何信息递送介质。
本文已经公开了示例实施例，并且虽然采用了具体术语，但它们仅用于并仅应当被解释为一般说明性含义，并且不用于限制的目的。在一些实 例中，对本领域技术人员显而易见的是，除非另外明确指出，否则可单独使用与特定实施例相结合描述的特征、特性和/或元素，或可与其他实施例相结合描述的特征、特性和/或元件组合使用。因此，本领域技术人员将理解，在不脱离由所附的权利要求阐明的本发明的范围的情况下，可进行各种形式和细节上的改变。

Claims (12)

  1. 一种媒体文件内存管理方法，其中，所述方法包括：
    响应于根据媒体文件播放请求获取到对应的媒体文件，为所述媒体文件生成环状的内存页结构；其中，所述内存页结构包括推送指针、回收指针和多个内存页面，所述推送指针和所述回收指针指向不同的内存页面，所述内存页面挂载有所述媒体文件的媒体数据，各所述内存页面形成环状且能够按照预设的流动方向流动；
    在所述内存页面流动过程中，推送所述推送指针指向的内存页面所挂载的媒体数据，并在所述回收指针指向的内存页面中将所挂载的媒体数据替换为所述媒体文件中未被挂载的媒体数据，直至将所述媒体文件中最后的媒体数据挂载在所述内存结构中。
  2. 如权利要求1所述的方法，其中，所述方法还包括：
    在为所述媒体文件生成环状的内存页结构的过程中，根据操作系统空闲内存的大小设置所述回收指针的位置。
  3. 如权利要求2所述的方法，其中，所述内存页结构包括第一区域和第二区域，在所述第一区域内，所述流动方向为从所述回收指针到所述推送指针的方向，在所述第二区域内，所述流动方向为从所述推送指针到所述回收指针的方向；
    根据操作系统空闲内存的大小设置所述回收指针的位置，包括：
    响应于所述操作系统空闲内存大于预设第一阈值，将所述回收指针设置在所述第一区域；
    响应于所述操作系统空闲内存小于或等于预设第一阈值，将所述回收指针设置在所述第二区域。
  4. 如权利要求1所述的方法，其中，在为所述媒体文件生成环状的内存页结构之前，所述方法还包括：针对所述媒体文件，向操作系统申请第一内存页面，并根据所述第一内存页面建立内存池；
    所述为所述媒体文件生成环状的内存页结构，包括：至少根据所述内存池中的所述第一内存页面为所述媒体文件生成环状的内存页结构。
  5. 如权利要求4所述的方法，其中，所述至少根据所述内存池中的所述第一内存页面为所述媒体文件生成环状的内存页结构，包括：
    响应于所述内存池内的内存资源不足，向所述操作系统申请第二内存页面；
    根据所述内存池中的第一内存页面和所述第二内存页面，为所述媒体文件生成环状的内存页结构。
  6. 如权利要求4所述的方法，其中，在根据所述第一内存页面建立内存池之后，所述方法还包括：不记录对所述第一内存页面的占用。
  7. 如权利要求4所述的方法，其中，在根据所述第一内存页面建立内存池之后，所述方法还包括：
    响应于预设的内存页面回收条件满足，将所述内存池中的所述第一内存页面释放给所述操作系统。
  8. 如权利要求7所述的方法，其中，所述内存页面回收条件包括以下至少之一：与xfs文件系统对应的内存页面第一回收条件、与ext文件系统对应的内存页面第二回收条件、与fuse文件系统对应的内存页面第三回收条件；
    所述预设的内存页面回收条件满足，包括：所述内存页面第一回收条 件、所述内存页面第二回收条件和所述内存页面第三回收条件中的至少一个满足。
  9. 如权利要求4-8任一项所述的方法，其中，所述方法还包括：
    响应于所述媒体文件中的全部媒体数据推送完成，将所述内存页结构中的内存页面释放到所述内存池中。
  10. 一种媒体文件内存管理装置，包括内存页结构生成模块、媒体数据推送模块和内存页面回收模块，
    所述内存页结构生成模块被配置成，响应于根据媒体文件播放请求获取到对应的媒体文件，为所述媒体文件生成环状的内存页结构；其中，所述内存页结构包括推送指针、回收指针和多个内存页面，所述推送指针和所述回收指针指向不同的内存页面，所述内存页面挂载有所述媒体文件的媒体数据，各所述内存页面形成环状且能够按照预设的流动方向流动；
    所述媒体数据推送模被配置成，在所述内存页面流动过程中，推送所述推送指针指向的内存页面所挂载的媒体数据；
    所述内存页面回收模块被配置成，在所述内存页面流动过程中，在所述回收指针指向的内存页面中将所挂载的媒体数据替换为所述媒体文件中未被挂载的媒体数据，直至将所述媒体文件中最后的媒体数据挂载在所述内存结构中。
  11. 一种计算机设备，包括：
    至少一个处理器；
    存储装置，其上存储有至少一个程序；
    当所述至少一个程序被所述至少一个处理器执行时，使得所述至少一个处理器实现如权利要求1-9任一项所述的媒体文件内存管理方法。
  12. 一种计算机可读介质，其上存储有计算机程序，其中，所述程序被执行时实现如权利要求1-9任一项所述的媒体文件内存管理方法。
PCT/CN2022/128907 2021-11-04 2022-11-01 媒体文件内存管理方法、装置、计算机设备和可读介质 WO2023078228A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111300555.4 2021-11-04
CN202111300555.4A CN116069683A (zh) 2021-11-04 2021-11-04 媒体文件内存管理方法、装置、计算机设备和可读介质

Publications (1)

Publication Number Publication Date
WO2023078228A1 true WO2023078228A1 (zh) 2023-05-11

Family

ID=86177538

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/128907 WO2023078228A1 (zh) 2021-11-04 2022-11-01 媒体文件内存管理方法、装置、计算机设备和可读介质

Country Status (2)

Country Link
CN (1) CN116069683A (zh)
WO (1) WO2023078228A1 (zh)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102521279A (zh) * 2011-11-25 2012-06-27 Tcl集团股份有限公司 一种流媒体文件播放方法、系统及播放器
CN104683861A (zh) * 2013-11-27 2015-06-03 北京大唐高鸿软件技术有限公司 局域网中基于udp缓存播放媒体流的方法
CN106534889A (zh) * 2016-11-15 2017-03-22 北京工商大学 一种基于环形缓冲的实时流媒体多路转发方法
EP3661216A1 (en) * 2018-11-30 2020-06-03 InterDigital CE Patent Holdings A method and apparatus for loop-playing video content
CN112911376A (zh) * 2021-02-01 2021-06-04 华录智达科技股份有限公司 一种基于实时视频播放流畅的播放方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102521279A (zh) * 2011-11-25 2012-06-27 Tcl集团股份有限公司 一种流媒体文件播放方法、系统及播放器
CN104683861A (zh) * 2013-11-27 2015-06-03 北京大唐高鸿软件技术有限公司 局域网中基于udp缓存播放媒体流的方法
CN106534889A (zh) * 2016-11-15 2017-03-22 北京工商大学 一种基于环形缓冲的实时流媒体多路转发方法
EP3661216A1 (en) * 2018-11-30 2020-06-03 InterDigital CE Patent Holdings A method and apparatus for loop-playing video content
CN112911376A (zh) * 2021-02-01 2021-06-04 华录智达科技股份有限公司 一种基于实时视频播放流畅的播放方法

Also Published As

Publication number Publication date
CN116069683A (zh) 2023-05-05

Similar Documents

Publication Publication Date Title
US9178928B2 (en) Scalable content streaming system with server-side archiving
US8505057B2 (en) Demand-based edge caching video content system and method
US20080209067A1 (en) System And Method For Highly-Scalable Real-Time And Time-Based Data Delivery Using Server Clusters
US20140165119A1 (en) Offline download method, multimedia file download method and system thereof
US20050262246A1 (en) Systems and methods for load balancing storage and streaming media requests in a scalable, cluster-based architecture for real-time streaming
US20050262245A1 (en) Scalable cluster-based architecture for streaming media
US11310550B2 (en) System and method for storing multimedia files using an archive file format
Kumar et al. Cloud enabled media streaming using Amazon Web Services
US10715880B2 (en) Method for creation and distribution of segmented video over distributed multicast-aware sparse networks with low latency
US9241174B1 (en) Demand-based edge caching video content system and methods
AU2021343451B2 (en) Configurable access-based cache policy control
WO2020082073A1 (en) Distributed state recovery in a system having dynamic reconfiguration of participating nodes
WO2023078228A1 (zh) 媒体文件内存管理方法、装置、计算机设备和可读介质
US11825136B2 (en) Video transcoding method and apparatus
CN112243136B (zh) 内容播放方法、视频存储方法和设备
US20140149499A1 (en) Remote request fulfillment and delivery
US10887642B1 (en) Managing encoder configuration based on content delivery network performance
US10270883B2 (en) Scheduling downloads
CN102438171A (zh) 节目源文件处理方法、装置及网络电视系统
US20230171441A1 (en) Method providing to a user terminal a target multimedia content available at a master server
US20220086207A1 (en) Dynamic variant list modification to achieve bitrate reduction
CN114520832A (zh) 一种数据流处理方法、装置、设备及计算机存储介质
CN115335822A (zh) 用于管理媒体对象的存储的系统、方法和设备
EP4128800A1 (en) Fast video stream startup
Jung et al. A continuous stream service and high service availability mechanism for continuous media servers

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22889253

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE