CN116107843B - Method for determining performance of operating system, task scheduling method and equipment - Google Patents
Method for determining performance of operating system, task scheduling method and equipment Download PDFInfo
- Publication number
- CN116107843B CN116107843B CN202310379850.6A CN202310379850A CN116107843B CN 116107843 B CN116107843 B CN 116107843B CN 202310379850 A CN202310379850 A CN 202310379850A CN 116107843 B CN116107843 B CN 116107843B
- Authority
- CN
- China
- Prior art keywords
- operating system
- data
- access
- memory
- resource
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the specification provides a method for determining the performance of an operating system, a task scheduling method, task scheduling equipment and a storage medium. The method comprises the following steps: during the running process of an operating system, acquiring resource parameters of processing resources related to data access operation on the operating system, wherein at least part of the resource parameters are acquired through a kernel dynamic tracking technology; and determining the current access performance parameter of the operating system based on the resource parameter, wherein the access performance parameter is used for representing the capability of the operating system to access data to the storage device. By the method, the current data access performance of the operating system can be evaluated more accurately, the optimal data access strategy is determined based on the current performance of the operating system, the data access efficiency of the operating system is improved, and meanwhile, the stable operation of the operating system can be ensured.
Description
Technical Field
Embodiments of the present disclosure relate to the field of computer technologies, and in particular, to a method for determining performance of an operating system, a task scheduling method, a task scheduling device, and a storage medium.
Background
In the context of data access, typically, the resources available to an operating system to implement data access operations change in real time during execution. And the ability of the operating system to access the data at present can be accurately evaluated, so that the access strategy can be better optimized. For example, data is fully accessed when the task load of the operating system to access the data is small, and the access of the data is reduced when the load of the operating system to store the data is large. For another example, in the scenario of storing data using a distributed system, the allocation of data access tasks may be based on the current performance of the operating system of each device node to access the data. In either of the above scenarios, it is critical to accurately determine the current ability of an operating system to access data, and therefore, it is necessary to provide a scheme that can determine the performance of an operating system to access data in real time.
Disclosure of Invention
To overcome the problems in the related art, embodiments of the present disclosure provide a method, a task scheduling method, a device, and a storage medium for determining performance of an operating system.
According to a first aspect of embodiments of the present specification, there is provided a method of determining operating system performance, the method comprising:
During the running process of an operating system, acquiring resource parameters of processing resources related to data access operation on the operating system, wherein at least part of the resource parameters are acquired through a kernel dynamic tracking technology;
and determining the current access performance parameter of the operating system based on the resource parameter, wherein the access performance parameter is used for representing the capability of the operating system to access data to the storage device.
In some embodiments, the access performance parameters include storage load and/or available storage bandwidth; the storage load is used for representing the current task amount of the data storage task of the operating system, and the available storage bandwidth is used for representing the data writing amount of the operating system to the storage device in unit time.
In some embodiments, the operating system is a Linux system, and the access performance parameter of the operating system is determined based on the access performance parameter of the file system of the operating system and the access performance parameter of the general block layer of the operating system when the mode of the operating system for accessing data to the storage device is a direct access mode;
when the mode of the operating system for accessing data to the storage device is a cache access mode, determining access performance parameters of the operating system by the access performance parameters of a file system of the operating system and the access performance parameters of a memory management module of the operating system;
The access performance parameters of the general block layer are determined based on the resource parameters of the general block layer, the access performance parameters of the file system are determined based on the resource parameters of the file system, and the access performance parameters of the memory management module are determined based on the resource parameters of the memory management module.
In some embodiments, in the case where the operating system is in a direct access mode in which it accesses data to a storage device, the available storage bandwidth is the smaller of the available storage bandwidth of the generic block layer and the available storage bandwidth of the file system;
and under the condition that the mode of the operating system for accessing data to the storage device is a cache access mode, the available storage bandwidth is the smaller one of the available storage bandwidth of the memory management module and the available storage bandwidth of the file system.
In some embodiments, where the operating system is in a direct access mode in which data is accessed to a storage device, the storage load is determined based on a ratio of available storage bandwidth of the generic block layer to available storage bandwidth of the file system;
and under the condition that the mode of the operating system for accessing data to the storage device is a cache access mode, the storage load is determined based on the ratio of the available storage bandwidth of the memory management module to the available storage bandwidth of the file system.
In some embodiments, the resource parameters of the generic block layer include one or more of: the data read-write request resource pool of the general block layer comprises the quantity of the remained processing resources, the data quantity which can be processed by each processing resource and the processing delay of the general block layer.
In some embodiments, the available memory bandwidth of the generic block layer is positively correlated with the number of processing resources, the amount of data that each of the processing resources can handle, and negatively correlated with the processing latency of the generic block layer.
In some embodiments, the resource parameters of the memory management module include one or more of: a memory dirty page threshold, a size of available memory, a size of used memory, wherein the size of available memory is determined based on a number of unoccupied memory pages, a number of active file memory pages, a number of inactive file memory pages.
In some embodiments, the resource parameters of the file system include one or more of the following: the size of the data to be accessed, and the processing delay of the file system.
In some embodiments, the available storage bandwidth of the file system is positively correlated with the size of the data to be accessed, negatively correlated with the processing latency of the file system.
In some embodiments, the kernel dynamic tracking technique includes an eBPF technique.
According to a second aspect of embodiments of the present specification, there is provided a task scheduling method, adapted to a distributed system, where the distributed system includes a plurality of device nodes, the method including:
acquiring current processing performance parameters of an operating system of each of the plurality of device nodes, wherein the processing performance parameters are determined based on the method mentioned in the first aspect;
and allocating tasks to the equipment nodes based on the processing performance parameters of the equipment nodes.
According to a third aspect of embodiments of the present specification, there is provided an apparatus comprising a processor, a memory, a computer program stored in the memory and executable by the processor, the computer program when executed implementing the method of the first aspect and/or the second aspect.
According to a fourth aspect of embodiments of the present specification, there is provided a computer storage medium having stored thereon a computer program which, when executed by a processor, implements the method of the first aspect and/or the second aspect described above.
The beneficial effects of the embodiment of the specification are that: during operation of the operating system, resource parameters of processing resources in the operating system associated with the data access operation may be obtained, and access performance parameters characterizing the ability of the operating system to access data to the storage device may be determined based on the resource parameters. Considering that the processing resources available for data access operation of an operating system are mainly affected by the underlying hardware resources in the running process, that is, most of the resource parameters are often in the kernel mode of the operating system and change in real time in the running process, in order to accurately acquire the real-time resource parameters of the operating system to evaluate the capability of the operating system to access data in real time, the kernel dynamic tracking technology can be utilized to acquire part of the resource parameters related to the current data access operation in the operating system, and the access performance parameters of the operating system can be determined by combining the resource parameters. By the method, the current data access performance of the operating system can be evaluated more accurately, the optimal data access strategy is determined based on the current performance of the operating system, the data access efficiency of the operating system is improved, and meanwhile, the stable operation of the operating system can be ensured.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the embodiments of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the embodiments of the specification and together with the description, serve to explain the principles of the embodiments of the specification.
FIG. 1 is a schematic diagram illustrating a method of determining operating system performance in accordance with an exemplary embodiment of the present disclosure;
FIG. 2 is a block diagram of a Linux system according to an exemplary embodiment of the present disclosure;
FIG. 3 is a schematic diagram of a Linux system general block layer storage data according to an exemplary embodiment of the present disclosure;
FIG. 4 is a diagram of a Linux system direct access data mode according to an exemplary embodiment of the present disclosure;
FIG. 5 is a diagram illustrating a cache access data of a Linux system according to an exemplary embodiment of the present disclosure;
fig. 6 is a logic block diagram of an apparatus according to an exemplary embodiment of the present description.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the embodiments of the present specification. Rather, they are merely examples of apparatus and methods consistent with aspects of the embodiments of the present description as detailed in the accompanying claims.
The terminology used in the embodiments of the description presented herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the embodiments of the description presented herein. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any or all possible combinations of one or more of the associated listed items.
It should be understood that although the terms first, second, third, etc. may be used in the embodiments of the present specification to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, the first information may also be referred to as second information, and similarly, the second information may also be referred to as first information, without departing from the scope of the embodiments of the present description. The word "if" as used herein may be interpreted as "at … …" or "at … …" or "responsive to a determination", depending on the context.
In the context of data access, generally, resources available for implementing data access operations during the running process of an operating system change in real time, for example, a change in the current available memory margin, whether a kernel I/O subsystem has I/O requests (i.e., data read/write requests) stacked, whether a storage medium is in an idle state, etc., where the change in the available resources affects the current operating system's ability to access data. And the ability of the operating system to access the data at present can be accurately evaluated, so that the access strategy can be better optimized. For example, data is fully accessed when the task load of the operating system to access the data is small, and the access of the data is reduced when the load of the operating system to access the data is large. For another example, in the scenario of accessing data using a distributed system, the allocation of data access tasks may be based on the current performance of the operating system of each device node to access data. Regardless of the scenario described above, it is critical to accurately determine the ability of an operating system to currently access data.
Currently, there is no solution that can effectively determine the real-time data access capability of an operating system. For example, taking a Linux system as an example, the saturation of a disk is represented by the time ratio of processing I/O requests in a unit time in the Linux system, so as to reflect the current data access performance. However, the higher the duty cycle, the longer the I/O request can be handled in a unit time, which may be because the I/O request is too many, and thus the lower the duty cycle, the slower the I/O request may be issued, and the less time the I/O request is handled, and in fact, the indicator cannot effectively characterize the saturation of the disk, and further, the current ability of the operating system to access data cannot be effectively characterized. Therefore, there is a need to provide a solution that can accurately determine the performance of an operating system to access data in real time.
Based on this, the embodiments of the present disclosure provide a method for determining, in real time, the performance of an operating system for accessing data, where, during the running of the operating system, resource parameters of processing resources related to a data access operation in the operating system may be obtained, and access performance parameters that characterize the ability of the operating system to access data to a storage device are determined based on these resource parameters. Considering that the processing resources available for data access operation of an operating system are mainly affected by the underlying hardware resources in the running process, that is, most of the resource parameters are often in the kernel mode of the operating system and change in real time in the running process, in order to accurately acquire the real-time resource parameters of the operating system to evaluate the capability of the operating system to access data in real time, the kernel dynamic tracking technology can be utilized to acquire part of the resource parameters related to the current data access operation in the operating system, and the access performance parameters of the operating system can be determined by combining the resource parameters. By the method, the current data access performance of the operating system can be evaluated more accurately, the optimal data access strategy is determined based on the current performance of the operating system, the data access efficiency of the operating system is improved, and meanwhile, the stable operation of the operating system can be ensured.
The method for determining the performance of the operating system provided in the embodiment of the present disclosure may be implemented by a specially developed software program, for example, may be an APP, after the APP is installed in a device, the performance of access data of the operating system of the device may be monitored, or may also be a service, which is integrated on the device when the device leaves the factory, and is responsible for monitoring the performance of access data of the operating system of the device, where the embodiment of the present disclosure is not limited.
The operating system in the embodiments of the present disclosure may be used in various types of operating systems, for example, linux systems, android systems, and the like, and the embodiments of the present disclosure are not limited thereto.
The data access operation in the embodiments of the present disclosure refers to an operation that an operating system writes data to a storage device (such as a magnetic disk) or an operation that an operating system reads data from a storage device.
As shown in fig. 1, a method for determining operating system performance according to an embodiment of the disclosure may include the following steps:
s102, acquiring resource parameters of processing resources related to data access operation on an operating system in the running process of the operating system, wherein at least part of the resource parameters are acquired through a kernel dynamic tracking technology;
In step S102, during the running of the operating system, resource parameters of processing resources related to the data access operation on the operating system may be acquired. The resource parameter may be a parameter of various processing resources required for the data access operation, and these parameters may be used to characterize the current processing resources. For example, the resource parameter may be how much memory is currently available, the processing delay of the various software modules in the operating system for performing the data access operation, and so on. Considering that the types of resource parameters are more, different types of resource parameters can be obtained through different modes, for example, some resource parameters can be obtained directly through a certain interface of an operating system, and some resource parameters can be fixed performance parameters of certain hardware, so that the resource parameters can be stored in advance and then obtained from stored data.
In addition, considering that most of the resource parameters affecting the data access performance of the operating system are related to the underlying hardware, the resource parameters are often located in the kernel mode of the operating system, and the resource parameters change in real time and cannot be obtained in a general manner. Moreover, such resource parameters often have a large impact on the current performance of the operating system to access data. For example, in the Linux system, all data in the Linux system needs to be written into the storage device or read from the storage device through a general block layer in the Linux system, so that the number of processing resources in a data read-write request resource pool of the general block layer, processing delay of the general block layer and other resource parameters often have a great influence on the data access performance of the operating system, and the resource parameters are parameters of a lower layer in a kernel mode and cannot be obtained in a general mode and need to be obtained through a dynamic kernel tracking technology. Thus, in order to obtain such resource parameters for more accurate assessment of the performance of the operating system's current access data, some kernel dynamic tracking techniques may be used to obtain such resource parameters.
For example, in some embodiments, the dynamic kernel tracking technology may be eBPF (Extended Berkeley Packet Filter) technology, but of course, may be other technologies with similar functions, and the embodiments of the present disclosure are not limited thereto.
S104, determining the current access performance parameter of the operating system based on the resource parameter, wherein the access performance parameter is used for representing the capability of the operating system to access data to the storage device.
After the above-mentioned resource parameters are obtained, the current access performance parameters of the operating system may be determined by using the resource parameters in step S104, where the access performance parameters may be various parameters for characterizing the ability of the operating system to access data to the storage device. For example, the access performance parameter may be the size of the load of the operating system currently accessing the data task, or the bandwidth of resources currently available to the operating system for accessing the data, and so on.
In some embodiments, to more intuitively represent the performance of the operating system in storing data, the access performance parameter may include one or more of a storage load, available storage bandwidth, where the storage load is used to characterize how much of the operating system's current data storage tasks are. For example, the more data write requests that are received by the current operating system, the greater the amount of data each data write request requires to be written to the storage device, the more heavily it is stored. The available storage bandwidth is used to characterize how much data an operating system can write to a storage device per unit of time, e.g., the greater the amount of data that the current operating system can write to the storage device per unit of time, the greater its available storage bandwidth.
In some embodiments, it is contemplated that a plurality of software modules associated with data access operations are typically included in an operating system, and that the software modules used may be different in different data access modes. The current data access performance of each software module is used to determine the overall data access performance of the operating system. Thus, for different data access modes, the current access performance parameters of the software modules used in that mode may be determined, where each software module's current access performance parameters may be determined by the resource parameters of each software module. The overall access performance parameters of the operating system in the data access mode can then be derived based on the access performance parameters of the software modules.
For example, as shown in fig. 2, a Linux system is an architecture diagram of a Linux system, where the Linux system includes a user state and a kernel state, the user state mainly includes an upper layer application, and the kernel state generally includes a plurality of software modules such as a file system, a memory management module, a general block layer, and the like. The general block layer is used for converting operations on different storage devices into operations on logical data blocks, which is equivalent to abstracting different storage devices into a data block group, the file system is used for managing the data blocks, after the storage devices are abstracted, the file system can use the same interface to read and write the data blocks no matter for mechanical hard disks or various storage devices such as magnetic disks, all the data read and write operations need to pass through the file system, and the memory management module is responsible for completing the operations of reading and writing data into a memory.
The data access modes in the Linux system include two types, a direct access mode and a cache access mode. In the cache access mode, the data is accessed by means of memory, whereas in the direct access mode, the data is accessed by means of no memory.
The following describes the data flow of the operating system to the storage device in two modes:
(1) Direct access mode: in the direct access mode, when an upper layer application receives a data writing request (I/O request), data to be written is first transmitted to a file system, the file system processes the data to be written to a certain extent, then the data to be written is transmitted to a general block layer, and then the general block layer writes the data to be written to a storage device (such as a disk). Similarly, when data is read, the general block layer reads the data from the storage device, and then returns the data to the file system, and the data is returned to the upper layer application by the file system.
It can be seen that in the direct access mode, in the process of writing data into the storage device, since the data to be written needs to be written into the storage device, the writing operation of the data is completed, and therefore, the processing efficiency of the whole process is affected by the current data access performance of the two software modules of the file transfer system and the universal block layer.
(2) Cache access mode: in the cache access mode, when the upper layer application receives a data writing request, the data to be written is firstly transmitted to the file system, the file system processes the data to be written to a certain extent, the data to be written is slowly transmitted to the memory management module, and the memory management module caches the data to be written to the memory (page cache), and at the moment, a response of successful writing is returned to the upper layer application. And then the memory management module transmits the data to be written in the memory to the universal block layer, and the universal block layer writes the data to be written in the storage device. Similarly, for reading data from the storage device, the general block layer reads data from the storage device, the memory management module caches the data into the memory, and the data is subsequently transmitted to the file system.
In this mode, in the process of writing data into the storage device, as long as the data to be written is cached into the memory, the data writing operation is considered to be completed, so that the processing efficiency of the whole process is affected by the current data access performance of the two software modules, namely the file transfer system and the memory management module.
Thus, in some embodiments, if the operating system is a Linux system, in the case that the mode of the operating system for accessing data to the storage device is a direct access mode, the access performance parameters of the entire operating system may be determined by combining the access performance parameters of the file system of the operating system and the access performance parameters of the general block layer of the operating system.
In some embodiments, if the operating system is a Linux system, in the case that the mode of the operating system for accessing data to the storage device is a cache access mode, the access performance parameters of the entire operating system may be comprehensively determined by combining the access performance parameters of the file system of the operating system and the access performance parameters of the memory management module of the operating system.
The access performance parameter of the universal block layer is determined based on the resource parameter of the universal block layer, wherein the resource parameter of the universal block layer is the resource parameter of the processing resource used by the universal block layer in the process of data access and is used for representing the current available resource of the universal block layer.
The access performance parameter of the file system is determined based on the resource parameter of the file system, wherein the resource parameter of the file system is the resource parameter of the processing resource used by the file system when the file system performs data access and is used for representing the current available resource of the file system.
The access performance parameter of the memory management module is determined based on the resource parameter of the memory management module, and the resource parameter of the memory management module is the resource parameter of the processing resource used by the memory management module when the memory management module performs data access and is used for representing the current available resource of the memory management module.
In some embodiments, if the operating system is a Linux system, where the operating system is in a direct access mode to access data to the storage device, the available storage bandwidth of the operating system is the smaller of the available storage bandwidth of the generic block layer and the available storage bandwidth of the file system. In the case that the mode of the operating system accessing data to the storage device is the cache access mode, the available storage bandwidth of the operating system is the smaller of the available storage bandwidth of the memory management module and the available storage bandwidth of the file system.
In the direct access mode, since the data access operation is implemented by the combination of the file system and the generic block layer, the available memory bandwidth of the operating system as a whole is affected by the available memory bandwidths of the two software modules. Thus, the available storage bandwidth of the file system may be determined using the resource parameters of the file system, and the available storage bandwidth of the generic block layer may be determined using the current resource parameters of the generic block layer. Considering that the file system and the generic block layer are serially performing write operations of data, the overall available memory bandwidth of the operating system is determined by the smaller of the available memory bandwidths of the two. Therefore, the smaller of the available storage bandwidth of the file system and the available storage bandwidth of the generic block layer can be used as the final available storage bandwidth of the operating system.
Also, in the cache access mode, since the data access operation is performed by the file system and the memory management module in combination, the available memory bandwidth of the entire operating system is affected by the available memory bandwidths of the two software modules. Therefore, the available storage bandwidth of the file system can be determined by using the resource parameters of the file system, and the available storage bandwidth of the memory management module can be determined by using the current resource parameters of the memory management module. Since both are also serially performing data access operations, the overall available memory bandwidth of the operating system is determined by the smaller of the available memory bandwidths of both. Therefore, after determining the available storage bandwidth of the file system and the available storage bandwidth of the memory management module, the smaller of the two may be used as the final available storage bandwidth of the operating system.
As shown in fig. 3, one or more data read/write request resource pools (i.e., I/O request resource pools) are typically included in the generic block layer, each data read/write request resource pool corresponding to a hardware queue of a storage device (e.g., disk) for writing data in the generic block layer to the storage device. The data read-write request resource pool comprises a plurality of data read-write request processing resources, and each processing resource can be used for processing one data read-write request. The total number of processing resources in the data read-write request resource pool is generally amplified appropriately according to the depth of the corresponding hardware queue, for example, by 2 times, that is, the depth of the hardware queue is 128, and then the total number of processing resources in the data read request resource pool of the general block layer is 256. When the universal block layer receives a data read-write request, a processing resource (the number of the processing resources remained in the data read-write request resource pool is reduced by 1) can be obtained from the data read-write request resource pool and used for processing the data read-write request, and when the data read-write request is processed, the processing resource can be recovered (the number of the processing resources remained in the data read-write request resource pool is increased by 1). When the processing resources in the data read-write request resource pool are taken out, the general block layer can process new data read-write requests after the current occupied processing resources are released.
It is clear that the amount of processing resources remaining in the current data read/write request resource pool, and the amount of data that each processing resource can handle, can affect the currently available memory bandwidth of the generic block layer. Wherein the amount of data that can be processed by each processing resource is related to the type of storage device, e.g., if the storage device is a disk, the amount of data that can be processed by each processing resource is determined by the number of disk sectors that each processing resource can cover, and the size of each sector of the disk.
In addition, the general block layer has a certain processing delay when accessing data to a storage device (such as a magnetic disk), where the processing delay refers to the time consumed by the general block layer to package and issue data to be written to the storage device by using processing resources in a data read-write request resource pool, and the processing delay also affects the currently available storage bandwidth of the general block layer to a certain extent.
Thus, in some embodiments, the resource parameters of the generic block layer include one or more of the following: the data read-write request resource pool of the general block layer comprises the quantity of the remained processing resources, the data quantity which can be processed by each processing resource and the processing delay of the general block layer. The amount of processing resources remaining in the data read-write request resource pool of the general block layer and the processing delay of the general block layer can be obtained through a dynamic kernel tracking technology, and the processable data amount of each processing resource is related to the performance of a storage device (such as a magnetic disk), and the parameter can be predetermined.
In some embodiments, the available memory bandwidth of the generic block layer is inversely related to the amount of processing resources remaining in the data read/write request resource pool, the amount of data that each processing resource can handle, and inversely related to the processing latency of the generic block layer. The more the number of the processing resources remained in the data read-write request resource pool and the larger the data volume which can be processed by each processing resource, the larger the available storage bandwidth of the universal block layer. The longer the time it takes for the universal block layer to package and issue the data to be written to the storage device by using the processing resource, i.e. the longer the processing delay, the smaller the available storage bandwidth.
For example, in some embodiments, taking a storage device as an example of a disk, the available storage bandwidth B1 of the generic block layer may be calculated by the following formula (1):
formula (1)
Wherein, B1 is the available storage bandwidth of the general block layer, N is the number of processing resources remaining in the data read-write request resource pool in the general block layer, N is the number of disk sectors that each processing resource can cover, K is the size of each sector of the disk, and t1 is the processing delay of the general block layer.
For the memory management module, the available memory bandwidth is mainly determined by the size of the unoccupied memory space remaining in the current memory, that is, the size of the remaining idle memory can be obtained by only determining the current total available memory size in the memory and then removing the current available memory size. Typically, the storage space in a memory is divided into a plurality of memory pages, each memory page typically having a storage space size of 4K, wherein the memory pages to which data is written are typically referred to as dirty pages. Considering that the storage space in the memory is limited, data cannot be written infinitely, some limits are set in the memory, when the limits are triggered, an upper application can enter a blocking state, data cannot be submitted again, the data can be written into the memory continuously after the total dirty pages are reduced, and one of the limits is a dirty page threshold value. I.e., when the number of dirty pages in the memory is greater than the dirty page threshold, the upper layer application cannot commit the data any more. Therefore, the size of the current available memory can be determined based on the dirty page threshold and the total memory size, and then the available memory size is removed, so that the available memory bandwidth of the memory management module can be obtained.
Thus, in some embodiments, the resource parameters of the memory management module may include one or more of the following: a memory dirty page threshold, a size of available memory, a size of used memory.
The available memory size may be determined based on a sum of a number of memory pages currently unoccupied, a number of active file memory pages, a number of inactive file memory pages, and a size of each memory page. The size of the current used memory may be determined based on the sum of the current number of dirty pages and the number of dirty pages being written back, and the size of each memory page.
The dirty page threshold may be obtained by a per proc/sys/vm interface under the linux system, and the number of memory pages that are currently unoccupied, the number of memory pages in an active file, the number of memory pages in an inactive file, the number of current dirty pages, and the number of dirty pages that are being written back may be obtained by a per proc/meminfo interface under the linux system.
For example, in some embodiments, the available storage bandwidth B2 of the memory management module may be determined by the following equation (2):
formula (2)
Wherein B2 is the available memory bandwidth of the inner pipe management module,is dirty page threshold, < >>Is available memory >Dirty pages, & gt>Dirty pages being written back.
In general, the available storage bandwidth of the file system is a fixed value for each upper layer application, but the available storage bandwidth of the file system is below a theoretical value due to a certain processing delay of the file system. Typical file system processing data is in blocks, and currently operating systems basically set the block unit to the memory page size, e.g., 4KB, and thus ultimately equate to processing in pages; therefore, the same processing logic is used for representing the linear relationship between the size of the processing delay of the file system and the size of the data transmitted by the upper application under the condition that the available bandwidth of the memory management module and the available bandwidth of the general block layer do not become bottlenecks, namely, the processing logic of the file system layer does not fall into a blocking waiting state due to memory resources and general block layer resources.
Thus, in some embodiments, the resource parameters of the file system include one or more of the following: the size of the data to be accessed, and the processing delay of the file system.
The processing delay and the size of the data to be accessed can be obtained through kernel dynamic tracking technology (such as an eBPF program implanted on a key API (read_iter and write_iter interface implemented by a file system), and the processing delay of the file system can be obtained through the size of the data to be accessed by fitting a linear relation between the processing delay of the file system and the size of the data based on the processing delay and the size of the data to be accessed.
In some embodiments, the available storage bandwidth of the file system is positively correlated with the size of the data to be accessed, and negatively correlated with the processing latency of the file system.
For example, in some embodiments, the available storage bandwidth B3 of the file system may be determined by the following equation (3):
formula (3)
Wherein B3 is the available storage bandwidth of the file system, and t2 is the processing delay of the file system.
In some embodiments, if the operating system is a Linux system, where the operating system is in a direct access mode to access data to the storage device, the storage load of the operating system may be determined based on a ratio of the available storage bandwidth of the generic block layer to the available storage bandwidth of the file system. As shown in FIG. 4, in general, as each upper layer application stores data through the file system, the file system has an available storage bandwidth threshold for each upper layer application that is related to the performance of the file system itself. Since there may be multiple upper applications (e.g., upper application 1 and upper application 2 in the figure) storing data through the file system at the same time, after the data to be stored of the multiple upper applications reach the common block layer, the data to be stored of the multiple upper applications need to be stored into the storage device through the common block layer, so that the sum of the data storage bandwidths corresponding to the multiple upper applications cannot exceed the available bandwidth of the common block layer. That is, if the ratio of the available storage bandwidth of the generic block layer to the available storage bandwidth of the file system is larger, it means that there are more free resources in the generic block layer at this time, and thus the overall load of the operating system is smaller. Otherwise, the overall load of the operating system is large.
For example, in some embodiments, the storage load L1 of the operating system may be determined by equation (4):
formula (4)
Where L1 represents the storage load of the operating system, B1 is the available storage bandwidth of the generic block layer, and B2 is the available storage bandwidth of the file system.
Similarly, in the case where the mode in which the operating system accesses data to the storage device is the cache access mode, the storage load is determined based on a ratio of the available storage bandwidth of the memory management module to the available storage bandwidth of the resources of the file system. As shown in FIG. 5, in general, as each upper layer application stores data through the file system, the file system has one available storage bandwidth threshold for each upper layer application. Since there may be multiple upper applications (e.g., upper application 1 and upper application 2 in the figure) storing data through the file system at the same time, after the data to be stored of the multiple upper applications reach the memory management module, all the data to be stored of the multiple upper applications need to be cached in the memory, and since the resources of the memory are limited, the sum of the data storage bandwidths corresponding to the multiple upper applications cannot exceed the available bandwidth of the memory management module. Therefore, if the ratio of the available storage bandwidth of the memory management module to the available storage bandwidth of the file system is larger, it means that there are more free resources in the memory at this time, so the overall load of the operating system is smaller. Otherwise, the overall load of the operating system is large.
For example, in some embodiments, the storage load L2 of the operating system may be determined by equation (5):
formula (5)
Wherein L2 represents the storage load of the operating system, B3 is the available storage bandwidth of the memory management module, and B2 is the available storage bandwidth of the file system.
It will be appreciated that the schemes of the above embodiments can be freely combined to obtain new schemes without conflicting precursors, and for reasons of length are not exemplified here.
Further, the embodiment of the specification also provides a task scheduling method, which is applicable to a distributed system, wherein the distributed system comprises a plurality of device nodes, and each device node can be used for executing a data access task. The task scheduling method may be performed by a device that allocates tasks to a plurality of device nodes in a distributed system, where the device may be one of the plurality of device nodes or another device. In order to more reasonably distribute the data access task to improve the processing efficiency, the current access performance parameters of the operating system of each device node in the plurality of device nodes may be obtained first, where the access performance parameters may be determined by the data access method in any embodiment. After determining the access performance parameters of the respective device nodes, the respective device nodes may be assigned tasks based on the access performance parameters of the respective device nodes. For example, if it is determined that the access performance of the device node is poor based on the access performance parameter, for example, the task currently storing data is heavy, some tasks are allocated less, otherwise, some tasks are allocated more.
By accurately determining the current data access performance of the operating system by using the method in the embodiment, and then performing task allocation based on the performance, the processing efficiency of the whole distributed system can be improved.
Corresponding to the embodiment of the memory allocation method provided in the embodiment of the present disclosure, the present disclosure further provides an apparatus for determining performance of an operating system, where the apparatus includes:
the acquisition module is used for acquiring resource parameters of processing resources related to data access operation on an operating system in the running process of the operating system, wherein at least part of the resource parameters are acquired through a kernel dynamic tracking technology;
and the determining module is used for determining the current access performance parameter of the operating system based on the resource parameter, wherein the access performance parameter is used for representing the capability of the operating system to access data to the storage device.
The implementation process of the functions and roles of each unit in the above device is specifically shown in the implementation process of the corresponding steps in the above memory allocation method, and will not be described herein again.
For the device embodiments, reference is made to the description of the method embodiments for the relevant points, since they essentially correspond to the method embodiments. The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purposes of the embodiments of the present disclosure. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
In terms of hardware, as shown in fig. 6, a hardware structure diagram of a device where the compatibility testing apparatus of the embodiment of the present disclosure is located is shown, where the device may generally include other hardware besides the processor 62 and the memory 64 shown in fig. 6, such as a forwarding chip responsible for processing a packet; the device may also be a distributed device in terms of hardware architecture, possibly comprising a plurality of interface cards, for the extension of the message processing at the hardware level. The memory 64 stores computer instructions that when executed by the processor 62 implement the methods of any of the embodiments described above.
The user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or fully authorized by each party, and the collection, use and processing of related data is required to comply with the relevant laws and regulations and standards of the relevant country and region, and is provided with corresponding operation entries for the user to select authorization or rejection.
Since a part of the embodiments of the present specification contributing to the prior art or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium, comprising several instructions for causing a terminal device to perform all or part of the steps of the methods of the embodiments of the present specification. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The foregoing description of the preferred embodiments is merely illustrative of the present invention and is not intended to limit the embodiments to the specific embodiments shown, but any modifications, equivalents, improvements, etc. within the spirit and principles of the embodiments are intended to be included within the scope of the embodiments shown.
Claims (13)
1. A method of determining operating system performance, the method comprising:
during the running process of an operating system, acquiring resource parameters of processing resources related to data access operation on the operating system, wherein at least part of the resource parameters are acquired through a kernel dynamic tracking technology;
determining a current access performance parameter of the operating system based on the resource parameter, wherein the access performance parameter is used for representing the capability of the operating system to access data to a storage device;
if the operating system is a Linux system, and the mode of the operating system for accessing data to the storage device is a direct access mode, the access performance parameter of the operating system is determined by the access performance parameter determined based on the resource parameter of the file system of the operating system and the access performance parameter determined based on the resource parameter of the general block layer of the operating system; in the case that the mode of the operating system accessing data to the storage device is a cache access mode, the access performance parameter of the operating system is determined by an access performance parameter determined based on a resource parameter of a file system of the operating system and an access performance parameter determined based on a resource parameter of a memory management module of the operating system.
2. The method of claim 1, the access performance parameter comprising a storage load and/or available storage bandwidth; the storage load is used for representing the current task amount of the data storage task of the operating system, and the available storage bandwidth is used for representing the data writing amount of the operating system to the storage device in unit time.
3. The method of claim 2, wherein in the case where the operating system is in a direct access mode in which it accesses data to a storage device, the available storage bandwidth is the smaller of the available storage bandwidth of the generic block layer and the available storage bandwidth of the file system;
and under the condition that the mode of the operating system for accessing data to the storage device is a cache access mode, the available storage bandwidth is the smaller one of the available storage bandwidth of the memory management module and the available storage bandwidth of the file system.
4. The method of claim 2, wherein the storage load is determined based on a ratio of available storage bandwidth of the generic block layer to available storage bandwidth of the file system in the event that the operating system is in a direct access mode to access data to a storage device;
And under the condition that the mode of the operating system for accessing data to the storage device is a cache access mode, the storage load is determined based on the ratio of the available storage bandwidth of the memory management module to the available storage bandwidth of the file system.
5. The method of claim 3 or 4, the resource parameters of the generic block layer comprising one or more of: the data read-write request resource pool of the general block layer comprises the quantity of the remained processing resources, the data quantity which can be processed by each processing resource and the processing delay of the general block layer.
6. The method of claim 5, wherein the available memory bandwidth of the generic block layer is positively correlated with the number of processing resources, the amount of data that can be processed by each of the processing resources, and negatively correlated with the processing latency of the generic block layer.
7. The method of claim 3 or 4, wherein the resource parameters of the memory management module include one or more of: a memory dirty page threshold, a size of available memory, a size of used memory, wherein the size of available memory is determined based on a number of unoccupied memory pages, a number of active file memory pages, a number of inactive file memory pages.
8. The method of claim 3 or 4, the resource parameters of the file system comprising one or more of: the size of the data to be accessed, and the processing delay of the file system.
9. The method of claim 8, wherein the available storage bandwidth of the file system is positively correlated with the size of the data to be accessed and negatively correlated with the processing latency of the file system.
10. The method of claim 1, the kernel dynamic tracking technique comprising an eBPF technique.
11. A task scheduling method, adapted to a distributed system, where the distributed system includes a plurality of device nodes, the method comprising:
acquiring current access performance parameters of an operating system of each of the plurality of device nodes, wherein the access performance parameters are determined based on the method of any one of claims 1-10;
and allocating tasks to the equipment nodes based on the processing performance parameters of the equipment nodes.
12. An electronic device comprising a processor, a memory, a computer program stored in the memory for execution by the processor, the processor being configured to perform the method of any of the preceding claims 1-10 or 11.
13. A computer storage medium having stored thereon a computer program which, when executed by a processor, implements the method of any of the preceding claims 1-10 or 11.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310379850.6A CN116107843B (en) | 2023-04-06 | 2023-04-06 | Method for determining performance of operating system, task scheduling method and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310379850.6A CN116107843B (en) | 2023-04-06 | 2023-04-06 | Method for determining performance of operating system, task scheduling method and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116107843A CN116107843A (en) | 2023-05-12 |
| CN116107843B true CN116107843B (en) | 2023-09-26 |
Family
ID=86258224
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310379850.6A Active CN116107843B (en) | 2023-04-06 | 2023-04-06 | Method for determining performance of operating system, task scheduling method and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116107843B (en) |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1059439A (en) * | 1990-08-31 | 1992-03-11 | 国际商业机器公司 | Nonsynchronous channel/dasd communication system |
| EP2312442A2 (en) * | 2009-08-27 | 2011-04-20 | Siemens Product Lifecycle Management Software Inc. | System and method for thread scheduling in processors |
| CN105487990A (en) * | 2014-09-19 | 2016-04-13 | 中兴通讯股份有限公司 | Method and device for transmitting information messages between CPU and chip |
| US9384051B1 (en) * | 2012-09-14 | 2016-07-05 | Emc Corporation | Adaptive policy generating method and system for performance optimization |
| CN108304269A (en) * | 2017-01-13 | 2018-07-20 | 阿里巴巴集团控股有限公司 | A kind of sending, receiving method of data, device and communications framework |
| CN111143357A (en) * | 2019-12-13 | 2020-05-12 | 江苏未来智慧信息科技有限公司 | High-speed data query platform based on shared memory |
| CN111756575A (en) * | 2020-06-19 | 2020-10-09 | 星辰天合（北京）数据科技有限公司 | Performance analysis method and device of storage server and electronic equipment |
| CN111902804A (en) * | 2018-06-25 | 2020-11-06 | 阿里巴巴集团控股有限公司 | System and method for managing resources of a storage device and quantifying I/O request costs |
| CN112463400A (en) * | 2020-10-30 | 2021-03-09 | 中国电子科技集团公司第五十二研究所 | Real-time data distribution method and device based on shared memory |
| CN112486888A (en) * | 2020-12-11 | 2021-03-12 | 盛立金融软件开发（杭州）有限公司 | Market data transmission method, device, equipment and medium |
| CN113419825A (en) * | 2021-04-01 | 2021-09-21 | 阿里巴巴新加坡控股有限公司 | Resource performance estimation method, device, system, electronic equipment and computer readable storage medium |
| CN114564460A (en) * | 2022-02-25 | 2022-05-31 | 苏州浪潮智能科技有限公司 | Parameter tuning method, device, equipment and medium based on distributed storage system |
| CN114741218A (en) * | 2022-03-22 | 2022-07-12 | 阿里巴巴（中国）有限公司 | Method, device, equipment, system and medium for extracting abnormal index of operating system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080163234A1 (en) * | 2006-07-06 | 2008-07-03 | Akorri Networks, Inc. | Methods and systems for identifying application system storage resources |
-
2023
- 2023-04-06 CN CN202310379850.6A patent/CN116107843B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5155814A (en) * | 1990-08-31 | 1992-10-13 | International Business Machines Corporation | Nonsynchronous channel/dasd communication system |
| CN1059439A (en) * | 1990-08-31 | 1992-03-11 | 国际商业机器公司 | Nonsynchronous channel/dasd communication system |
| EP2312442A2 (en) * | 2009-08-27 | 2011-04-20 | Siemens Product Lifecycle Management Software Inc. | System and method for thread scheduling in processors |
| US9384051B1 (en) * | 2012-09-14 | 2016-07-05 | Emc Corporation | Adaptive policy generating method and system for performance optimization |
| CN105487990A (en) * | 2014-09-19 | 2016-04-13 | 中兴通讯股份有限公司 | Method and device for transmitting information messages between CPU and chip |
| CN108304269A (en) * | 2017-01-13 | 2018-07-20 | 阿里巴巴集团控股有限公司 | A kind of sending, receiving method of data, device and communications framework |
| CN111902804A (en) * | 2018-06-25 | 2020-11-06 | 阿里巴巴集团控股有限公司 | System and method for managing resources of a storage device and quantifying I/O request costs |
| CN111143357A (en) * | 2019-12-13 | 2020-05-12 | 江苏未来智慧信息科技有限公司 | High-speed data query platform based on shared memory |
| CN111756575A (en) * | 2020-06-19 | 2020-10-09 | 星辰天合（北京）数据科技有限公司 | Performance analysis method and device of storage server and electronic equipment |
| CN112463400A (en) * | 2020-10-30 | 2021-03-09 | 中国电子科技集团公司第五十二研究所 | Real-time data distribution method and device based on shared memory |
| CN112486888A (en) * | 2020-12-11 | 2021-03-12 | 盛立金融软件开发（杭州）有限公司 | Market data transmission method, device, equipment and medium |
| CN113419825A (en) * | 2021-04-01 | 2021-09-21 | 阿里巴巴新加坡控股有限公司 | Resource performance estimation method, device, system, electronic equipment and computer readable storage medium |
| CN114564460A (en) * | 2022-02-25 | 2022-05-31 | 苏州浪潮智能科技有限公司 | Parameter tuning method, device, equipment and medium based on distributed storage system |
| CN114741218A (en) * | 2022-03-22 | 2022-07-12 | 阿里巴巴（中国）有限公司 | Method, device, equipment, system and medium for extracting abnormal index of operating system |
Non-Patent Citations (2)
| Title |
|---|
| 付长冬,舒继武,沈美明,郑纬民.基于RAM/Disk混合设备模型的FC-SAN存储系统.计算机研究与发展.2004,(第03期),全文. * |
| 王柏生.《深度探索Linux系统虚拟化:原理与实现》，"通用块层".2020,第190-194页. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116107843A (en) | 2023-05-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106776368B (en) | Cache management method, device and system during data reading | |
| US7613870B2 (en) | Efficient memory usage in systems including volatile and high-density memories | |
| US20170371807A1 (en) | Cache data determining method and apparatus | |
| CN102831088A (en) | Data migration method and device based on mixing memory | |
| US9727465B2 (en) | Self-disabling working set cache | |
| CN109656779A (en) | Internal memory monitoring method, device, terminal and storage medium | |
| CN111857592A (en) | Data storage method and device based on object storage system and electronic equipment | |
| CN101504594A (en) | Data storage method and apparatus | |
| CN115421924A (en) | Memory allocation method, device and equipment | |
| CN114265670A (en) | Memory block sorting method, medium and computing device | |
| CN101706788A (en) | Cross-area access method for embedded file system | |
| CN110674051A (en) | Data storage method and device | |
| US6202134B1 (en) | Paging processing system in virtual storage device and paging processing method thereof | |
| CN116107843B (en) | Method for determining performance of operating system, task scheduling method and equipment | |
| CN112947851A (en) | NUMA system and page migration method in NUMA system | |
| CN116578410A (en) | Resource management method, device, computer equipment and storage medium | |
| CN109508140B (en) | Storage resource management method and device, electronic equipment and system | |
| CN114741218A (en) | Method, device, equipment, system and medium for extracting abnormal index of operating system | |
| CN113778821A (en) | Solid state disk and medium access management method thereof | |
| CN115934364B (en) | Memory management method and device and electronic equipment | |
| JP2006277530A (en) | Allocation system, allocation apparatus, allocation method and program for the same | |
| CN117056363B (en) | Data caching method, system, equipment and storage medium | |
| CN111190543B (en) | Storage method and system for sharing NVDIMM storage resources among threads | |
| CN116126546B (en) | Performance optimization method and device, electronic equipment and medium | |
| CN111078122B (en) | Data processing method, device and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |