CN116166508B - IO data analysis method, device, equipment, storage medium and system - Google Patents
IO data analysis method, device, equipment, storage medium and system Download PDFInfo
- Publication number
- CN116166508B CN116166508B CN202310412396.XA CN202310412396A CN116166508B CN 116166508 B CN116166508 B CN 116166508B CN 202310412396 A CN202310412396 A CN 202310412396A CN 116166508 B CN116166508 B CN 116166508B
- Authority
- CN
- China
- Prior art keywords
- data
- server
- program
- copy
- user space
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 89
- 238000007405 data analysis Methods 0.000 title claims abstract description 46
- 238000012545 processing Methods 0.000 claims abstract description 69
- 238000004458 analytical method Methods 0.000 claims abstract description 61
- 230000006835 compression Effects 0.000 claims abstract description 20
- 238000007906 compression Methods 0.000 claims abstract description 20
- 230000006870 function Effects 0.000 claims description 42
- 230000008569 process Effects 0.000 claims description 31
- 230000006837 decompression Effects 0.000 claims description 12
- 230000008859 change Effects 0.000 claims description 10
- 238000004891 communication Methods 0.000 claims description 9
- 238000012544 monitoring process Methods 0.000 claims description 7
- 238000004364 calculation method Methods 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 5
- 230000035945 sensitivity Effects 0.000 claims description 5
- 238000012800 visualization Methods 0.000 claims description 5
- 238000012163 sequencing technique Methods 0.000 claims description 2
- 239000000523 sample Substances 0.000 claims 1
- 230000000694 effects Effects 0.000 abstract description 11
- 238000005457 optimization Methods 0.000 abstract description 8
- 230000005540 biological transmission Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 9
- 238000005516 engineering process Methods 0.000 description 7
- 238000013480 data collection Methods 0.000 description 5
- 230000006399 behavior Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3041—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is an input/output interface
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
- G06F11/3072—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
- G06F11/3082—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting the data filtering being achieved by aggregating or compressing the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3452—Performance evaluation by statistical analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Debugging And Monitoring (AREA)
Abstract
The disclosure relates to an IO data analysis method, device, equipment and storage medium. Responding to an IO data acquisition request, and acquiring IO data of a target application based on a capturing program in a kernel space and a copying program in a user space; carrying out serialization processing and compression processing on the IO data to generate compressed data corresponding to the IO data; and uploading the compressed data to a server, wherein the server is used for decompressing the compressed data and performing inverse serialization processing to obtain IO data, and analyzing the IO data by the server to obtain an analysis result of the IO data, wherein the analysis result of the IO data is used as reference data for optimizing the application of the super computing system. In this way, the IO data of the application can be acquired in the kernel space and the user space, and the granularity and the hierarchy of the IO data are reduced, so that the analysis precision of the IO data is improved, and finally the optimization effect of the super computer system application is improved.
Description
Technical Field
The disclosure relates to the technical field of data analysis, and in particular relates to an IO data analysis method, device, equipment, storage medium and system.
Background
With the continued development of computer technology, supercomputers have been widely used in such fields as electric power, weather prediction, and aerospace. When using supercomputers, applications of supercomputers are often optimized, and then a developer is required to observe the behavior of the applications before and after the application is adjusted, so that the developer can determine an adjustment scheme and evaluate the influence of the adjustment scheme on the applications, wherein a key index affecting the application performance is IO data.
In order to facilitate a developer to determine the influence of the adjustment scheme and the evaluation scheme on the application, the related technology generally adopts Darshan to collect IO data and then analyzes the collected IO data. However, darshan uses a pile inserting mode to collect IO data, so that the IO data can be collected only in a user space, the granularity and the hierarchy of the IO data are too high, the analysis precision of the IO data is reduced, and the optimization effect of the super computer system application is further affected.
Disclosure of Invention
In order to solve the technical problems described above or at least partially solve the technical problems described above, the present disclosure provides an IO data analysis method, apparatus, device, storage medium, and system.
In a first aspect, the present disclosure provides an IO data analysis method applied to a local device, where the method includes:
responding to an IO data acquisition request, and acquiring IO data of a target application based on a capturing program in a kernel space and a copying program in a user space;
carrying out serialization processing and compression processing on the IO data to generate compressed data corresponding to the IO data;
uploading the compressed data to a server, wherein the server is used for performing decompression processing and inverse serialization processing on the compressed data to obtain the IO data, and analyzing the IO data by the server to obtain an analysis result of the IO data, wherein the analysis result of the IO data is used as reference data for optimizing the super computing system application.
In a second aspect, the present disclosure provides an IO data analysis apparatus configured to a local device, the apparatus including:
the acquisition module is used for responding to the IO data acquisition request and acquiring IO data of the target application based on the acquisition program in the kernel space and the copy program in the user space;
the generation module is used for carrying out serialization processing and compression processing on the IO data and generating compressed data corresponding to the IO data;
The analysis module is used for uploading the compressed data to a server, wherein the server is used for carrying out decompression processing and reverse sequencing processing on the compressed data to obtain the IO data, and the server is used for analyzing the IO data to obtain an analysis result of the IO data, wherein the analysis result of the IO data is used as reference data for optimizing the application of the super computing system.
In a third aspect, embodiments of the present disclosure also provide an apparatus, the apparatus comprising:
one or more processors;
storage means for storing one or more programs,
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method provided by the first aspect.
In a fourth aspect, embodiments of the present disclosure also provide a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method provided by the first aspect.
In a fifth aspect, an embodiment of the present disclosure further provides an IO data analysis system, including: a local device and a server;
the local equipment is used for responding to an IO data acquisition request, acquiring IO data of a target application based on a capture program in a kernel space and a copy program in a user space, carrying out serialization processing and compression processing on the IO data, generating compressed data corresponding to the IO data, and uploading the compressed data to a server;
The server is used for decompressing and deserializing the compressed data to obtain the IO data, and analyzing the IO data to obtain an analysis result of the IO data.
Compared with the prior art, the technical scheme provided by the embodiment of the disclosure has the following advantages:
an IO data analysis method, device, storage medium and system of an embodiment of the present disclosure, where the method is applied to a local device, and the method includes: responding to an IO data acquisition request, and acquiring IO data of a target application based on a capturing program in a kernel space and a copying program in a user space; carrying out serialization processing and compression processing on the IO data to generate compressed data corresponding to the IO data; and uploading the compressed data to a server, wherein the server is used for decompressing the compressed data and performing inverse serialization processing to obtain IO data, and analyzing the IO data by the server to obtain an analysis result of the IO data, wherein the analysis result of the IO data is used as reference data for optimizing the application of the super computing system. In this way, the IO data of the application can be acquired in the kernel space and the user space, and the granularity and the hierarchy of the IO data are reduced, so that the analysis precision of the IO data is improved, and finally the optimization effect of the super computer system application is improved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure.
In order to more clearly illustrate the embodiments of the present disclosure or the solutions in the prior art, the drawings that are required for the description of the embodiments or the prior art will be briefly described below, and it will be obvious to those skilled in the art that other drawings can be obtained from these drawings without inventive effort.
Fig. 1 is a schematic architecture diagram of an IO data analysis system according to an embodiment of the present disclosure;
fig. 2 is a flow chart of an IO data analysis method according to an embodiment of the present disclosure;
FIG. 3 is a schematic diagram of another IO data analysis system according to an embodiment of the present disclosure;
fig. 4 is a schematic structural diagram of an IO data analysis device according to an embodiment of the present disclosure;
fig. 5 is a schematic structural diagram of an IO data analysis device according to an embodiment of the present disclosure.
Detailed Description
In order that the above objects, features and advantages of the present disclosure may be more clearly understood, a further description of aspects of the present disclosure will be provided below. It should be noted that, without conflict, the embodiments of the present disclosure and features in the embodiments may be combined with each other.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure, but the present disclosure may be practiced otherwise than as described herein; it will be apparent that the embodiments in the specification are only some, but not all, embodiments of the disclosure.
Currently, many technologies have the capability to capture I/O data for a single application running and specific component in the storage system, but for supercomputing systems for hundreds of thousands of compute kernels and capacity-reachable PB storage, continuous characterization of most technologies on supercomputing systems is almost impossible to meet the needs of timeliness, availability, etc. In order to meet the requirements of timeliness and effectiveness, the related technology adopts Darshan to acquire IO data. However, darshan inserts the function into the function library to realize IO data collection, and uses Darshan to recompile, which results in serious limitation on expandability and universality. In addition, since Darshan uses a pile inserting mode to collect data, the Darshan can only collect IO data in a user space, so that the granularity and the hierarchy of the IO data are large, and the optimization of the super computer system application is limited.
In order to solve the above problems, embodiments of the present disclosure provide an IO data analysis method, apparatus, device, storage medium, and system. The IO data analysis method provided by the embodiment of the disclosure is applied to the IO data analysis system shown in fig. 1. Referring to the figure, the IO data analysis system includes: a local device 100 and a server 200.
The local device 100 is configured to collect, in response to an IO data collection request, IO data of a target application based on a capture program in a kernel space and a copy program in a user space, perform serialization processing and compression processing on the IO data, generate compressed data corresponding to the IO data, and upload the compressed data to a server;
the server 200 is configured to perform decompression processing and inverse serialization processing on the compressed data to obtain IO data, and analyze the IO data to obtain an analysis result of the IO data.
Optionally, the local device 100 includes a client 110 and a proxy 120, and the server 200 includes a target collection device 210, a database 220, and an analysis device 230.
Specifically, in the process that the client 110 copies the IO data from the target storage area according to the first copy threshold by using the copy program in the user space, it is detected whether the IO data is successfully copied to the user space; if the IO data is successfully copied to the user space, the first copy threshold is adjusted to a second copy threshold by the client 110, and the IO data stored in the target storage area is continuously copied to the user space according to the second copy threshold by using a copy program in the user space, wherein the first copy threshold is smaller than the second copy threshold; if the IO data is not successfully copied to the user space, the first copy threshold is adjusted to a third copy threshold by the client 110, and the IO data stored in the target storage area is continuously copied to the user space according to the third copy threshold by using a copy program in the user space, where the first copy threshold is greater than the third copy threshold.
Specifically, through the proxy end 120, the IO data is subjected to serialization processing and compression processing, so as to generate compressed data corresponding to the IO data.
Specifically, based on the remote procedure call mode, the compressed data is uploaded to the target collecting device 210 according to the extensible serialization structure, where the target collecting device 210 is configured to perform decompression processing and inverse serialization processing on the compressed data to obtain IO data, and the target collecting device 210 stores the IO data in the database 220 according to a preset storage mode.
Specifically, the analysis device 230 determines an IO data change map based on the IO data stored in the database; and/or, the analysis device 230 determines a file system data prefetch accuracy based on the IO data stored in the database; and/or, the analysis device 230 determines, based on the IO data stored in the database, a degree of balance between different nodes for the number of IO data acquisition requests during operation of the job; and/or, the analysis device 230 determines the job I/O read-write sensitivity intensity based on the IO data stored in the database; and/or, the analysis device 230 determines a degree of balance of the I/O load among different nodes during operation of the job based on the IO data stored in the database.
By the method, the IO data of the application can be acquired in the kernel space and the user space, and the granularity and the hierarchy of the IO data are reduced, so that the analysis precision of the IO data is improved, and finally the optimization effect of the super computer system application is improved.
On the basis of the above embodiments, an IO data analysis method provided by the embodiments of the present disclosure will be described first with reference to fig. 2 to 3.
Fig. 2 shows a flow chart of an IO data analysis method according to an embodiment of the present disclosure.
In the embodiment of the present disclosure, the IO data analysis method shown in fig. 2 may be performed by the local device in fig. 1. As shown in fig. 2, the IO data analysis method may include the following steps.
S210, responding to an IO data acquisition request, and acquiring IO data of a target application based on a capture program in a kernel space and a copy program in a user space.
In this embodiment, when the IO data needs to be analyzed, the user may send an IO data acquisition request to the local device, or the local device periodically generates the IO data acquisition request, and the local device may specifically acquire the IO data of the target application through the client. Specifically, the client may collect IO data of the target application based on the related program in the user space and in combination with the related program in the kernel space.
The kernel space is an area accessed by the kernel of the operating system, is independent of a common application program, and is a protected memory space.
Wherein the user space is a memory area accessible to common applications.
The capturing program is a functional program for capturing IO data in the kernel space and is used for capturing the IO data of the target application and storing the captured IO data in the kernel space.
The copy program is a functional program that copies IO data stored in the kernel space to the user space.
The target application refers to an application program needing to be optimized in the super computer system. Specifically, the target application may be a producer that produces IO data, or may be a specific application program, or may be an application program corresponding to a specific process.
Alternatively, the target application may have a calculation function, a storage function, a visualization function, a monitoring function. For example, the target application with the calculation function may specifically provide calculation functions such as submitting a job status, viewing a job status, canceling a job status, etc., the target application with the storage function may specifically provide storage functions such as file preview, file download, file deletion, etc., the target application with the visualization function may provide functions such as remote visualization software service, etc., and the target application with the monitoring function may provide functions such as monitoring information of a job.
S220, carrying out serialization processing and compression processing on the IO data to generate compressed data corresponding to the IO data.
In this embodiment, after the client in the local device collects the IO data, the client may send the IO data to the proxy in the local device, so that the proxy performs serialization processing and compression processing on the IO data, and generates compressed data corresponding to the IO data.
It can be understood that the purpose of the proxy end is mainly to improve the working efficiency of the client end, and the main work of the client end is to collect and take out IO data from the kernel space so as to ensure the transmission efficiency of the IO data. Specifically, the client in the local device can adopt an Advanced inter-process communication (Advanced IPC) mode to send the IO data to the proxy, and the reliability and efficiency of the Advanced inter-process communication transfer are high, so that the influence on the system is small, and the sending efficiency and reliability of the IO data are improved.
S230, uploading the compressed data to a server, wherein the server is used for performing decompression processing and anti-serialization processing on the compressed data to obtain IO data, and analyzing the IO data by the server to obtain an analysis result of the IO data, wherein the analysis result of the IO data is used as reference data for optimizing the super computing system application.
In this embodiment, the client may specifically upload the compressed data to the server through the proxy, that is, the purpose of the proxy is mainly to pay attention to how efficiently and reliably the IO data is uploaded to the server.
In this embodiment, optionally, S230 specifically includes: based on a remote procedure call mode, uploading the compressed data to target collection equipment in a server according to an extensible serialization structure, wherein the target collection equipment is used for decompressing the compressed data and performing reverse serialization processing on the compressed data to obtain IO data, and storing the IO data to a database in the server according to a preset storage mode.
It will be appreciated that based on the efficient and reliable requirements of IO data, the proxy-side uploads compressed data to the target collection device in the server in a scalable serialization structure (protobuf) based on a remote procedure call (gRPC) approach. The gRPC is a high-performance and general open source RPC framework, is mainly developed for mobile application and is designed based on HTTP/2 protocol standards, and simultaneously supports most programming languages. Protobuf is a language independent, platform independent, extensible, serialized structured data format for use in the fields of communication protocols, data storage. As the protobuf has higher performance and efficiency and better compression effect, gRPC adopts a binary format transmission protocol, multiplexing can be supported, and therefore, the efficiency and reliability of IO data transmission from the proxy end to the target collection device are improved.
It can be understood that the super computing system is huge and needs to transmit a larger data volume, so as to avoid that the same collecting device is connected with all agent ends, and thus the pressure of the collecting device is too high. To ensure the performance of the collection device to a greater extent, optionally, "upload compressed data to the target collection device in the server in an extensible serialization structure based on remote procedure call mode", includes: acquiring a plurality of collecting devices which are in communication connection with the local device from a server, and determining the residual resource quantity of each collecting device; selecting a target collection device from the plurality of collection devices, wherein the amount of remaining resources is greater than or equal to a preset resource amount threshold; based on the remote procedure call mode, the compressed data is uploaded to a target collection device in a server according to an extensible serialization structure.
The amount of the remaining resources of each collecting device refers to the remaining resources of each collecting device, and can be used for measuring the capability of each collecting device to process compressed data. It will be appreciated that the greater the amount of resources remaining for each collection device, the more powerful the collection device is characterized by its ability to process compressed data, and conversely, the less the amount of resources remaining for each collection device, the less the collection device is characterized by its ability to process compressed data.
The preset resource quantity threshold is a preset resource size and is used for determining the target collection device.
Therefore, the collection equipment with stronger processing capacity can be selected to receive the compressed data sent by the proxy end, so that the processing efficiency and the processing effect of the compressed data are guaranteed, in addition, the compressed data are transmitted in a Protobuf+gRPC mode, and the reliability and the efficiency of the compressed data sending process are guaranteed.
Further, after receiving the compressed data, the target collecting device performs decompression processing and inverse serialization processing on the compressed data, so as to restore the IO data. Specifically, the target collecting device is written in advance by adopting the Go language, and due to the concurrent characteristic of the Go language and the fact that the target collecting device can use more abundant resources of the system, the method is beneficial to guaranteeing the restoring effect of compressed data and is fully utilized subsequently.
Further, after the target collecting device restores the IO data, the IO data may be stored in the database. The time sequence database is a database specially used for processing time sequence data, the time sequence database has time sequence relations in different I/O data which are not difficult to analyze through the form of the I/O data, and the time sequence database is a database service integrating high-efficiency reading, writing, compression storage and real-time computing capacity, and obviously, the time sequence database is suitable for real-time monitoring of equipment and business service, so that prediction and alarm can be carried out.
Alternatively, the database may store IO data in the following storage structure:
the capturing time is the entity identification (PID, CPU ID, R/W, etc.) the file related identification (file system name, file name, etc.) the index name is the index value.
Furthermore, after the IO data is stored in the database, the analysis equipment in the server can immediately acquire the stored IO data from the database, or acquire the stored IO data from the database after a certain time, and deeply mine the IO data.
Optionally, the analysis device in the server determines an IO data change map based on the IO data stored in the database; and/or the number of the groups of groups,
analyzing equipment in the server determines the file system data prefetching accuracy based on IO data stored in a database; and/or the number of the groups of groups,
the analysis equipment in the server determines the balance degree of the times of IO data acquisition requests among different nodes in the operation process based on the IO data stored in the database; and/or the number of the groups of groups,
the analysis equipment in the server determines the read-write sensitivity intensity of the operation I/O based on the IO data stored in the database; and/or the number of the groups of groups,
the analysis equipment in the server determines the balance degree of the I/O load among different nodes in the operation process of the job based on the IO data stored in the database.
The IO data change graph comprises but is not limited to a thermodynamic diagram, a graph, a histogram, a scatter diagram and the like, so that the load condition of different IO libs called by the job can be conveniently known, the core IO and load peaks in the job can be conveniently known, and a curve is drawn from a local to global angle. Optionally, the IO data change map specifically includes: the portable operating system interfaces (Portable Operating System Interface, POSIX) operate count change curves and the ratio of IO (input/output) quantities (such as read-write times, read-write data quantities and file quantities) of each node involved in the operation are convenient to know the condition of POSIX I/O called by the operation; the multipoint interface (MultiPointInterface, MPI) operates the counting change curve, the job involves the IO quantity (such as read-write times, read-write data quantity, file quantity) of each node to account for, facilitate knowing the MPI/O condition that the job calls; standard (STD) operation count change curve, job related node IO quantity (such as read-write times, read-write data quantity, file quantity) duty ratio, so as to be convenient for knowing STD I/O condition called by job; the virtual file system (Virtual File Systems, VFS) operates the counting change curve, the job involves each node IO quantity (for example, read-write times, read-write data quantity, file quantity) duty ratio, the kernel IO working condition while the job IO is involved is easy to know; and based on the file, showing the time of operating the file by different nodes and reading and writing the file quantity, so as to be convenient for finding out the operation core IO. The accuracy of file system data prefetching may be determined by:
Wherein,,
the file system data prefetching accuracy rate;
when a job reads a file containing IO data, the number of caches (page caches) in the system contained in the IO data; />
The IO data acquisition request comprises the number of times of reading IO by the operation request.
The degree of balance of the number of IO data acquisition requests among different nodes in the operation process can be determined by the following modes:
wherein,,
the degree of balance of the times of IO data acquisition requests among different nodes in the operation process is the degree of balance of the times of IO data acquisition requests among different nodes; />
The number of the calculation nodes used in the process of analyzing IO data; />
When IO data is analyzed, a certain node sends out the number of reading requests of the IO data during operation; />
When IO data is analyzed, a certain node sends out the number of write requests of the IO data during operation; />
When IO data is analyzed, the number of all read requests is sent out; />
Is the number of all write requests issued when analyzing the IO data.
The sensitivity of the operation I/O read-write can be determined by the following modes:
wherein,,
is the number of read operations, +.>
The number of job read operations contained in the IO data; WC is the number of write operations, +.>
The number of job read operations contained in the IO data; / >
When it indicates that IO data containsThe operation of (1) is write intensive operation, +.>
When the operation included in the IO data is read-intensive operation.
The balance degree of the I/O load among different nodes in the operation process of the job can be determined by the following modes:
wherein,,
is the total read operation data contained in the IO data analysis request, subscript +.>
Is the total amount of read operation data of a node in the operation contained in IO data in the operation running process, and subscript +.>
The total read operation data of the jobs contained in the IO data in the running process; />
Is the total write operation data amount contained in the IO data analysis request, and subscripts
Is the total data amount of writing operation of a certain node in the operation contained in IO data in the operation running process, and subscripts
Is the total amount of write operation data of the jobs contained in the IO data in the running process.
Therefore, the analysis equipment in the server can analyze the IO data in multiple dimensions, so that multiple analysis results are determined, and the comprehensive analysis of the IO data is realized.
The embodiment of the disclosure provides an IO data analysis method applied to local equipment, comprising the following steps: responding to an IO data acquisition request, and acquiring IO data of a target application based on a capturing program in a kernel space and a copying program in a user space; carrying out serialization processing and compression processing on the IO data to generate compressed data corresponding to the IO data; and uploading the compressed data to a server, wherein the server is used for decompressing the compressed data and performing inverse serialization processing to obtain IO data, and analyzing the IO data by the server to obtain an analysis result of the IO data, wherein the analysis result of the IO data is used as reference data for optimizing the application of the super computing system. In this way, the IO data of the application can be acquired in the kernel space and the user space, and the granularity and the hierarchy of the IO data are reduced, so that the analysis precision of the IO data is improved, and finally the optimization effect of the super computer system application is improved.
In another embodiment of the present disclosure, in order to reduce performance loss and ensure reliability of IO data during IO data acquisition, the IO data is temporarily stored based on a capture program in a kernel space, and then the stored IO data is copied to a user space based on a copy program in the user space, so that the IO data is further sent to a proxy based on a related program in the user space.
In some embodiments of the present disclosure, "collecting IO data of a target application based on a trap program in kernel space and a copy program in user space" in execution S210 includes:
s2101, capturing IO data of a target application based on a capturing program in a kernel space, and storing the IO data into a target storage area;
s2102, based on the copy program in the user space, the IO data in the target storage area is copied to the user space.
The target storage area is in a temporary cache of kernel space in the client.
It will be appreciated that the client is collected in a system call (syscalls) and a Virtual File System (VFS), that is, in the kernel space, so that the IO data collected based on the berkeley packet filter (Berkeley Packet Filter, BPF) mechanism is stored in a storage area in the kernel space, and then the IO data stored in the storage area can be sent and analyzed after being fetched.
Based on this, fig. 3 shows a schematic structural diagram of another IO data analysis system in order to facilitate understanding of the manner of collecting IO data. As shown in fig. 3, the client 310 in the local device 300 includes a User Space (User Space) configured with a transmission trace function (transmit trace), an acquisition trace function (get trace), and a plurality of bpf programs (bpf prog 1). Specifically, the multiple bpf programs in the user Space are static Function programs, when IO data collection is performed, the client writes the multiple bpf programs in the user Space into multiple Kernel Function (Kernel Function) in Kernel Space (Kernel Space), so that the multiple bpf programs in the Kernel Space have the capability of collecting IO data, the multiple bpf are used as capturing programs in the Kernel Space to collect IO data of target application, the IO data are stored in a target storage area (MAP) in the Kernel Space, then the tracking Function in the user Space is used as a copying program, the IO data are actively obtained from the target storage area by using the copying program, or the IO data pushed by the target storage area are obtained, and the IO data are copied to the user Space, so that the IO data collection is further completed. Optionally, each kernel function further comprises an original function (original function), the bpf program in each kernel function comprising: capture trace function (capture trace) and write function (write into).
It will be appreciated that the storage space of the target storage area is limited, but the speed at which the I/O data is collected is determined by the I/O data producer. This results in undefined behavior of the target storage area if the I/O data is copied to the user space when the target storage area is full, while the acquisition action still takes place. However, it can be determined that the above situation must cause the loss of the collected IO data, if the collection speed of the I/O data is fast, the copying speed of the target storage area is slow, which will cause a large amount of IO data to be lost, and the reliability of the IO data collection process cannot be ensured, and if one piece of IO data is collected, one piece of data is copied to the user space, although the loss of the IO data can be avoided, too many data copies will be brought, and the loss of the collection to the system is greatly increased.
In order to reduce the loss generated in the process of collecting the IO data while ensuring the reliability of collecting the IO data, in this embodiment, S2102 specifically includes:
in the process that a client in the local equipment copies IO data from a target storage area according to a first copy threshold by using a copy program in a user space, detecting whether the IO data is successfully copied to the user space;
If the IO data is successfully copied to the user space, the first copy threshold is adjusted to a second copy threshold through the client, and the IO data stored in the target storage area is continuously copied to the user space according to the second copy threshold by utilizing a copy program in the user space, wherein the first copy threshold is smaller than the second copy threshold;
if the IO data is not successfully copied to the user space, the first copy threshold is adjusted to a third copy threshold through the client, and the IO data stored in the target storage area is continuously copied to the user space according to the third copy threshold by utilizing a copy program in the user space, wherein the first copy threshold is larger than the third copy threshold.
It can be understood that when the IO data is acquired, the first copy threshold is set as an initial acquisition threshold, and the copy threshold is adjusted in a sliding window mode based on the actual copy condition of the IO data. Specifically, if the IO data is successfully copied to the user space, the IO data is successfully acquired, the reliability of the IO data acquisition is higher, and in order to reduce the loss of acquisition to the system, the data volume of single transmission is increased, so that the number of times of IO data transmission is reduced, the reliability of the IO data acquisition is ensured, and meanwhile, the loss generated in the IO data acquisition process can be reduced. In contrast, if the IO data is not successfully copied to the user space, the IO data is not successfully acquired, the reliability of IO data acquisition is lower, and in order to improve the reliability of IO data acquisition, the data volume of single transmission is reduced to improve the number of times of IO data transmission, so that the probability of successfully copying the IO data can be improved, the reliability of the acquired IO data is ensured, and meanwhile, the loss generated in the acquisition process of the IO data can be reduced.
The first copy threshold, the second copy threshold, and the third copy threshold may be determined according to a size of the target storage area. Alternatively, the first copy threshold may be 50% of the target storage area, the second copy threshold may be 85% of the target storage area, and the third copy threshold may be 25% of the target storage area.
Therefore, when IO data is acquired, the copying threshold value is adjusted in a sliding window mode based on the actual copying condition of the IO data, so that the reliability of the acquired IO data is ensured, and meanwhile, the loss generated in the acquisition process of the IO data can be reduced.
Further, the user space in the client 310 may transmit the IO data to the proxy 320 based on the transmission tracking function and in an advanced inter-process communication manner, so that the proxy 320 performs serialization processing and compression processing on the IO data by using a pre-processing tracking function (pre-processing trace) to obtain compressed data. The proxy 320 then uploads the compressed data to the target collection device 340 in the server 330 in an extensible serialization structure using a send trace function (send trace) and using a remote procedure call. Further, the target collecting device 340 performs decompression and inverse serialization on the compressed data based on a remote procedure call service (rpc server) module to obtain IO data, and the target collecting device 340 stores the IO data to the database 350 according to a preset storage mode based on an time sequence database (influxdb client) module. Finally, the analysis device 360 analyzes the IO data to obtain an analysis result of the IO data.
In conclusion, the function in different devices and the communication mode among the devices are utilized to collect and transmit IO data, so that the collected IO data is analyzed to realize collection and analysis of the IO data, the collection process of the IO data is applicable to various system versions, the effect of high universality is achieved, moreover, the collection process of the IO data is not perceived by a user, is irrelevant to development language of computing science application, and IO load can be comprehensively analyzed from two aspects of a user layer and a system layer, so that advice is provided for application optimization.
The embodiment of the disclosure further provides an IO data analysis device for implementing the IO data analysis method, and the description is given below with reference to fig. 4. In the embodiment of the present disclosure, the IO data analysis apparatus may be configured in the local device in fig. 1.
Fig. 4 shows a schematic structural diagram of an IO data analysis device according to an embodiment of the present disclosure.
As shown in fig. 4, the IO data analysis module 400 may include:
the acquisition module 410 is configured to acquire, in response to an IO data acquisition request, IO data of a target application based on a capture program in a kernel space and a copy program in a user space;
the generating module 420 is configured to perform serialization processing and compression processing on the IO data, and generate compressed data corresponding to the IO data;
The analysis module 430 is configured to upload the compressed data to a server, where the server is configured to perform decompression processing and inverse serialization processing on the compressed data to obtain the IO data, and the server is configured to analyze the IO data to obtain an analysis result of the IO data, where the analysis result of the IO data is used as reference data for optimizing an application of the super computing system.
The device is configured in a local device, responds to an IO data acquisition request, and acquires IO data of a target application based on a capture program in a kernel space and a copy program in a user space; carrying out serialization processing and compression processing on the IO data to generate compressed data corresponding to the IO data; and uploading the compressed data to a server, wherein the server is used for decompressing the compressed data and performing inverse serialization processing to obtain IO data, and analyzing the IO data by the server to obtain an analysis result of the IO data, wherein the analysis result of the IO data is used as reference data for optimizing the application of the super computing system. In this way, the IO data of the application can be acquired in the kernel space and the user space, and the granularity and the hierarchy of the IO data are reduced, so that the analysis precision of the IO data is improved, and finally the optimization effect of the super computer system application is improved.
In some embodiments, the acquisition module 410 includes:
the capturing unit is used for capturing IO data of the target application based on a capturing program in the kernel space and storing the IO data into a target storage area;
and the copying unit is used for copying the IO data in the target storage area to the user space based on the copying program in the user space.
In some embodiments, the copy unit is specifically for:
in the process that a client in the local equipment copies the IO data from the target storage area according to a first copy threshold by using a copy program in the user space, detecting whether the IO data is successfully copied to the user space;
and if the IO data is successfully copied to the user space, adjusting the first copy threshold to a second copy threshold through the client, and continuing to copy the IO data stored in the target storage area to the user space according to the second copy threshold by using a copy program in the user space, wherein the first copy threshold is smaller than the second copy threshold.
In some embodiments, the copy unit is further to: and if the IO data is not successfully copied to the user space, adjusting the first copy threshold to a third copy threshold through the client, and continuing to copy the IO data stored in the target storage area to the user space according to the third copy threshold by using a copy program in the user space, wherein the first copy threshold is larger than the third copy threshold.
In some embodiments, the generating module 420 is specifically configured to: and carrying out serialization processing and compression processing on the IO data through the proxy end in the local equipment to generate compressed data corresponding to the IO data.
In some embodiments, the analysis module 430 is specifically configured to:
and uploading the compressed data to target collection equipment in the server according to an extensible serialization structure based on a remote procedure call mode, wherein the target collection equipment is used for carrying out decompression processing and reverse serialization processing on the compressed data to obtain the IO data, and the target collection equipment stores the IO data into a database in the server according to a preset storage mode.
In some embodiments, the analysis module 430 is specifically configured to:
acquiring a plurality of collecting devices which are in communication connection with the local device from the server, and determining the residual resource quantity of each collecting device;
selecting a target collection device from the plurality of collection devices, the remaining resource amount being greater than or equal to a preset resource amount threshold;
and uploading the compressed data to target collection equipment in the server according to the extensible serialization structure based on the remote procedure call mode.
In some embodiments, the analysis device in the server determines an IO data change map based on the IO data stored in the database; and/or the number of the groups of groups,
the analysis equipment in the server determines the file system data prefetching accuracy based on the IO data stored in the database; and/or the number of the groups of groups,
the analysis equipment in the server determines the balance degree of the times of IO data acquisition requests among different nodes in the operation process based on the IO data stored in the database; and/or the number of the groups of groups,
the analysis equipment in the server determines the read-write sensitivity intensity of the operation I/O based on the IO data stored in the database; and/or the number of the groups of groups,
and the analysis equipment in the server determines the balance degree of the I/O load among different nodes in the operation process of the job based on the IO data stored in the database.
It should be noted that, the IO data analysis apparatus 400 shown in fig. 4 may perform the steps in the method embodiments shown in fig. 2 to 3, and implement the processes and effects in the method or system embodiments shown in fig. 2 to 3, which are not described herein.
Fig. 5 shows a schematic structural diagram of an IO data analysis device according to an embodiment of the present disclosure.
As shown in fig. 5, the IO data analysis device may include a processor 501 and a memory 502 storing computer program instructions.
In particular, the processor 501 may include a Central Processing Unit (CPU), or an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or may be configured to implement one or more integrated circuits of embodiments of the present application.
The processor 501 reads and executes the computer program instructions stored in the memory 502 to perform the steps of the IO data analysis method provided by the embodiments of the present disclosure.
In one example, the IO data analysis device may further include a transceiver 503 and a bus 504. As shown in fig. 5, the processor 501, the memory 502, and the transceiver 503 are connected to each other via the bus 504 and perform communication with each other.
The following is an embodiment of a computer readable storage medium provided by an embodiment of the present disclosure, where the computer readable storage medium and the IO data analysis method of each embodiment described above belong to the same inventive concept, and details of the embodiment of the computer readable storage medium are not described in detail, and reference may be made to the embodiment of the IO data analysis method described above.
The present embodiments provide a storage medium containing computer executable instructions that when executed by a computer processor are for performing a method of IO data analysis.
Of course, the storage medium containing the computer executable instructions provided in the embodiments of the present disclosure is not limited to the above method operations, but may also perform the related operations in the IO data analysis method provided in any embodiment of the present disclosure.
From the above description of embodiments, it will be apparent to those skilled in the art that the present disclosure may be implemented by means of software and necessary general purpose hardware, but may of course also be implemented by means of hardware, although in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present disclosure may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a FLASH Memory (FLASH), a hard disk, or an optical disk of a computer, where the instructions include a computer cloud platform (which may be a personal computer, a server, or a network cloud platform, etc.) to perform the IO data analysis method provided by the embodiments of the present disclosure.
Note that the above is only a preferred embodiment of the present disclosure and the technical principle applied. Those skilled in the art will appreciate that the present disclosure is not limited to the particular embodiments described herein, and that various obvious changes, rearrangements and substitutions can be made by those skilled in the art without departing from the scope of the disclosure. Therefore, while the present disclosure has been described in connection with the above embodiments, the present disclosure is not limited to the above embodiments, but may include many other equivalent embodiments without departing from the spirit of the present disclosure, the scope of which is determined by the scope of the appended claims.
Claims (12)
1. An IO data analysis method, applied to a local device, the method comprising:
responding to an IO data acquisition request, and acquiring IO data of a target application based on a capturing program in a kernel space and a copying program in a user space;
carrying out serialization processing and compression processing on the IO data to generate compressed data corresponding to the IO data;
uploading the compressed data to a server, wherein the server is used for performing decompression processing and inverse serialization processing on the compressed data to obtain the IO data, and analyzing the IO data by the server to obtain an analysis result of the IO data, wherein the analysis result of the IO data is used as reference data for optimizing the super computing system application;
The acquiring IO data of the target application based on the capturing program in the kernel space and the copying program in the user space comprises the following steps:
capturing IO data of the target application based on a capture program in the kernel space, and storing the IO data into a target storage area, wherein the target application is an application program needing to be optimized in a super computer system and comprises a calculation function, a storage function, a visualization function and a monitoring function;
and copying the IO data in the target storage area to the user space based on the copy program in the user space.
2. The method of claim 1, wherein the copying the IO data in the target storage area to the user space based on the copy program in the user space comprises:
in the process that a client in the local equipment copies the IO data from the target storage area according to a first copy threshold by using a copy program in the user space, detecting whether the IO data is successfully copied to the user space;
and if the IO data is successfully copied to the user space, adjusting the first copy threshold to a second copy threshold through the client, and continuing to copy the IO data stored in the target storage area to the user space according to the second copy threshold by using a copy program in the user space, wherein the first copy threshold and the second copy threshold are determined according to the size of the target area, and the first copy threshold is smaller than the second copy threshold.
3. The method according to claim 2, wherein the method further comprises:
and if the IO data is not successfully copied to the user space, adjusting the first copy threshold to a third copy threshold by the client, and continuing to copy the IO data stored in the target storage area to the user space according to the third copy threshold by using a copy program in the user space, wherein the first copy threshold and the third copy threshold are determined according to the size of the target area, and the first copy threshold is larger than the third copy threshold.
4. The method of claim 1, wherein the performing serializing and compressing the IO data to generate compressed data corresponding to the IO data comprises:
and carrying out serialization processing and compression processing on the IO data through the proxy end in the local equipment to generate compressed data corresponding to the IO data.
5. The method of claim 1, wherein uploading the compressed data to a server comprises:
and uploading the compressed data to target collection equipment in the server according to an extensible serialization structure based on a remote procedure call mode, wherein the target collection equipment is used for carrying out decompression processing and reverse serialization processing on the compressed data to obtain the IO data, and the target collection equipment stores the IO data into a database in the server according to a preset storage mode.
6. The method of claim 5, wherein uploading the compressed data to a target collection device in the server in an extensible serialization structure based on a remote procedure call style comprises:
acquiring a plurality of collecting devices which are in communication connection with the local device from the server, and determining the residual resource quantity of each collecting device;
selecting a target collection device from the plurality of collection devices, the remaining resource amount being greater than or equal to a preset resource amount threshold;
and uploading the compressed data to target collection equipment in the server according to the extensible serialization structure based on the remote procedure call mode.
7. The method of claim 5, wherein the step of determining the position of the probe is performed,
the analysis equipment in the server determines an IO data change chart based on the IO data stored in the database; and/or the number of the groups of groups,
the analysis equipment in the server determines the file system data prefetching accuracy based on the IO data stored in the database; and/or the number of the groups of groups,
the analysis equipment in the server determines the balance degree of the times of IO data acquisition requests among different nodes in the operation process based on the IO data stored in the database; and/or the number of the groups of groups,
The analysis equipment in the server determines the read-write sensitivity intensity of the operation I/O based on the IO data stored in the database; and/or the number of the groups of groups,
and the analysis equipment in the server determines the balance degree of the I/O load among different nodes in the operation process of the job based on the IO data stored in the database.
8. An IO data analysis apparatus, configured to a local device, the apparatus comprising:
the acquisition module is used for responding to the IO data acquisition request and acquiring IO data of the target application based on the acquisition program in the kernel space and the copy program in the user space;
the generation module is used for carrying out serialization processing and compression processing on the IO data and generating compressed data corresponding to the IO data;
the analysis module is used for uploading the compressed data to a server, wherein the server is used for carrying out decompression processing and reverse sequencing processing on the compressed data to obtain the IO data, and the server is used for analyzing the IO data to obtain an analysis result of the IO data, wherein the analysis result of the IO data is used as reference data for optimizing the application of the super computing system;
the acquisition module comprises:
The capturing unit is used for capturing IO data of the target application based on a capturing program in the kernel space and storing the IO data into a target storage area, wherein the target application is an application program needing to be optimized in a super computer system and comprises a computing function, a storage function, a visualization function and a monitoring function;
and the copying unit is used for copying the IO data in the target storage area to the user space based on the copying program in the user space.
9. An IO data analysis apparatus, characterized by comprising:
a processor;
a memory for storing executable instructions;
wherein the processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the method of any of the preceding claims 1-7.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the storage medium stores a computer program, which, when executed by a processor, causes the processor to implement the method of any of the preceding claims 1-7.
11. An IO data analysis system, comprising: a local device and a server;
The local equipment is used for responding to an IO data acquisition request, acquiring IO data of a target application based on a capture program in a kernel space and a copy program in a user space, carrying out serialization processing and compression processing on the IO data, generating compressed data corresponding to the IO data, and uploading the compressed data to a server;
the server is used for decompressing the compressed data and performing reverse serialization processing to obtain the IO data, and analyzing the IO data to obtain an analysis result of the IO data;
the local device is specifically configured to:
capturing IO data of the target application based on a capture program in the kernel space, and storing the IO data into a target storage area, wherein the target application is an application program needing to be optimized in a super computer system and comprises a calculation function, a storage function, a visualization function and a monitoring function;
and copying the IO data in the target storage area to the user space based on the copy program in the user space.
12. The system of claim 11, wherein the system further comprises a controller configured to control the controller,
the local equipment comprises a client and a proxy;
The server comprises a target collecting device, a database and an analyzing device.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310412396.XA CN116166508B (en) | 2023-04-18 | 2023-04-18 | IO data analysis method, device, equipment, storage medium and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310412396.XA CN116166508B (en) | 2023-04-18 | 2023-04-18 | IO data analysis method, device, equipment, storage medium and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116166508A CN116166508A (en) | 2023-05-26 |
| CN116166508B true CN116166508B (en) | 2023-07-07 |
Family
ID=86420403
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310412396.XA Active CN116166508B (en) | 2023-04-18 | 2023-04-18 | IO data analysis method, device, equipment, storage medium and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116166508B (en) |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1925465A (en) * | 2006-09-22 | 2007-03-07 | 中国科学院计算技术研究所 | Method for realizing data packet catching based on sharing internal memory |
Family Cites Families (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8121148B2 (en) * | 2005-03-24 | 2012-02-21 | Ixia | Protocol stack using shared memory |
| CN100464304C (en) * | 2006-08-29 | 2009-02-25 | 飞塔信息科技（北京）有限公司 | Device and method for realizing zero copy based on Linux operating system |
| CN106096455A (en) * | 2016-08-08 | 2016-11-09 | 王波 | A kind of main frame kernel data reduction protection method |
| CN108509255B (en) * | 2017-02-24 | 2021-01-22 | 龙芯中科技术股份有限公司 | Method and device for processing hardware interrupt |
| CN109992398B (en) * | 2017-12-29 | 2021-06-25 | Oppo广东移动通信有限公司 | Resource management method, resource management device, mobile terminal and computer-readable storage medium |
| US11792307B2 (en) * | 2018-03-28 | 2023-10-17 | Apple Inc. | Methods and apparatus for single entity buffer pool management |
| US11036683B2 (en) * | 2018-07-10 | 2021-06-15 | International Business Machines Corporation | Hot-pluggable file system interface |
| CN109697034B (en) * | 2018-12-19 | 2022-04-29 | 北京字节跳动网络技术有限公司 | Data writing method and device, electronic equipment and storage medium |
| CN112558858B (en) * | 2019-09-10 | 2022-11-15 | 无锡江南计算技术研究所 | Method for acquiring access characteristics of super-large-scale application external memory based on dynamic IO perception |
| CN115248764A (en) * | 2021-04-27 | 2022-10-28 | 腾讯科技（深圳）有限公司 | Method, device, equipment and storage medium for determining performance index |
| CN114422452A (en) * | 2021-12-14 | 2022-04-29 | 杭州安恒信息安全技术有限公司 | Data transmission method, device, processing equipment, storage medium and chip |
| CN114637577A (en) * | 2022-03-25 | 2022-06-17 | 阿里巴巴（中国）有限公司 | Method and device for identifying use state of instance |
| CN115033407B (en) * | 2022-08-09 | 2022-11-04 | 微栈科技(浙江)有限公司 | System and method for collecting and identifying flow suitable for cloud computing |
| CN115622772A (en) * | 2022-10-10 | 2023-01-17 | 中国银行股份有限公司 | Financial data transmission method and application gateway for financial business service |
-
2023
- 2023-04-18 CN CN202310412396.XA patent/CN116166508B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1925465A (en) * | 2006-09-22 | 2007-03-07 | 中国科学院计算技术研究所 | Method for realizing data packet catching based on sharing internal memory |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116166508A (en) | 2023-05-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107370806B (en) | HTTP status code monitoring method, device, storage medium and electronic equipment | |
| CN112988284B (en) | Application program starting method and device, computer equipment and storage medium | |
| US9965327B2 (en) | Dynamically scalable data collection and analysis for target device | |
| US20160328314A1 (en) | System and method for providing code coverage | |
| US20190042457A1 (en) | Cache (partition) size determination method and apparatus | |
| CN108292257B (en) | System and method for annotating client-server transactions | |
| CN110515808B (en) | Database monitoring method and device, computer equipment and storage medium | |
| CN111666201A (en) | Regression testing method, device, medium and electronic equipment | |
| CN110659435A (en) | Page data acquisition processing method and device, computer equipment and storage medium | |
| CN110941530A (en) | Method and device for acquiring monitoring data, computer equipment and storage medium | |
| WO2022161047A1 (en) | Event tracking analysis method and apparatus for page | |
| CN110597704A (en) | Application program pressure testing method, device, server and medium | |
| CN109697034B (en) | Data writing method and device, electronic equipment and storage medium | |
| CN116166508B (en) | IO data analysis method, device, equipment, storage medium and system | |
| US10606601B2 (en) | Method, system and non-transitory computer-readable storage medium for analyzing access to storage device | |
| CN113656391A (en) | Data detection method and device, storage medium and electronic equipment | |
| CN112307386A (en) | Information monitoring method, system, electronic device and computer readable storage medium | |
| CN114238048B (en) | Automatic testing method and system for Web front-end performance | |
| CN111226201A (en) | Memory allocation in data analysis system | |
| CN112100473A (en) | Crawler method based on application interface, terminal and storage medium | |
| CN114690988B (en) | Test method and device and electronic equipment | |
| CN107122287B (en) | System performance analysis method and device | |
| CN112631733B (en) | Data acquisition method and device | |
| Walton et al. | LDMS Darshan Connector: For Run Time Diagnosis of HPC Application I/O Performance | |
| US20240338251A1 (en) | Thread local event based profiling with performance and scaling analysis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |