CN115525603A - Storage statistics method and device, computer readable storage medium and AI device - Google Patents
Storage statistics method and device, computer readable storage medium and AI device Download PDFInfo
- Publication number
- CN115525603A CN115525603A CN202211214751.4A CN202211214751A CN115525603A CN 115525603 A CN115525603 A CN 115525603A CN 202211214751 A CN202211214751 A CN 202211214751A CN 115525603 A CN115525603 A CN 115525603A
- Authority
- CN
- China
- Prior art keywords
- counted
- directory
- size
- modification time
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003860 storage Methods 0.000 title claims abstract description 58
- 238000000034 method Methods 0.000 title claims abstract description 51
- 230000004048 modification Effects 0.000 claims abstract description 91
- 238000012986 modification Methods 0.000 claims abstract description 91
- 230000015654 memory Effects 0.000 claims description 23
- 238000004590 computer program Methods 0.000 claims description 16
- 238000012545 processing Methods 0.000 claims description 9
- 238000007619 statistical method Methods 0.000 abstract description 9
- 238000013473 artificial intelligence Methods 0.000 description 44
- 230000005540 biological transmission Effects 0.000 description 6
- 230000008859 change Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 6
- 238000012549 training Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 230000007334 memory performance Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
Abstract
The embodiment of the application provides a storage statistical method, a device, a computer readable storage medium and AI equipment, wherein the method comprises the following steps: acquiring the modification time of the current record of the catalog to be counted and the modification time of the last record of the catalog to be counted; under the condition that the modification time of the current record is the same as that of the last record and the directory to be counted comprises the subdirectories, re-counting the size of the next-level subdirectory of the directory to be counted; when the modification time of the current record is different from the modification time of the last record and the directory to be counted comprises the subdirectories, the size of the next level subdirectory of the directory to be counted and the size of the file directly subordinate to the directory to be counted are counted again, and the size of the directory to be counted is re-determined according to the size of the next level subdirectory of the directory to be counted after being counted again and the size of the file directly subordinate to the directory to be counted after being counted again. The scheme solves the problem that too much resource IO (input/output) stored by the storage statistical method is consumed.
Description
Technical Field
The embodiment of the application relates to the field of AI storage, in particular to a storage statistical method, a device, a computer readable storage medium and AI equipment.
Background
An important function of the Artificial Intelligence platform is operation management of file storage, including management of user files, management of data set files, and the like, a user can generate files in file management operation and also train generated files in business, the large amount of file generation operation consumes storage resources of a cluster, an AI (Artificial Intelligence) cluster has a very high requirement on storage, cluster storage is accompanied by frequent IO (input and output) operations, how to perform fast file statistics on massive files in cluster storage, the storage performance does not affect training tasks and other file operations of the platform, and the problem is solved primarily in the AI cluster and is concerned with the model training work efficiency of cluster users.
In some current schemes, all files are directly traversed to count the size of the file, so that stored resources IO are continuously consumed in the traversing process, and resources such as a CPU (Central Processing Unit) and a MEM (Memory) of a business service are also continuously consumed.
Disclosure of Invention
The embodiment of the application provides a storage statistical method, a storage statistical device, a computer readable storage medium and AI equipment, so as to at least solve the problem that IO (input/output) of resources stored by the storage statistical method in the related art is excessively consumed.
According to an embodiment of the present application, there is provided a storage statistic method including:
acquiring the modification time of the current record of the catalog to be counted and the modification time of the last record of the catalog to be counted;
under the condition that the modification time of the current record is the same as the modification time of the last record and the directory to be counted comprises the subdirectories, re-counting the size of the next-level subdirectory of the directory to be counted, and re-determining the size of the directory to be counted according to the re-counted size of the next-level subdirectory of the directory to be counted;
and under the condition that the modification time of the current record is different from the modification time of the last record and the directory to be counted comprises the subdirectories, re-counting the size of the next level subdirectory of the directory to be counted and the size of the file directly subordinate to the directory to be counted, and re-determining the size of the directory to be counted according to the re-counted size of the next level subdirectory of the directory to be counted and the re-counted size of the file directly subordinate to the directory to be counted.
In one exemplary embodiment, the method further comprises: and under the condition that the modification time of the current record is the same as the modification time of the last record and the directory to be counted does not comprise the subdirectories, the directory to be counted is not counted again.
In another exemplary embodiment, the method further comprises: and under the condition that the modification time of the current record is different from the modification time of the last record and the directory to be counted does not comprise subdirectories, re-counting the size of the file directly subordinate to the directory to be counted, and re-determining the size of the directory to be counted according to the size of the file directly subordinate to the directory to be counted after re-counting.
In yet another exemplary embodiment, the method further comprises: and constructing an index directory, wherein the index directory is used for representing a path from a root index directory to the directory to be counted, and the index directory is also used for representing a path from the root index directory to the file.
In yet another exemplary embodiment, the method further comprises: according to the index directory, indexing a next level subdirectory subordinate to the directory to be counted; and indexing subdirectories of the next level subdirectory subordinate to the directory to be counted according to the index directory until the files directly subordinate to the subdirectories are indexed.
In another exemplary embodiment, the method further comprises: determining the time for acquiring the modification time of the current record of the catalog to be counted as a first time; and determining the time for acquiring the modification time of the last record of the catalog to be counted as a second time, wherein the time difference between the first time and the second time is greater than or equal to one hour.
In another exemplary embodiment, the method further comprises: generating a size display control; and controlling the size display control to display the sizes of the catalogs at all levels.
In another exemplary embodiment, the type of the directory to be counted is at least one of the following: user directories, shared directories, data set directories and model directories.
According to another embodiment of the present application, there is provided a storage statistic apparatus including:
the device comprises an acquisition unit, a calculation unit and a processing unit, wherein the acquisition unit is used for acquiring the modification time of the current record of the catalog to be counted and the modification time of the last record of the catalog to be counted;
the first statistical unit is used for counting the size of the next-level subdirectory of the directory to be counted again under the condition that the modification time of the current record is the same as the modification time of the last record and the directory to be counted comprises subdirectories, and re-determining the size of the directory to be counted according to the newly counted size of the next-level subdirectory of the directory to be counted;
and the second counting unit is used for counting the size of the next level subdirectory of the directory to be counted again and the size of the file directly subordinate to the directory to be counted again under the condition that the modification time of the current record is different from the modification time of the last record and the directory to be counted comprises subdirectories, and re-determining the size of the directory to be counted again according to the counted size of the next level subdirectory of the directory to be counted again and the counted size of the file directly subordinate to the directory to be counted.
According to a further embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program, when executed by a processor, carries out the steps of any of the methods.
According to yet another embodiment of the application, there is provided an AI device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing any of the method steps when executing the computer program.
According to the method and the device, the mode for obtaining the size of the directory to be counted is adaptively adjusted according to the fact that the modification time of the current record of the directory to be counted is the same as or different from the modification time of the last record of the directory to be counted and whether the directory to be counted comprises the subdirectory. The method is an incremental counting method, and compared with a method for directly traversing all files to count the sizes of the files, the method greatly reduces the workload of counting and saves the stored resource IO.
Drawings
Fig. 1 is a block diagram of an AI device hardware structure for executing a memory statistics method according to an embodiment of the present application;
FIG. 2 is a flow chart of a method of storing statistics according to an embodiment of the present application;
FIG. 3 is a diagram of an index directory according to an embodiment of the present application;
FIG. 4 is a schematic diagram of another index directory according to an embodiment of the present application;
fig. 5 is a block diagram of a storage statistic device according to an embodiment of the present application.
Detailed Description
Embodiments of the present application will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the accompanying drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order.
For convenience of description, some terms or expressions referred to in the embodiments of the present application are explained below:
AI clustering: the cluster is used for managing and using resources such as GPU, CPU, file storage and the like, and the upper layer can support large-scale clusters of services such as AI training, reasoning and the like.
File indexing: generally referred to as an index file, where random access to records requires knowledge of the address of the record, corresponding to the index directory in this document.
Tree structure: a tree structure is a hierarchy of nested structures. The outer layer and the inner layer of a tree structure have similar structures, so the structure can be represented recursively.
The method embodiments provided in the embodiments of the present application may be executed in an AI device or similar computing device. Taking an AI device as an example, fig. 1 is a hardware structure block diagram of an AI device storing a statistical method according to an embodiment of the present application. As shown in fig. 1, the AI device may include one or more processors 102 (only one is shown in fig. 1) (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), and a memory 104 for storing data, wherein the AI device may further include a transmission device 106 for communication functions and an input-output device 108. It will be understood by those of ordinary skill in the art that the configuration shown in fig. 1 is merely illustrative and is not intended to limit the configuration of the AI apparatus described above. For example, the AI device may also include more or fewer components than shown in fig. 1, or have a different configuration than shown in fig. 1.
The memory 104 can be used for storing computer programs, for example, software programs and modules of application software, such as computer programs corresponding to the storage statistics method in the embodiment of the present application, and the processor 102 executes the computer programs stored in the memory 104, thereby executing various functional applications and data processing, i.e., implementing the method described above. The memory 104 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory remotely located from the processor 102, which may be connected to the AI devices via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission device 106 is used to receive or transmit data via a network. The specific example of the network described above may include a wireless network provided by a communication provider of the AI device. In one example, the transmission device 106 includes a Network adapter (NIC) that can be connected to other Network devices through a base station to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is used to communicate with the internet in a wireless manner.
In the present embodiment, a method for operating the AI device is provided, and fig. 2 is a flowchart according to an embodiment of the present application, where as shown in fig. 2, the flowchart includes the following steps:
step S202, acquiring the modification time of the current record of the catalog to be counted and the modification time of the last record of the catalog to be counted;
step S204, under the condition that the modification time of the current record is the same as the modification time of the last record and the directory to be counted comprises the subdirectories, re-counting the size of the next-level subdirectory of the directory to be counted, and re-determining the size of the directory to be counted according to the re-counted size of the next-level subdirectory of the directory to be counted;
as shown in fig. 3, for example, the directory to be counted is a directory a1, and the directory a1 at this time includes a directory b1, a file b2, and a file b3. That is, the directory b1 is a sub-directory of the directory a1, and if it is found that the modification time of the current record of the directory a1 is the same as the modification time of the last record, it can be determined that the files b2 and b3 are not changed, that is, the change in time is caused by the change of the file directly belonging to the directory to be counted, that is, in the case of the change of the file directly belonging to the directory to be counted, the modification time is different. However, if the modification time is the same, it cannot be determined whether the subdirectory of the directory to be counted has changed, that is, it is not determined whether the size of the directory b1 has changed, and at this time, the size of the directory b1 needs to be counted again.
Step S206, under the condition that the modification time of the current record is different from the modification time of the last record and the directory to be counted comprises the subdirectories, re-counting the size of the next level subdirectory of the directory to be counted and the size of the file directly subordinate to the directory to be counted, and re-determining the size of the directory to be counted according to the re-counted size of the next level subdirectory of the directory to be counted and the re-counted size of the file directly subordinate to the directory to be counted.
As shown in fig. 3, for example, the directory to be counted is the directory b1, and if the modification time of the current record of the directory b1 is found to be different from the modification time of the last record, then it can be determined that one or both of the files c2 and c3 directly subordinate to the directory b1 have changed. But it is uncertain whether the size of the directory c1 subordinate to the directory b1 has changed. Therefore, in this case, it is necessary to count not only the size of the file but also the size of the directory.
Through the steps, firstly, the modification time of the current record of the directory to be counted and the modification time of the last record of the directory to be counted are obtained, then, under the condition that the modification time of the current record is the same as the modification time of the last record and the directory to be counted comprises subdirectories, the size of the subdirectory of the next level of the directory to be counted is newly counted, the size of the directory to be counted is re-determined according to the newly counted size of the subdirectory of the next level of the directory to be counted, finally, under the condition that the modification time of the current record is different from the modification time of the last record and the directory to be counted comprises subdirectories, the size of the subdirectory of the next level of the directory to be counted and the size of the file directly subordinate to the directory to be counted are newly counted, and the size of the directory to be counted is re-determined according to the newly counted size of the subdirectory of the next level of the directory to be counted and the newly counted size of the file directly subordinate to the directory to be counted. And adaptively adjusting the mode of obtaining the size of the directory to be counted according to the fact that the modification time of the current record of the directory to be counted is the same as or different from the modification time of the last record of the directory to be counted and whether the directory to be counted comprises the subdirectory. Compared with the mode of directly traversing all files to count the sizes of the files, the mode of incremental statistics greatly reduces the workload of statistics and saves the stored resource IO.
In some embodiments, the above method further comprises: and under the condition that the modification time of the current record is the same as that of the last record and the directory to be counted does not comprise the subdirectories, the directory to be counted is not counted again.
As shown in fig. 4, for example, the directory to be counted is directory a2, the modification time of the current record of directory a2 is the same as the modification time of the last record, and directory a2 only includes file e1 and file e2, where it is stated that both file e1 and file e2 are changed, so it is not necessary to count the size of directory a2 again at this time.
In other embodiments, the method further comprises: and under the condition that the modification time of the current record is different from the modification time of the last record and the directory to be counted does not comprise subdirectories, re-counting the size of the file directly subordinate to the directory to be counted, and re-determining the size of the directory to be counted according to the re-counted size of the file directly subordinate to the directory to be counted.
As shown in fig. 4, for example, the directory to be counted is directory a2, the modification time of the current record of directory a2 is different from the modification time of the last record, and directory a2 only includes file e1 and file e2, so it is unclear at this time that the difference in time is caused by the change of file e1, the change of file e2, or the change of both file e1 and file e2, and therefore, the sizes of file e1 and file e2 need to be counted again at this time.
In other embodiments, the method further comprises: and constructing an index directory, wherein the index directory is used for representing a path from the root index directory to the directory to be counted, and the index directory is also used for representing a path from the root index directory to the file. Fig. 3 shows an index directory, and fig. 4 shows an index directory.
Further, the above further includes: according to the index directory, indexing a next level subdirectory belonging to the directory to be counted; and indexing the subdirectories subordinate to the next level subdirectory of the directory to be counted according to the index directory until the files directly subordinate to the subdirectory are indexed.
A more specific embodiment relates to a specific memory statistics method, comprising the steps of:
step S1: and constructing an index directory, wherein the index directory is used for representing a path from the root index directory to the directory to be counted, and the index directory is also used for representing a path from the root index directory to the file.
As shown in fig. 3, the constructed index directory includes: a1/b1, a1/b1/c2, a1/b1/c3, a1/b1/c2/d1, a1/b1/c2/d2, a1/b1/c2/d3, a1/b2 and a1/b3.
As shown in fig. 4, the constructed index directory includes: a1/b1, a1/b2, a1/b1/c1, a1/b1/c2, a1/b1/c3/d1, a1/b1/c3/d2, a1/b1/c3/d3, a2/e1 and a2/e2.
Step S2: the initial sizes of the directories at all levels are obtained according to the index directories, and a specific description is shown in table 1, wherein the initial size, the modification time and the owner of the current record are recorded in table 1, 2022161600 indicates 16 in 2022. 111 to an owner, 222 to an owner, 333 to an owner, 444 to an owner, 555 to an owner, 666 to an owner, 777 to an owner, 888 to an owner, 999 to an owner.
TABLE 1 detailed information Table of catalog of each level
And step S3: judging whether the modification time of the current record of each level of directory is the same as the modification time of the last record or not according to the index directory, then determining the size of each level of directory according to the same or different time, for example, for the index directories a1/b1/c2/d2, determining whether the time of a1 is the same or not, then carrying out size statistics, then determining whether the time of b1 is the same or not, then determining whether the time of c2 is the same or not, then carrying out size statistics, if d2 is a file, counting is ended at d2, if d2 is not a file, then continuing to count the files subordinate to d2, namely counting from the root directory to the smallest file level.
Specifically, according to the scheme, a tree index directory similar to a hash is constructed according to the tree hierarchy structure of the file directory. For example, searching and updating a storage path/a/b/c/d, wherein the storage path/a needs to be found, then the storage path/a/b is found, then the storage path/a/b/c/d is obtained, and finally the storage information of the storage path/a/b/c/d is obtained.
In some embodiments, the above method further comprises: determining the time for acquiring the modification time of the current record of the catalog to be counted as a first time; and determining the time for acquiring the modification time of the last record of the catalog to be counted as a second time, wherein the time difference between the first time and the second time is greater than or equal to one hour. The time difference between the first time and the second time is set to be one hour or several hours, so that resources are greatly saved and more resources are released compared with a mode of counting once for several seconds or several minutes in the related art.
Still further, the above method further comprises: generating a size display control; and controlling the size display control to display the sizes of the catalogs at all levels. The size of each level of directory can be intuitively obtained.
Specifically, the first display line may be displayed at the same height as the size of the directory, the second display line may be displayed at the same height as the second level directory, the third display line may be displayed at the same height as the third level directory, and so on.
Specifically, the type of the directory to be counted is at least one of the following types: user directories, shared directories, data set directories and model directories. The user directory refers to an AI platform user home directory (a directory which can only be accessed by the user), the user creates an AI task, selects a file and a data set in the user directory, and a task generation model and a generation log are both included in the user directory; the shared directory refers to a public directory to which a user or a system administrator shares files, and the directory can be accessed by the user; the data set catalog is a catalog only storing data sets, and the data volume of the data set catalog is very large; the model catalog refers to a catalog for managing models for managing the model generated by the task. Of course, directories other than user directories, shared directories, data set directories, model directories may also be included. The setting can be performed by those skilled in the art according to actual conditions.
The main body for executing the above steps may be an AI device cluster, but is not limited thereto.
The method in the embodiment can meet the requirement of enterprises and scientific research and university on computing power, is applied to AI equipment clusters, can realize fast file statistics on massive files by cluster storage by adopting the scheme of the application, reduces IO operations, and does not influence training tasks and other file operations of a platform on storage performance. In addition, compared with a mode of directly traversing all files to count the file sizes, resources such as CPUs (central processing units), MEMs (memory access units) and the like consuming business services can be reduced, and meanwhile, a more ideal counting result is achieved. Aiming at the statistics of massive files (above TB level), the statistical delay is short and can be ignored, and the error of the statistical result is small.
In addition, at present, an AI cloud manufacturer faces to individual customers, because storage of the AI cloud manufacturer belongs to management of the AI cloud manufacturer, and uploading entries of all storage files are on a storage server of the AI cloud manufacturer, the storage size statistical method of the service scene is simpler than a method of directly traversing all files to perform file size statistics, and only the files need to be uploaded and deleted to update the file sizes in the storage process of a user, and meanwhile, a storage directory of the user is continuously monitored, so that a real-time statistical result can be obtained. The present application overcomes this drawback.
Another existing storage statistical method is a method for providing a quota by storage, for example, file systems such as nfs and beegfs, etc., the quota has a very obvious advantage, high statistical speed and accurate statistical result, but is only limited to the size statistics of stored users, and the disadvantage is very obvious, and firstly, the size of a directory level cannot be counted; secondly, the storage support is single, and multiple storages are not supported for mounting an AI scene; thirdly, the size of the public directory and the size of the data set directory cannot be counted for the service usage size in the AI scene; finally, the quota scheme has more or less some impact on memory performance. For the size statistics of mass files stored privately in the AI scene, the consumption of networks, disks, CPUs and memories for storing resources is very large, and the method is not suitable for the AI scene. And task training of the AI platform is also very costly to the stress of storage. The scheme of the application can be just applied to AI scenes, and the defect is overcome.
The scheme of the application is particularly suitable for carrying out the statistics of the directory size of massive files, and has the following advantages: the method comprises the steps that firstly, different AI service scenes can be supported, and user directories, public directories and data set directories in the AI are subjected to rapid size statistics; the second proposed incremental statistical method is to prevent the storage resources from being consumed due to frequent cyclic traversal storage, and meanwhile, improve the efficiency of storage statistics; and thirdly, establishing indexes for storing each directory, and rapidly traversing the size of each directory. Meanwhile, the method provided by the scheme occupies very little network and disk IO resources, and reduces the pressure of AI cluster storage. The overall resource utilization rate of the AI cluster is improved, the file operation of the storage nodes is not influenced, the AI training task of the computing nodes is not influenced, and the pressure of file operation of the AI cluster management nodes is effectively relieved.
It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer-executable instructions and that, although a logical order is illustrated in the flowcharts, in some cases, the steps illustrated or described may be performed in an order different than here.
Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present application.
In this embodiment, a storage statistics apparatus is further provided, and the apparatus is used to implement the foregoing embodiments and preferred embodiments, which have already been described and are not described again. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
Fig. 5 is a block diagram of a storage statistic device according to an embodiment of the present application, and as shown in fig. 5, the device includes:
an obtaining unit 51, configured to obtain a modification time of a current record of a to-be-counted directory and a modification time of a last record of the to-be-counted directory;
a first statistics unit 52, configured to, when the modification time of the current record is the same as the modification time of the last record and the directory to be counted includes a subdirectory, newly count the size of a next-level subdirectory of the directory to be counted, and re-determine the size of the directory to be counted according to the newly counted size of the next-level subdirectory of the directory to be counted;
a second counting unit 53, configured to, when the modification time of the current record is different from the modification time of the last record and the to-be-counted directory includes a sub-directory, re-count the size of the sub-directory of the to-be-counted directory at the next level and the size of the file directly belonging to the to-be-counted directory, and re-determine the size of the to-be-counted directory according to the re-counted size of the sub-directory of the to-be-counted directory at the next level and the re-counted size of the file directly belonging to the to-be-counted directory.
The storage statistic device adaptively adjusts the size of the directory to be counted according to the fact that the modification time of the current record of the directory to be counted is the same as or different from the modification time of the last record of the directory to be counted, and whether the directory to be counted comprises the subdirectory or not. Compared with the mode of directly traversing all files to count the sizes of the files, the mode of incremental statistics greatly reduces the workload of statistics and saves the stored resource IO.
In some embodiments, the apparatus further includes a processing unit, where the processing unit is configured to not perform statistics again on the directory to be counted if the modification time of the current record is the same as the modification time of the last record and the directory to be counted does not include a sub-directory.
In some embodiments, the apparatus further includes a third statistical unit, where the third statistical unit is configured to, when the modification time of the current record is different from the modification time of the last record and the directory to be counted does not include a subdirectory, re-count the size of the file directly belonging to the directory to be counted, and re-determine the size of the directory to be counted according to the re-counted size of the file directly belonging to the directory to be counted.
In addition, the device further comprises a construction unit, wherein the construction unit is used for constructing an index directory, the index directory is used for representing a path from the root index directory to the directory to be counted, and the index directory is also used for representing a path from the root index directory to the file.
Further, the device also comprises a first index unit and a second index unit, wherein the first index unit is used for indexing a next-level subdirectory subordinate to the directory to be counted according to the index directory; the second indexing unit is used for indexing the subdirectory subordinate to the next level subdirectory of the directory to be counted according to the index directory until the file subordinate to the subdirectory is directly indexed.
Further, the device further comprises a first determining unit and a second determining unit, wherein the first determining unit is used for determining the time for acquiring the modification time of the current record of the catalog to be counted as a first time; the second determining unit is configured to determine a time when the modification time of the last record of the to-be-counted directory is obtained as a second time, where a time difference between the first time and the second time is greater than or equal to one hour.
Further, the device also comprises a generating unit and a control unit, wherein the generating unit is used for generating the size display control; the control unit is used for controlling the size display control to display the sizes of the catalogs at all levels.
It should be noted that, the above modules may be implemented by software or hardware, and for the latter, the following may be implemented, but not limited to: the modules are all positioned in the same processor; alternatively, the modules are respectively located in different processors in any combination.
Embodiments of the present application further provide a computer-readable storage medium having a computer program stored therein, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.
In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to: various media capable of storing computer programs, such as a usb disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk.
Embodiments of the present application also provide an AI device comprising a memory having a computer program stored therein and a processor configured to execute the computer program to perform the steps of any of the above method embodiments.
In an exemplary embodiment, the AI device may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.
For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary embodiments, and details of this embodiment are not repeated herein.
It will be apparent to those skilled in the art that the various modules or steps of the present application described above may be implemented using a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and they may be implemented using program code executable by the computing devices, such that they may be stored in a memory device and executed by the computing devices, and in some cases, the steps shown or described may be performed in an order different than that described herein, or they may be separately fabricated into separate integrated circuit modules, or multiple ones of them may be fabricated into a single integrated circuit module. Thus, the present application is not limited to any specific combination of hardware and software.
The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the principle of the present application shall be included in the protection scope of the present application.
Claims (11)
1. A memory statistics method, comprising:
acquiring the modification time of the current record of the catalog to be counted and the modification time of the last record of the catalog to be counted;
when the modification time of the current record is the same as the modification time of the last record and the directory to be counted comprises subdirectories, re-counting the size of the next-level subdirectory of the directory to be counted, and re-determining the size of the directory to be counted according to the re-counted size of the next-level subdirectory of the directory to be counted;
and under the condition that the modification time of the current record is different from the modification time of the last record and the to-be-counted directory comprises subdirectories, re-counting the size of the next-level subdirectory of the to-be-counted directory and the size of the file directly subordinate to the to-be-counted directory, and re-determining the size of the to-be-counted directory according to the re-counted size of the next-level subdirectory of the to-be-counted directory and the re-counted size of the file directly subordinate to the to-be-counted directory.
2. The method of claim 1, further comprising:
and under the condition that the modification time of the current record is the same as the modification time of the last record and the directory to be counted does not comprise the subdirectories, the directory to be counted is not counted again.
3. The method of claim 1, further comprising:
and under the condition that the modification time of the current record is different from the modification time of the last record and the directory to be counted does not comprise subdirectories, re-counting the size of the file directly subordinate to the directory to be counted, and re-determining the size of the directory to be counted according to the size of the file directly subordinate to the directory to be counted after re-counting.
4. The method of claim 1, further comprising:
and constructing an index directory, wherein the index directory is used for representing a path from a root index directory to the directory to be counted, and the index directory is also used for representing a path from the root index directory to the file.
5. The method of claim 4, further comprising:
indexing a next level subdirectory subordinate to the directory to be counted according to the index directory;
and indexing the subdirectories subordinate to the next level subdirectory of the directory to be counted according to the index directory until the files directly subordinate to the subdirectory are indexed.
6. The method of claim 1, further comprising:
determining the time for acquiring the modification time of the current record of the catalog to be counted as a first time;
and determining the time for acquiring the modification time of the last record of the catalog to be counted as a second time, wherein the time difference between the first time and the second time is greater than or equal to one hour.
7. The method according to any one of claims 1 to 6, further comprising:
generating a size display control;
and controlling the size display control to display the size of each level of directory.
8. The method according to any one of claims 1 to 6, wherein the type of the directory to be counted is at least one of:
user directories, shared directories, data set directories, model directories.
9. A memory statistics device, comprising:
the device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring the modification time of the current record of the catalog to be counted and the modification time of the last record of the catalog to be counted;
the first counting unit is used for counting the size of the next-level subdirectory of the directory to be counted again under the condition that the modification time of the current record is the same as the modification time of the last record and the directory to be counted comprises subdirectories, and re-determining the size of the directory to be counted according to the size of the next-level subdirectory of the directory to be counted;
and the second counting unit is used for counting the size of the next level subdirectory of the directory to be counted again and the size of the file directly subordinate to the directory to be counted again under the condition that the modification time of the current record is different from the modification time of the last record and the directory to be counted comprises subdirectories, and re-determining the size of the directory to be counted again according to the counted size of the next level subdirectory of the directory to be counted again and the counted size of the file directly subordinate to the directory to be counted again.
10. A computer-readable storage medium, in which a computer program is stored, which computer program, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 8.
11. AI device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the method as claimed in any of claims 1 to 8 when executing the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211214751.4A CN115525603A (en) | 2022-09-30 | 2022-09-30 | Storage statistics method and device, computer readable storage medium and AI device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211214751.4A CN115525603A (en) | 2022-09-30 | 2022-09-30 | Storage statistics method and device, computer readable storage medium and AI device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115525603A true CN115525603A (en) | 2022-12-27 |
Family
ID=84701412
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211214751.4A Pending CN115525603A (en) | 2022-09-30 | 2022-09-30 | Storage statistics method and device, computer readable storage medium and AI device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115525603A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116089364A (en) * | 2023-04-11 | 2023-05-09 | 山东英信计算机技术有限公司 | Storage file management method and device, AI platform and storage medium |
-
2022
- 2022-09-30 CN CN202211214751.4A patent/CN115525603A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116089364A (en) * | 2023-04-11 | 2023-05-09 | 山东英信计算机技术有限公司 | Storage file management method and device, AI platform and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108683720B (en) | Container cluster service configuration method and device | |
| CN110147411A (en) | Method of data synchronization, device, computer equipment and storage medium | |
| CN110555012B (en) | Data migration method and device | |
| CN100484017C (en) | Method for statistics of mass performance data in network element management system | |
| CN103209087B (en) | Distributed information log statistical processing methods and system | |
| CN110413599A (en) | Generating date and storage system and method | |
| CN111522636B (en) | Application container adjusting method, application container adjusting system, computer readable medium and terminal device | |
| CN104317800A (en) | Hybrid storage system and method for mass intelligent power utilization data | |
| CN108874803B (en) | Data storage method, device and storage medium | |
| Lai et al. | Towards a framework for large-scale multimedia data storage and processing on Hadoop platform | |
| US9449071B2 (en) | Providing reconstructed data based on stored aggregate data in response to queries for unavailable data | |
| CN113010565B (en) | Server real-time data processing method and system based on server cluster | |
| WO2021051625A1 (en) | Data table construction method, apparatus and device, and storage medium | |
| CN108848132B (en) | Power distribution scheduling main station system based on cloud | |
| CN106815260B (en) | Index establishing method and equipment | |
| CN114722092A (en) | Time series data processing method, system, storage medium and electronic equipment | |
| CN115525603A (en) | Storage statistics method and device, computer readable storage medium and AI device | |
| CN103530335A (en) | In-stockroom operation method and device of electric power measurement acquisition system | |
| CN105162622A (en) | Storage method and system | |
| CN108430067A (en) | A kind of Internet service mass analysis method and system based on XDR | |
| CN116303246A (en) | Storage increment statistical method, device, computer equipment and storage medium | |
| CN112667747B (en) | Dynamic configuration multi-database distributed persistence method supporting user-defined plug-in | |
| CN113055490A (en) | Data storage method and device | |
| CN114020214A (en) | Storage cluster capacity expansion method and device, electronic equipment and readable storage medium | |
| CN113419938A (en) | Control method, device and equipment for user concurrent access |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |