CN115964353A - Distributed file system and access metering method thereof - Google Patents
Distributed file system and access metering method thereof Download PDFInfo
- Publication number
- CN115964353A CN115964353A CN202310247122.XA CN202310247122A CN115964353A CN 115964353 A CN115964353 A CN 115964353A CN 202310247122 A CN202310247122 A CN 202310247122A CN 115964353 A CN115964353 A CN 115964353A
- Authority
- CN
- China
- Prior art keywords
- metadata
- metering
- node
- slave
- metadata node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a distributed file system and an access metering method thereof, wherein the distributed file system comprises metadata nodes and data nodes, the metadata nodes maintain metadata corresponding to data stored in the data nodes, and the method comprises the following steps: a corresponding sub-process is derived on the basis of the main process of the metadata node, and the sub-process realizes corresponding metering operation, so that additional metering node deployment is avoided, equipment resources are saved, and metering cost is reduced; the metadata maintained by the main process is obtained through the sub-process, the obtained metadata is traversed to generate a metering result of the access behavior of the user aiming at the data node, and a log does not need to be additionally and independently pulled, so that the metering result can be obtained in real time, and the timeliness of the metering result is guaranteed; the time for exporting the metering result is shortened, and the metering efficiency is improved.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a distributed file system and an access metering method thereof.
Background
A distributed file system is a system that uses a cluster of servers (or computers) to store a large number of files, logically the files form a unified file system that is pre-programmed, and is also referred to as a "cluster file system". Currently, the popular Distributed File systems include HDFS (Hadoop Distributed File System), NAS (Network attached Storage), NFS (Network File System), moose fs (moose File System), and the like. After providing services to users, the distributed file system has a need for metering access behaviors of users to data nodes. In the related technology, an off-line metering node needs to be additionally deployed on the basis of a distributed file system, a corresponding metering process is configured on a main metadata node, the metering process periodically pulls a relevant log from the main metadata node, analyzes the log and sends the log to the off-line metering node, and finally the metering node calculates data and generates a metering result of an access behavior of a user to the data node.
However, in the access metering scheme of the distributed file system mentioned in the above related art, because the metering node needs to be additionally configured, which is equivalent to occupy additional device resources, the metering cost is increased; moreover, because the metering node is deployed off-line, generally calculated on a daily basis, the timeliness of the metering result cannot meet the requirements of users; meanwhile, the log pulling, loading, analyzing and transmitting can take a long time, so that the measurement result can be obtained within hours, and the overall measurement efficiency is influenced.
Disclosure of Invention
In order to overcome the problems in the related art, the application provides a distributed file system and an access metering method thereof.
According to a first aspect of an embodiment of the present application, there is provided an access metering method for a distributed file system, where the distributed file system includes metadata nodes and data nodes, and the metadata nodes maintain metadata corresponding to data stored in the data nodes, and the method includes:
deriving corresponding sub-processes on the basis of the main process of the metadata node;
and acquiring the metadata maintained by the main process through the sub-process, and traversing the acquired metadata to generate a metering result of the access behavior of the user to the data node.
According to a second aspect of embodiments of the present application, there is provided a distributed file system, including:
a data node;
a metadata node, where the metadata node maintains metadata corresponding to data stored in the data node, and is configured to implement the steps of the method according to the first aspect.
According to a third aspect of embodiments of the present application, there is provided an electronic apparatus, including:
a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the steps of the method of the first aspect.
According to a fourth aspect of embodiments herein, there is provided a computer-readable storage medium having stored thereon executable instructions; wherein the instructions, when executed by the processor, perform the steps of the method of the first aspect.
In the embodiment of the application, the corresponding sub-process is derived on the basis of the main process of the original metadata node, and the sub-process realizes the corresponding metering operation, so that the metering node is prevented from being additionally deployed, the equipment resource is saved, and the metering cost is reduced; in addition, because the sub-process can directly acquire the metadata maintained by the main process, the log does not need to be additionally and independently pulled, the metering result can be obtained in real time, and the timeliness of the metering result is ensured; the time for exporting the metering result is shortened, and the metering efficiency is improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.
FIG. 1 is a schematic diagram of an architecture of a distributed file system provided in an exemplary embodiment of the present application;
FIG. 2 is a flowchart of an access metering method for a distributed file system according to an exemplary embodiment of the present application;
FIG. 3 is a diagram of a directory tree provided by an exemplary embodiment of the present application;
FIG. 4 is a diagram illustrating a method for spawning a child process from a metadata node for metering, according to an exemplary embodiment of the present application;
FIG. 5 is a flow diagram of a master metadata node polling slave metadata nodes according to an exemplary embodiment of the present application;
FIG. 6 is a flow chart of a round of scheduling provided by an exemplary embodiment of the present application;
FIG. 7 is a schematic illustration of a metering failure provided by an exemplary embodiment of the present application;
FIG. 8 is a schematic diagram of a metering result write cluster according to an exemplary embodiment of the present application;
FIG. 9 is a schematic block diagram of an electronic device provided in an exemplary embodiment of the present application;
fig. 10 is a block diagram of an access metering device of a distributed file system according to an exemplary embodiment of the present application.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings in which the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of one or more embodiments of the application.
It should be noted that: in other embodiments, the steps of the respective methods are not necessarily performed in the order shown and described herein. In some other embodiments, the method may include more or fewer steps than those described herein. Moreover, individual steps described in this application may, in other embodiments, be divided into multiple steps for description; multiple steps described in this application may be combined into a single step in other embodiments. It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present application. The word "if" as used herein may be interpreted as "at" \8230; "or" when 8230; \8230; "or" in response to a determination ", depending on the context.
User information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, presented data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by various parties, and the collection, use and processing of the relevant data requires compliance with relevant laws and regulations and standards in relevant countries and regions, and is provided with corresponding operation entrances for the user to choose authorization or denial.
The overall architecture of a distributed file system can generally be divided into three parts: when a user needs to initiate write once, the Client can create a file to the Master and open the file, and the Master can select the positions of three copies and feed back the positions to the Client. And the Client finds the data node according to the positions of the three copies and writes the data in. Logically functionally, it can be considered that: the Client performs integral control, the Master provides storage of metadata, and the data nodes provide storage of data. Therefore, the online storage service with high cost performance can be provided for users based on the cooperation of the three parts. Furthermore, the distributed file system also has a metering requirement of a user for the use condition of the online storage service, and metering refers to acquiring information such as file sizes and numbers of different file types, different copies and different media in a specified directory in the distributed file system. For example, the user may be billed based on the metering results, the storage space of the distributed file system may be optimized based on the metering results, and so on.
Thus, in the related art, a solution for additionally configuring an offline metering node is proposed. Specifically, a metering process running on a principal metadata node may pull a Checkpoint from a Master end, parse and generate a corresponding directory List (List) by loading the Checkpoint, transmit the List as an output to an offline additionally deployed metering node, and perform calculation by the metering node to obtain a final metering result. However, in the above solution, because the metering node needs to be configured additionally, it is equivalent to occupy additional device resources, and the metering cost is increased; moreover, because the metering node is deployed off-line, generally calculated on a daily basis, the timeliness of the metering result cannot meet the requirements of the user; meanwhile, the measurement process takes a long time for pulling, analyzing and List transmission of the log, so that the measurement result can be obtained within several hours, and the overall measurement efficiency is affected.
In view of the above, the present application provides an improved access metering scheme for a distributed file system to solve the problems in the related art, which is described in detail below with reference to the accompanying drawings.
Fig. 1 is a schematic diagram of an architecture of a distributed file system provided in an exemplary embodiment of the present application, where the architecture may include a client 11, metadata nodes (e.g., a master metadata node 12 and a slave metadata node 13), and at least one data node (e.g., a data node 14, a data node 15, and a data node 16).
The user may initiate access to the data node through the client 11, thereby completing operations such as writing or reading data. The form of the client 11 may be various, for example, it may be a web client or a mobile client, etc., which is not limited in this application.
The metadata node (Master) may be deployed on a separate host physical server, or the metadata node may be deployed on a virtual server (e.g., a cloud server) hosted by the host cluster. Of course, although in the embodiment shown in fig. 1, the master metadata node 12 and the slave metadata node 13 are respectively disposed on different physical devices, in some embodiments, the master metadata node 12 and the slave metadata node 13 may be disposed on the same physical device, which is not limited in this application. It is understood that the distributed file system may be in the form of a Master and multiple slaves, that is, the number of slave metadata nodes (slave masters) may be multiple, such as 2 slave metadata nodes or 3 slave metadata nodes, and the like, which is not limited in this application. Based on the form of one master and multiple slaves, even if a certain master metadata node fails, any slave metadata node can take over, so that high availability of the cluster is ensured, and service interruption is avoided. Furthermore, a corresponding sub-process can be derived on the basis of the main process of the metadata node, so that the measurement operation is performed through the derived sub-process, the additional arrangement of the measurement node is avoided, the equipment resource is saved, and the measurement cost is reduced. In addition, due to the parent-child relationship between the child process and the main process, the child process can directly acquire the metadata maintained by the main process, and the log does not need to be additionally and independently pulled, so that the metering result can be obtained in real time, and the timeliness of the metering result is ensured; the time for exporting the metering result is shortened, and the metering efficiency is improved.
The data node may be a physical server deployed on an independent host, or the data node may be deployed on a virtual server (e.g., a cloud server) hosted by a cluster of hosts. Each data node can maintain a corresponding data copy, for example, 3 data nodes can maintain 3 data copies, so that the safety and reliability of data stored by a user are ensured based on the backup effect of the copies. The storage media of different data nodes may be different, for example, the storage media of the data node 14 is an SSD (solid state disk), and the storage media of the data node 15 is an HDD (mechanical hard disk).
Based on the distributed file system architecture, the corresponding sub-process is derived on the basis of the main process of the original metadata node, and the sub-process realizes the corresponding metering operation, so that the metering node is prevented from being additionally deployed, the equipment resource is saved, and the metering cost is reduced; in addition, because the sub-process can directly acquire the metadata maintained by the main process, the log does not need to be additionally and independently pulled, the metering result can be obtained in real time, and the timeliness of the metering result is ensured; the time for exporting the metering result is shortened, and the metering efficiency is improved.
Fig. 2 is a flowchart of an access metering method of a distributed file system according to an exemplary embodiment of the present application, where the distributed file system may include a metadata node and a data node, and the metadata node maintains metadata corresponding to data stored in the data node, and specifically includes the following steps:
The metadata node (Master) is mainly used for providing storage of metadata, so that functions such as cluster control, directory tree, file-to-data node mapping and the like are realized. A directory tree is a tree structure of directories and files of a distributed file system, which may be recorded on a Master node. As shown in fig. 3, fig. 3 is a schematic diagram of a directory tree according to an exemplary embodiment of the present application. Starting from the root directory ("/"), there may be two directories ("a/" and "b/") below, and 1 file ("file") below the corresponding target "a/". Therefore, corresponding metering results, such as storage media, storage usage, file quantity, directory quantity and the like, can be obtained based on traversing the directory tree recorded in the metadata node.
In order to generate a measurement result of the access behavior of a user with respect to a data node, a corresponding sub-process may first be spawned (fork) on the basis of a main process of a metadata node. The Fork function is mainly used for creating a new process, a process calling the Fork function can be called a parent process, the new process can be called a child process, and the created process (the child process) copies all user space information such as code segments and data segments of the parent process. The fork function is also referred to as a duplication, cloning function, etc. The derived sub-process can share a memory space with the main process, and the fork function can update the memory in a Copy-On-Write (Copy On Write) manner during the execution process. As previously described, when a fork function is called, the kernel may copy the entire address space of the parent process as is and assign the copied copy to the child process. But this action is very time consuming because it requires: allocating pages for page tables of the child processes, allocating pages for pages of the child processes, initializing page tables of the child processes, and copying pages of the parent processes into corresponding pages of the child processes. Creating an address space involves many memory accesses and in most cases it is often pointless to do so because many sub-processes start their execution by loading a new program, thus completely discarding the inherited address space. After Copy-On-Write (Copy On Write) is used, the kernel does not Copy the address space of the whole process, but allows the parent and child processes to share the same address space. The address space is only copied when a write is required, so that each process has its own address space. Specifically, for example, in metering, after receiving an Oplog sent by a Master metadata node (Master), a host process copies a memory of a corresponding Page (Page) only when the memory is modified, and the service of the host process is not terminated. Based on the copy-on-write mode, a large amount of data which cannot be used at all can be prevented from being copied, system resources are saved, and due to the sharing of memory space, the process of additionally pulling logs (such as Oplog, checkpoint and the like) independently is avoided, and the calculation efficiency of metering can be accelerated. Of course, it is understood that the creating manner of the sub-process is not limited to the above fork function, and it may also be created in other manners, which is not limited in this application.
As previously described, to ensure high availability of the cluster, the metadata nodes may include a Master metadata node (Master) and at least one slave metadata node (slave Master), so that even if the Master is abnormal, the slave Master can take over to ensure stable service. So that the corresponding sub-process can be derived from the master process of the master metadata node or any slave metadata node. For example, taking the embodiment shown in fig. 1 as an example, the corresponding sub-process may be derived from the main metadata node 12, or the corresponding sub-process may be derived from the sub-metadata node 13, which is not limited in this application. Of course, since the Master needs to provide external services, if the Master performs the metering operation, it may cause interference to the services and cause the services to be stuck, so as to avoid the phenomenon of stuck to the external services, it is also possible to perform the metering operation only by deriving the sub-process from the Master. Fig. 4 is a schematic diagram illustrating a sub-process derived from a metadata node for metering according to an exemplary embodiment of the present application. The metadata nodes may include a Master metadata node (Master), slave metadata nodes (slave Master-1 and slave Master-2), and at this time, corresponding child processes may be derived (fork) On the basis of a Master process of the slave Master-2, and the child processes are updated in the Copy-On-Write (Copy-On-Write) manner, so that a large amount of data that cannot be used at all can be prevented from being copied, and system resources are saved. Of course, as mentioned above, the Master may also derive a sub-process to perform the metering operation, which is not limited in this application. It should be noted that, although in the embodiment shown in fig. 4, the sub-process may be created by executing the fork function, and the memory space is shared between the sub-process and the main process. However, in some embodiments, the sub-process may be created in other manners, and correspondingly, the sub-process may also obtain the metadata maintained by the main process in other manners, not necessarily by way of sharing the memory space.
The number of the slave metadata nodes may be multiple, for example, it may be 3, 5 or 10, and assuming that the sub-process is derived from only the slave metadata node (as shown in fig. 4), in order to make the metering smooth and avoid the metering confusion caused by the participation of multiple slave nodes in the metering, the master metadata node may select one slave metadata node from the multiple slave metadata nodes, so that the selected slave metadata node performs the metering operation. Specifically, the primary metadata node may be selected by the following manner: polling each slave metadata node by the master metadata node to judge whether each slave metadata node is metering; and under the condition that the judgment result shows that all the slave metadata nodes are not metered, randomly selecting one slave metadata node as any slave metadata node. As shown in fig. 5, fig. 5 is a flowchart of polling each slave metadata node by a master metadata node according to an exemplary embodiment of the present application, and the polling of the master metadata node may include the following steps:
step 502, master inquires whether slave Master-1 is metering.
Step 504, the slave Master-1 feeds back to the Master Master that it is not metering.
Step 506, the Master asks the slave Master-2 whether metering is in progress.
Step 508, the slave Master-1 feeds back to the Master Master that it is not metering.
Under the condition that the number of the slave metadata nodes of the cluster is 2 (such as slave Master-1 and slave Master-2), based on the steps 502 to 508, the Master Master can determine that none of the slave masters is metering, and therefore, the Master Master can randomly select one slave Master as the slave Master to meter. For example, the Master Master selects the slave Master-2.
Step 510, the Master initiates metering to the slave Master-2.
And after receiving the metering request initiated by the Master, the slave Master-2 can derive a sub-process on the basis of the Master process, so that the metering operation is carried out based on the sub-process. While the Master may continue to query, performing step 512.
Step 512, master ask the slave Master whether slave Master-1 is metering.
Step 514, the slave Master-1 feeds back to the Master Master that it is not metering.
In step 516, the Master queries whether the slave Master-2 is metering.
Step 518, the slave Master-2 feeds back to the Master Master that it is metering.
In the polling round, the Master does not need to select the slave Master because the slave Master-2 is metering and the execution condition of 'no metering is performed' is not satisfied.
Step 520, master queries whether slave Master-2 is metering.
Step 522, the slave Master-2 feeds back to the Master that it is not metering.
Since the slave Master-2 feeds back the on-going measurement in the inquiries from step 512 to step 518, in the current round (step 520 to step 522), only the slave Master-2 is inquired, and the slave Master-2 feeds back the on-going measurement, it can be determined that all the slave masters are not on-going measurement, so that the Master can randomly select a slave Master as the on-going slave Master again. For example, the Master Master selects the slave Master-1.
Step 524, the Master initiates metering to the slave Master-1.
Of course, the interval duration of polling may be set according to practical situations, such as querying one round every 10 minutes, querying one round every 5 minutes, and the like, which is not limited in the present application.
Assuming that none of the slave metadata nodes are metered, the master metadata node randomly selects a slave metadata node that immediately begins metering in response to the selection of the master metadata node. There may thus be unnecessary overhead. For example: the distributed file system may be configured to perform metering every 2 hours, and assuming that metering has just ended, then all slave nodes are not metered at this time, and metering is triggered to be performed again immediately, but the re-execution of the metering operation is of no practical value (because only 2-hour metering is required), and system resources are wasted. Therefore, in order to avoid the above situation, the distributed file system may preset a metering interval duration, and if the metering interval duration is 2 hours, the primary metadata node may determine an interval duration between the last metering completion time and the current time; and under the condition that the judgment result shows that each slave metadata node is not metered and the determined interval duration is not less than the preset metering interval duration, randomly selecting one slave metadata node as any slave metadata node. For example, the last metering completion time is 2 pm and the preset metering interval duration is 2 hours, so that the primary metadata node may not need to perform the metering operation again from 2 pm to 4 pm, thereby avoiding the waste of resources caused by performing unnecessary metering operations for multiple times.
After fork goes out of the sub-process, the slave metadata node may perform a metering operation, but the performance of the metering operation occupies system resources of the slave metadata node, and may interfere with the normal operation of the slave metadata node, for example, cause a reduction in efficiency of log synchronization of the slave metadata node with respect to the master metadata node. Therefore, in order to reduce the interference on the slave metadata nodes to the maximum extent, the distributed file system is preset with a synchronization gap range, and the synchronization gap between each slave metadata node and the master metadata node for the metadata can be determined; and randomly selecting one slave metadata node from a plurality of slave metadata nodes of which the synchronization difference of the metadata is not larger than the preset synchronization difference range as any slave metadata node. For example, each slave metadata node of the distributed file system needs to be consistent with metadata maintained by the master metadata node, and may be implemented by a log synchronization method. Assuming that the ID of an operation log (Oplog) maintained on the Master is 100, the ID of an operation log maintained on the slave Master-1 is 98, the ID of an operation log maintained on the slave Master-2 is 93, a preset synchronization gap range is 5, a log synchronization gap between the slave Master-1 and the Master is 2, and a log synchronization gap between the slave Master-2 and the Master is 7, at this time, only the synchronization gap of metadata of the slave Master-1 is not greater than the preset synchronization gap range, and the slave Master-1 can be selected to perform metering operation. Therefore, the slave Master-2 is not interfered by measurement, and can be synchronized with the Master Master log as soon as possible, so that the interference on the slave metadata nodes is reduced to the maximum extent.
Fig. 6 is a flowchart of a round scheduling provided in an exemplary embodiment of the present application, which may specifically include the following steps:
As previously described, the master metadata repository node may query the various slave metadata nodes to determine whether each slave metadata node is metering. If the determination result indicates that the slave metadata nodes are not metered, executing step 606; otherwise, step 610 is executed to end the scheduling round.
And step 606, determining whether a new metering is needed according to the interval duration between the last metering completion time and the current time.
For example, if the interval between the last metering completion time and the current time is 1 hour, and the distributed file system outputs a metering result in 2 hours in advance, a new metering operation is not needed, and step 610 may be executed to end the scheduling operation, thereby avoiding unnecessary metering operations and avoiding waste of system resources; if the distributed file system outputs a measurement result within 30 minutes, a new measurement cycle is required, and step 608 may be executed.
The synchronization difference of each slave metadata node and the master metadata node is relative to the metadata, so that the slave metadata nodes with backward synchronization states can be prevented from being interfered as much as possible, the Oplog latest slave metadata node can be selected to perform metering operation, the backward slave metadata nodes can be ensured to catch up with the log progress of the master metadata node as soon as possible, and the interference on the slave metadata nodes is reduced to the maximum extent.
And step 610, finishing scheduling in one round.
Based on the above steps 602 to 610, it may be implemented to select a suitable slave metadata node at a suitable time, so that a subprocess is derived from the slave metadata node, and then a metering operation is performed to generate a metering result.
In an embodiment, the corresponding sub-process may be derived temporarily on the basis of the main process of the metadata node in response to a received metering request for the access behavior of the user. For example, taking a slave metadata node as an example, it may receive a metering request sent by a master metadata node, thereby temporarily deriving a corresponding sub-process on the basis of the master process of the slave metadata node. The metering request may be sent by the master metadata node when the slave metadata node is selected.
In the case of multi-thread concurrency, a multi-thread synchronization lock may be used, thereby avoiding the destruction of a resource when multiple threads access a resource. In the case of using the multi-threaded synchronization lock, the derived sub-process may fail to be metered. As shown in fig. 7, fig. 7 is a schematic diagram of a metering failure according to an exemplary embodiment of the present application. At time 1, thread 1 has used a synchronized lock on a resource, so that only thread 1 can access the corresponding resource. At time 2, the thread N performs a derivation (fork) operation, thereby deriving a sub-process in which the main thread runs and a main process in which the threads 1 to N run. While thread 1 of the master process still holds the synchronization lock. At time 3, because the sub-process and the main process share the memory space, the resource in the sub-process is still locked (the synchronization lock is inherited by the sub-process), and the main thread of the sub-process cannot access the resource. At time 4, thread 1 chooses to release the lock after the resource is used, and because the "copy-on-write" mode is adopted, for the sub-process, the sub-process only runs the thread (main thread) where fork is located, and other threads do not run (thread 1 is not available), so the resource is always in a locked state. At time 5, thread N attempts to access the resource, and therefore performs the locking operation, but for the child process, the lock is not released, and the locking operation cannot be performed, so that the child process cannot access the resource protected by the lock. This can lead to metrology failure. In view of this, in order to ensure the normal performance of the metering, when it is determined that each thread running in the main process of the metadata node has released the multi-thread synchronization lock, the corresponding sub-process may be derived on the basis of the main process of the metadata node, so that the occurrence of the sub-process resource access failure may be avoided.
And 204, acquiring the metadata maintained by the main process through the sub-process, and traversing the acquired metadata to generate a measurement result of the access behavior of the user for the data node.
As described above, the sub-process and the main process can share a memory space, so that the sub-process can directly obtain metadata from the shared memory space, thereby omitting an additional step of separately pulling an operation log, and effectively accelerating the metering efficiency.
The traversal operation is executed through the metadata acquired by the sub-process, so that a corresponding metering result can be generated, and particularly, the calendar tree can be traversed. As mentioned above, after the fork action is executed, the fork action is generally executed as a corresponding main thread by default, and the performance of the single thread has a certain limitation, so in an embodiment, a corresponding thread pool may be created based on the main thread executed in the sub-process; and acquiring the metadata maintained by the main process through a plurality of threads in a thread pool, and traversing the acquired metadata concurrently to generate a metering result of the access behavior of the user for the data node. Of course, when the memory space is shared between the sub-process and the main process, the metadata may be directly obtained from the shared memory space by a plurality of threads in the thread pool. Therefore, the efficiency of metering can be greatly improved by traversing the directory tree concurrently through the thread pool, and the output speed of the metering result is increased.
It should be noted that the measurement result of the access behavior described in the present application may specifically refer to a measurement result of a data storage condition of a data node by a user, such as: the storage media of the user, the specific storage data size of the user, the number of files stored by the user, the number of directories and the like, so that the user can be provided with a corresponding bill based on the data storage condition; of course, the access behavior described in this description is not limited to the data storage situation of the user, and may also refer to an access duration of the user for the data node, an access number of the user, or an access data amount of the user, for example: the user access time is 30 minutes, the user accesses 10 times in 24 hours, the user reads 100M of data volume and the like, so that a corresponding bill can be issued to the user based on the access times, the access time or the access data volume of the user, and therefore, the access behavior is not limited to the data storage condition of the user or the access time, the access time and the like of the user.
In one embodiment, the directory requiring metering can be flexibly configured, for example, if the full cluster is metered, the metering result file is also counted, the part is not user data and needs to be shielded, thereby improving the accuracy of the metering result. Specifically, the metering range configuration information can be read, and comprises a metering range to be traversed and/or a shielded metering range; and traversing the acquired metadata according to the metering range indicated by the metering configuration information to generate a metering result of the access behavior of the user for the data node. For example, the metering range configuration information may be as follows:
{ 'MeasureDir'/metering List
'/':[0],
'/dfs/':[2]},
'ignoredur'/masked directory
['/dir_a/',
'/dir_b/']}
Wherein, measureDir is a metering directory, which may be provided with a plurality of directories, and a specific directory of the second level may be configured behind each directory, and 0 represents the current level, that is, the current directory metering; ignoreDir is a directory of masks, in this embodiment specifically masks off/dir _ a/and/dir _ b/. Therefore, the measuring result accuracy can be effectively improved based on the flexible and user-defined configuration measuring catalog, the measuring requirements under different scenes are met, and the measuring result is more practical. Of course, the masked directory is not limited to the metering result file as described above, and may be flexibly set according to actual needs, which is not limited in this application.
In an embodiment, after the metering result is generated, it may be stored in a local storage space from the metadata node, such as a local disk, but there is a possibility of losing data only in the local storage space, so to ensure the reliability of the metering result, the metering result may be written into the cluster. That is, in the case of generating the metering result, the metering result may be stored in a local storage space corresponding to the metadata node; and creating metadata corresponding to the metering result on the metadata node by running a metering control script, and storing the metering result as corresponding data to a data node. Fig. 8 is a schematic diagram illustrating a metering result writing cluster according to an exemplary embodiment of the present disclosure. After the Master process outputs the corresponding metering result, the metering result can be stored to the local, then the slave metadata node can execute the metering control script, the locally stored metering result is checked periodically, whether the metering result is updated or not is judged, if the metering result is updated, a binary client of the distributed file system can be called, the binary client is different from the client shown in fig. 1, and therefore the metering result is written into a cluster, and the reliability of storing the metering result is improved in a cluster multi-copy storage mode.
Based on the access metering method of the distributed file system, a corresponding metering result may be obtained, and for example, the metering result provided by the present specification may include the following fields:
a cluster _ name field, which may represent a name of a cluster, for example, a name of a cluster corresponding to this metering may be "XXX";
the dir _ path field may indicate a directory name, such as the directory name of this measurement is "/abc/aa/";
the sub _ dir _ num field can be the number of subdirectories under the directory;
the sub _ file _ num field may be the number of subfiles under the directory;
the sub _ hardlink _ num field may be the number of hardlinks under the directory;
file _ type field: the file type of the current measurement can be represented, for example, the file type of the current measurement is a plurality of copies;
the max _ copy _ num field and the min _ copy _ num field may respectively indicate a maximum number of file copies and a minimum number of file copies;
the oplog _ ID field may indicate the ID of the corresponding operation log;
storage _ type field: may represent a storage medium, for example, the storage medium corresponding to this metering is the HDD as described above;
the copy _ num field may correspond to the number of copies of the medium;
the item _ value field may be a corresponding statistical result;
the file _ num field may be expressed as the number of files conforming to item _ key;
the hardlink _ num field may represent the number of hardchains for item _ key;
the logic _ size field may be expressed as a file cumulative logical length;
the group _ data _ chunks field represents the amount of data;
the group _ parity _ chunks field may represent the amount of redundancy;
the measure _ timestamp field may represent a timestamp of the start of metering;
the output _ timestamp field can represent the output timestamp of the metering result, and the difference value between the starting timestamp and the output timestamp can represent the time duration consumed by the metering.
Of course, the metering result corresponds to the actual metering requirement, and is not limited to the above field, for example, a user ID may be added, and the user ID may be adaptively adjusted according to the actual requirement, which is not limited in the present application.
Based on the above embodiments, it can be seen that, in the present application, the corresponding sub-process is derived on the basis of the main process of the original metadata node, and the sub-process implements the corresponding metering operation, so that additional deployment of metering nodes is avoided, the device resources are saved, and the metering cost is reduced; in addition, because the sub-process can directly acquire the metadata maintained by the main process, the log does not need to be additionally and independently pulled, the metering result can be obtained in real time, and the timeliness of the metering result is ensured; the time for exporting the metering result is shortened, and the metering efficiency is improved.
Corresponding to the embodiments of the method, the application also provides embodiments of the device, the electronic equipment and the storage medium.
FIG. 9 is a schematic block diagram of an electronic device in accordance with an exemplary embodiment. Referring to fig. 9, at the hardware level, the apparatus includes a processor 901, a network interface 902, a memory 903, a nonvolatile memory 904, and an internal bus 905, but may also include hardware required for other functions. One or more embodiments of the present application may be implemented based on software, for example, the processor 901 reads a corresponding computer program from the nonvolatile memory 904 into the memory 903 and then runs the computer program. Of course, besides software implementation, other implementations are not excluded from one or more embodiments of the present application, such as logic devices or a combination of software and hardware, and so on, that is, the execution subject of the following processing flow is not limited to each logic unit, and may also be hardware or logic devices.
Fig. 10 is a block diagram of an access metering device of a distributed file system according to an exemplary embodiment. Referring to fig. 10, the distributed file system includes a metadata node and a data node, the metadata node maintains metadata corresponding to data stored in the data node, and the apparatus may include:
a derivation unit 1002, configured to derive a corresponding sub-process on the basis of the main process of the metadata node;
and a traversing unit 1004, configured to acquire, by the sub-process, the metadata maintained by the main process, and traverse the acquired metadata to generate a measurement result of an access behavior of a user with respect to the data node.
Optionally, the main process and the sub-process share a memory space, and the traversal unit 1004 is specifically configured to: and acquiring the metadata from the shared memory space through the subprocess.
Optionally, the metadata nodes include a master metadata node and at least one slave metadata node. The derivation unit 1002 is specifically configured to: and deriving corresponding sub-processes on the basis of the main process of the main metadata node or any slave metadata node.
Optionally, the any slave metadata node is selected by the master metadata node by the following method:
the master metadata node polls each slave metadata node to judge whether each slave metadata node is metering;
and under the condition that the judgment result shows that all the slave metadata nodes are not metered, randomly selecting one slave metadata node as any slave metadata node.
Optionally, the distributed file system presets a metering interval duration, so that the primary metadata node determines an interval duration between the last metering completion time and the current time; and under the condition that the judgment result shows that each slave metadata node is not metered and the determined interval duration is not less than the preset metering interval duration, randomly selecting one slave metadata node as any one slave metadata node.
Optionally, a synchronization gap range is preset in the distributed file system, so that the master metadata node determines synchronization gaps for metadata between each slave metadata node and the master metadata node; and randomly selecting one slave metadata node from a plurality of slave metadata nodes of which the synchronization difference of the metadata is not larger than the preset synchronization difference range as any slave metadata node.
Optionally, the deriving unit 1002 is specifically configured to: and in response to the received metering request aiming at the access behavior of the user, deriving a corresponding sub-process on the basis of the main process of the metadata node.
Optionally, the deriving unit 1002 is specifically configured to: and under the condition that all threads running in the main process of the metadata node release the multi-thread synchronous lock, deriving corresponding sub-processes on the basis of the main process of the metadata node.
Optionally, the traversal unit 1004 is specifically configured to: creating a corresponding thread pool based on a main thread running in the sub-process; and acquiring the metadata maintained by the main process through a plurality of threads in a thread pool, and traversing the acquired metadata concurrently to generate a metering result of the access behavior of the user for the data node.
Optionally, the traversal unit 1004 is specifically configured to: reading measurement range configuration information, wherein the measurement range configuration information comprises a measurement range to be traversed and/or a shielded measurement range; and traversing the acquired metadata according to the metering range indicated by the metering configuration information to generate a metering result of the access behavior of the user for the data node.
Optionally, the apparatus further comprises:
a script unit 1006, configured to store the metering result in a local storage space corresponding to the metadata node when the metering result is generated; and creating metadata corresponding to the metering result on the metadata node by running a metering control script, and storing the metering result as corresponding data to a data node.
For the device embodiments, since they substantially correspond to the method embodiments, reference may be made to the partial description of the method embodiments for relevant points. The above-described embodiments of the apparatus are merely illustrative, and the modules described as separate components may or may not be physically separate, and the components displayed as modules may or may not be physical modules, may be located in one place, or may be distributed on a plurality of network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the application. One of ordinary skill in the art can understand and implement it without inventive effort.
The present application further provides a distributed file system, the distributed file system including: the access metering method comprises a data node and a metadata node, wherein the metadata node maintains metadata corresponding to data stored in the data node and is used for realizing the steps of the access metering method in any embodiment. For the description of the access metering method, reference may be made to the description of the foregoing embodiments, and details are not repeated here.
In a typical configuration, a computer device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
Computer-readable media, including both permanent and non-permanent, removable and non-removable media, may implement the information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising a," "8230," "8230," or "comprising" does not exclude the presence of other like elements in a process, method, article, or apparatus comprising the element.
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It should be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present application. The word "if" as used herein may be interpreted as "at" \8230; "or" when 8230; \8230; "or" in response to a determination ", depending on the context.
The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the scope of protection of the present application.
Claims (14)
1. An access metering method for a distributed file system, the distributed file system comprising metadata nodes and data nodes, the metadata nodes maintaining metadata corresponding to data stored by the data nodes, the method comprising:
deriving corresponding sub-processes on the basis of the main process of the metadata node;
and acquiring the metadata maintained by the main process through the sub-process, and traversing the acquired metadata to generate a measurement result of the access behavior of the user for the data node.
2. The method of claim 1, wherein the main process and the sub-process share a memory space therebetween, and the obtaining metadata maintained by the main process via the sub-process comprises:
and acquiring the metadata from the shared memory space through the subprocess.
3. The method of claim 1, the metadata nodes comprising a master metadata node and at least one slave metadata node, the deriving the corresponding sub-process on the basis of a master process of the metadata node comprising:
and deriving corresponding sub-processes on the basis of the main process of the main metadata node or any slave metadata node.
4. The method of claim 3, wherein any of the slave metadata nodes is selected by the master metadata node by:
the master metadata node polls each slave metadata node to judge whether each slave metadata node is metering;
and under the condition that the judgment result shows that all the slave metadata nodes are not metered, randomly selecting one slave metadata node as any slave metadata node.
5. The method according to claim 4, wherein a preset metering interval duration is preset in the distributed file system, and in a case that the determination result indicates that each slave metadata node is not metered, randomly selecting one slave metadata node as any one of the slave metadata nodes comprises:
determining the interval duration of the last metering completion time and the current time;
and under the condition that the judgment result shows that each slave metadata node is not metered and the determined interval duration is not less than the preset metering interval duration, randomly selecting one slave metadata node as any one slave metadata node.
6. The method of claim 4, wherein the distributed file system is preset with a synchronization gap range, and the randomly selecting one of the slave metadata nodes as the any one of the slave metadata nodes comprises:
determining synchronization gaps for metadata between each slave metadata node and the master metadata node;
and randomly selecting one slave metadata node from a plurality of slave metadata nodes of which the synchronization difference of the metadata is not greater than a preset synchronization difference range as any slave metadata node.
7. The method of claim 1, said deriving corresponding sub-processes on the basis of a main process of the metadata node, comprising:
in response to a received metering request for the user's access behavior, a corresponding sub-process is derived on the basis of the main process of the metadata node.
8. The method of claim 1, said deriving a corresponding sub-process on the basis of a main process of said metadata node, comprising:
and under the condition that all threads running in the main process of the metadata node release the multi-thread synchronous lock, deriving corresponding sub-processes on the basis of the main process of the metadata node.
9. The method of claim 1, obtaining, by the sub-process, metadata maintained by the main process, and traversing the obtained metadata to generate a measure of user access behavior to the data nodes, comprising:
creating a corresponding thread pool based on a main thread running in the sub-process;
and acquiring the metadata maintained by the main process through a plurality of threads in a thread pool, and traversing the acquired metadata concurrently to generate a metering result of the access behavior of the user for the data node.
10. The method of claim 1, the traversing the obtained metadata to generate a measure of user access behavior to the data nodes, comprising:
reading measurement range configuration information, wherein the measurement range configuration information comprises a measurement range to be traversed and/or a shielded measurement range;
and traversing the acquired metadata according to the metering range indicated by the metering range configuration information to generate a metering result of the access behavior of the user for the data node.
11. The method of claim 1, further comprising:
under the condition of generating the metering result, storing the metering result to a local storage space corresponding to the metadata node;
and creating metadata corresponding to the metering result on the metadata node by running a metering control script, and storing the metering result as corresponding data to a data node.
12. A distributed file system, the distributed file system comprising:
a data node;
a metadata node maintaining metadata corresponding to data stored by the data node for implementing the steps of the method according to any one of claims 1 to 11.
13. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 11.
14. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the method according to any one of claims 1 to 11 when executing the program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310247122.XA CN115964353B (en) | 2023-03-10 | 2023-03-10 | Distributed file system and access metering method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310247122.XA CN115964353B (en) | 2023-03-10 | 2023-03-10 | Distributed file system and access metering method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115964353A true CN115964353A (en) | 2023-04-14 |
| CN115964353B CN115964353B (en) | 2023-08-22 |
Family
ID=87354987
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310247122.XA Active CN115964353B (en) | 2023-03-10 | 2023-03-10 | Distributed file system and access metering method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115964353B (en) |
Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104320401A (en) * | 2014-10-31 | 2015-01-28 | 北京思特奇信息技术股份有限公司 | Big data storage and access system and method based on distributed file system |
| US20150127619A1 (en) * | 2013-11-04 | 2015-05-07 | Quantum Corporation | File System Metadata Capture and Restore |
| WO2017028719A1 (en) * | 2015-08-19 | 2017-02-23 | 阿里巴巴集团控股有限公司 | Metadata output method, client side, and metadata server |
| US20170212690A1 (en) * | 2016-01-22 | 2017-07-27 | Netapp, Inc. | Recovery from low space condition of an extent store |
| CN107562757A (en) * | 2016-07-01 | 2018-01-09 | 阿里巴巴集团控股有限公司 | Inquiry, access method based on distributed file system, apparatus and system |
| CN108289080A (en) * | 2017-01-09 | 2018-07-17 | 阿里巴巴集团控股有限公司 | A kind of methods, devices and systems accessing file system |
| CN109376015A (en) * | 2018-10-23 | 2019-02-22 | 苏州思必驰信息科技有限公司 | Solution and system are blocked in log for task scheduling system |
| US10289555B1 (en) * | 2017-04-14 | 2019-05-14 | EMC IP Holding Company LLC | Memory read-ahead using learned memory access patterns |
| US20200081754A1 (en) * | 2018-09-10 | 2020-03-12 | Avigilon Corporation | Secured multi-process architecture |
| CN111367659A (en) * | 2020-02-24 | 2020-07-03 | 苏州浪潮智能科技有限公司 | Resource management method, equipment and medium for nodes in Kubernetes |
| CN111414256A (en) * | 2020-03-27 | 2020-07-14 | 中国人民解放军国防科技大学 | Application program process derivation method, system and medium based on kylin mobile operating system |
| CN111444157A (en) * | 2019-01-16 | 2020-07-24 | 阿里巴巴集团控股有限公司 | Distributed file system and data access method |
| CN113051221A (en) * | 2021-03-31 | 2021-06-29 | 网易（杭州）网络有限公司 | Data storage method, device, medium, equipment and distributed file system |
| CN113342746A (en) * | 2021-08-06 | 2021-09-03 | 联想凌拓科技有限公司 | File management system, file management method, electronic device, and storage medium |
| CN113835870A (en) * | 2020-06-23 | 2021-12-24 | 华为技术有限公司 | Data processing method and system |
| CN114710515A (en) * | 2022-06-06 | 2022-07-05 | 浪潮电子信息产业股份有限公司 | Communication method and related assembly |
-
2023
- 2023-03-10 CN CN202310247122.XA patent/CN115964353B/en active Active
Patent Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150127619A1 (en) * | 2013-11-04 | 2015-05-07 | Quantum Corporation | File System Metadata Capture and Restore |
| CN104320401A (en) * | 2014-10-31 | 2015-01-28 | 北京思特奇信息技术股份有限公司 | Big data storage and access system and method based on distributed file system |
| WO2017028719A1 (en) * | 2015-08-19 | 2017-02-23 | 阿里巴巴集团控股有限公司 | Metadata output method, client side, and metadata server |
| CN106469087A (en) * | 2015-08-19 | 2017-03-01 | 阿里巴巴集团控股有限公司 | Metadata output intent, client and meta data server |
| US20170212690A1 (en) * | 2016-01-22 | 2017-07-27 | Netapp, Inc. | Recovery from low space condition of an extent store |
| CN107562757A (en) * | 2016-07-01 | 2018-01-09 | 阿里巴巴集团控股有限公司 | Inquiry, access method based on distributed file system, apparatus and system |
| CN108289080A (en) * | 2017-01-09 | 2018-07-17 | 阿里巴巴集团控股有限公司 | A kind of methods, devices and systems accessing file system |
| US10289555B1 (en) * | 2017-04-14 | 2019-05-14 | EMC IP Holding Company LLC | Memory read-ahead using learned memory access patterns |
| US20200081754A1 (en) * | 2018-09-10 | 2020-03-12 | Avigilon Corporation | Secured multi-process architecture |
| CN109376015A (en) * | 2018-10-23 | 2019-02-22 | 苏州思必驰信息科技有限公司 | Solution and system are blocked in log for task scheduling system |
| CN111444157A (en) * | 2019-01-16 | 2020-07-24 | 阿里巴巴集团控股有限公司 | Distributed file system and data access method |
| CN111367659A (en) * | 2020-02-24 | 2020-07-03 | 苏州浪潮智能科技有限公司 | Resource management method, equipment and medium for nodes in Kubernetes |
| CN111414256A (en) * | 2020-03-27 | 2020-07-14 | 中国人民解放军国防科技大学 | Application program process derivation method, system and medium based on kylin mobile operating system |
| CN113835870A (en) * | 2020-06-23 | 2021-12-24 | 华为技术有限公司 | Data processing method and system |
| CN113051221A (en) * | 2021-03-31 | 2021-06-29 | 网易（杭州）网络有限公司 | Data storage method, device, medium, equipment and distributed file system |
| CN113342746A (en) * | 2021-08-06 | 2021-09-03 | 联想凌拓科技有限公司 | File management system, file management method, electronic device, and storage medium |
| CN114710515A (en) * | 2022-06-06 | 2022-07-05 | 浪潮电子信息产业股份有限公司 | Communication method and related assembly |
Non-Patent Citations (4)
| Title |
|---|
| YANQI WANG; YUSHENG JIA; XIAODAN XIE；: "Research of Target Characteristics Storage Based on RDBMS and Hadoop", 2016 INTERNATIONAL CONFERENCE ON IDENTIFICATION, INFORMATION AND KNOWLEDGE IN THE INTERNET OF THINGS (IIKI), pages 417 - 420 * |
| 张和泉: "一种面向深度学习平台的存储中间层的研究与实现", 中国优秀硕士学位论文全文数据库信息科技辑, pages 137 - 26 * |
| 潘金山;吕红霞;倪少权;廖常宇;: "基于多进程的铁路列车运行图群体协同编制系统", 交通运输工程与信息学报, no. 04, pages 48 - 54 * |
| 许春聪;黄小猛;徐鹏志;吴诺;刘松彬;杨广文;: "CarrierFS:基于虚拟内存的分布式文件系统", 华中科技大学学报(自然科学版), no. 1, pages 37 - 42 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115964353B (en) | 2023-08-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Didona et al. | Transactional auto scaler: Elastic scaling of replicated in-memory transactional data grids | |
| US10642694B2 (en) | Monitoring containers in a distributed computing system | |
| EP3069274B1 (en) | Managed service for acquisition, storage and consumption of large-scale data streams | |
| US10025802B2 (en) | Automated configuration of log-coordinated storage groups | |
| CN110377395B (en) | Pod migration method in Kubernetes cluster | |
| CN112654978B (en) | Method, equipment and system for checking data consistency in real time in distributed heterogeneous storage system | |
| US9858322B2 (en) | Data stream ingestion and persistence techniques | |
| CN105468473B (en) | Data migration method and data migration device | |
| CN111183625A (en) | System and method for deleting nodes in a blockchain network | |
| AU2018202230A1 (en) | Client-configurable security options for data streams | |
| US10630566B1 (en) | Tightly-coupled external cluster monitoring | |
| US10067836B1 (en) | Configuration based intelligent protection modeling | |
| CN105830033A (en) | System and method for supporting persistent store versioning and integrity in a distributed data grid | |
| US11940877B2 (en) | Restoring a directory to a state prior to a past synchronization event | |
| Zhao et al. | Sdpaxos: Building efficient semi-decentralized geo-replicated state machines | |
| Benz et al. | Building global and scalable systems with atomic multicast | |
| CN109582459A (en) | The method and device that the trustship process of application is migrated | |
| CN112579550B (en) | Metadata information synchronization method and system of distributed file system | |
| CN115964353B (en) | Distributed file system and access metering method thereof | |
| CN115756955A (en) | Data backup and data recovery method and device and computer equipment | |
| CN108376104B (en) | Node scheduling method and device and computer readable storage medium | |
| CN112559568A (en) | Virtual article determination method and device and computer readable storage medium | |
| US11418588B2 (en) | Intelligent peer-to-peer container filesystem | |
| CN109033398B (en) | Method and device for distributing nodes | |
| CN115408148A (en) | Distributed data parallel processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |