WO2017028719A1 - 元数据输出方法、客户端和元数据服务器 - Google Patents

元数据输出方法、客户端和元数据服务器 Download PDF

Info

Publication number
WO2017028719A1
WO2017028719A1 PCT/CN2016/094320 CN2016094320W WO2017028719A1 WO 2017028719 A1 WO2017028719 A1 WO 2017028719A1 CN 2016094320 W CN2016094320 W CN 2016094320W WO 2017028719 A1 WO2017028719 A1 WO 2017028719A1
Authority
WO
WIPO (PCT)
Prior art keywords
metadata
output
configuration parameter
server
module
Prior art date
Application number
PCT/CN2016/094320
Other languages
English (en)
French (fr)
Inventor
姚文辉
刘俊峰
黄硕
张海勇
朱家稷
Original Assignee
阿里巴巴集团控股有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 阿里巴巴集团控股有限公司 filed Critical 阿里巴巴集团控股有限公司
Publication of WO2017028719A1 publication Critical patent/WO2017028719A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor

Definitions

  • the invention belongs to the field of cloud computing, and in particular to a metadata output method, a client and a metadata server.
  • the files or directories in the system are often counted to obtain the characteristics of the current user data and guide the subsequent development of the file system.
  • the information can help the users of the file system to analyze the data usage. Users use billing and other purposes.
  • the large-scale distributed storage system architecture adopted on the production systems of Internet enterprises adopts an architecture similar to GFS (Google File System).
  • GFS Google File System
  • all data modifications will form an operation log.
  • the log can be called again to the memory to achieve the memory state before the process restarts.
  • the metadata server MetalServer
  • the metadata service process periodically outputs the contents of the memory to the local disk to form a snapshot, and then deletes the operation log before the snapshot.
  • the snapshot file is first read and restored from the disk to the memory, and then the operation log generated after the snapshot is applied to the memory to avoid the long-term recovery problem caused by applying all operation logs.
  • the metadata service process is triggered to output the memory information as a snapshot on the disk (snapshot);
  • the existing method described above has the following problem: Currently, whether part or full amount of metadata is required, the memory state of all files needs to be restored. When the number of files is large, the memory usage of the metadata server is high, and an extra idle machine is needed to ensure that the system does not compete with the original metadata service process during the process of restoring the memory, resulting in system instability, resulting in an increase in implementation cost, especially This is especially true when multiple clusters need this feature at the same time.
  • the present application provides a metadata output method, a client, and a metadata server to solve the technical problem of recovering the memory state of all files when outputting metadata at a specific time in the prior art.
  • the present application discloses a metadata output method, comprising: receiving a call to a MetaServer process, the call including an output configuration parameter of the metadata; and a fork function Creating a child process of the metadata service process; controlling the child process to output corresponding metadata according to the output configuration parameter.
  • the receiving a call to a metadata service process includes output configuration parameters of the metadata comprising: receiving a remote procedure call from the client to the metadata service process.
  • the invoking includes output configuration parameters of the metadata, including: determining whether the child process of the metadata service process that was created last time has completed execution; when the last created When the child process of the metadata service process has completed execution, saving the output configuration parameter to the local configuration file; when the child process of the metadata service process created last time is not completed, returning the current metadata service process Busy news.
  • the controlling the sub-process to output the corresponding metadata according to the output configuration parameter includes: controlling the sub-process to write the output metadata according to the output configuration parameter into a temporary file of the local memory; when the metadata is output When finished, mark the child process has completed execution.
  • the output configuration parameter includes a plurality of output objects.
  • the controlling the sub-process to output the corresponding metadata according to the output configuration parameter comprises: determining the sub-ingress Whether the process has a deadlock state; when the child process has a deadlock state, the child process is killed.
  • the method further includes: determining whether the number of the un-uploaded temporary files reaches a preset threshold; When the number of the temporary files that are not uploaded reaches the preset threshold, the waiting state is entered to wait for the temporary file that has not been uploaded to be uploaded.
  • the invoking further includes an uploading configuration parameter of the metadata; the controlling the sub-process outputting the corresponding metadata according to the output configuration parameter, the method further comprising: outputting the output according to the uploading configuration parameter Metadata is uploaded to the distributed storage system.
  • the uploading configuration parameter includes an uploading rate and a target directory, where the uploading rate is used to control a data transmission rate of the uploading process; and the uploading the outputted metadata to the distributed storage system according to the uploading configuration parameter includes Uploading the output metadata to a temporary directory of the distributed storage system; when the output metadata is completely uploaded, renaming the temporary directory to the target directory, and marking the current directory The task has been completed.
  • the present application also discloses a metadata output method, including: configuring output configuration parameters of metadata; initiating a call to a metadata service process to a metadata server to create the A child process of the metadata service process, by which the child process outputs corresponding metadata according to the output configuration function.
  • the method further includes: configuring an upload configuration parameter of the metadata, while configuring the output configuration parameter of the metadata.
  • the method further includes: transmitting a progress query request to the metadata server; and receiving task status information returned by the metadata server.
  • the present application also discloses a metadata server, the metadata server comprising: a receiving module, configured to receive a call to a MetaServer process (MetaServer) process, where the call includes output of metadata a configuration module, configured to create a child process of the metadata service process by a fork function, and a processing module, configured to control the child process to output corresponding metadata according to the output configuration parameter.
  • a receiving module configured to receive a call to a MetaServer process (MetaServer) process, where the call includes output of metadata
  • a configuration module configured to create a child process of the metadata service process by a fork function
  • a processing module configured to control the child process to output corresponding metadata according to the output configuration parameter.
  • the receiving module includes: a receiving submodule, configured to receive a remote procedure call from the client to the metadata service process.
  • the receiving module includes: a first determining sub-module, configured to determine whether a child process of the metadata service process that was created last time has completed execution; and a saving sub-module, where the metadata service process is created last time Save the output configuration parameters to the local configuration file when the child process has finished executing; return the submodule for the previous one When the child process of the metadata service process that is created is not completed, the message that the metadata service process is currently busy is returned.
  • the processing module includes: a writing sub-module, configured to control the sub-process to write the output metadata according to the output configuration parameter into a temporary file of the local memory; and mark the sub-module, when the metadata is outputted When the child process is marked, the execution has completed.
  • the output configuration parameter includes a plurality of output objects.
  • the processing module includes: a second determining submodule, configured to determine whether the child process has a deadlock state; and a first processing submodule, configured to kill the child process when the child process has a deadlock state .
  • the processing module further includes: a third determining sub-module, configured to determine whether the number of the temporary files that are not uploaded reaches a preset threshold; and the second processing sub-module is configured to: when the temporary files are not uploaded When the number reaches the preset threshold, the waiting state is entered to wait for the temporary file that has not been uploaded to be uploaded.
  • the call further includes an uploading configuration parameter of the metadata;
  • the metadata server further includes: an uploading module, configured to upload the output metadata to the distributed storage system according to the uploading configuration parameter.
  • the uploading configuration parameter includes an uploading rate and a target directory, where the uploading rate is used to control a data transmission rate of the uploading process;
  • the uploading module includes: an uploading submodule, configured to upload the outputted metadata to the a temporary directory of the distributed storage system; a sub-module for renaming the temporary directory to the target directory when the output metadata is completely uploaded, and marking that the task is completed.
  • the present application further discloses a metadata output client, including: a first configuration module, configured to configure output configuration parameters of metadata; and a calling module, configured to initiate a metadata service to the metadata server The invocation of the process to create a child process of the metadata service process by a fork function, and the child process outputs corresponding metadata according to the output configuration function.
  • the client further includes: a second configuration module, configured to configure an upload configuration parameter of the metadata.
  • the client further includes: a query module, configured to send a progress query request to the metadata server; and a receiving module, configured to receive task state information returned by the metadata server.
  • the present application can obtain the following technical effects: no need to restore the memory state of all files, and can obtain the memory state of the corresponding output target at the time of the fork operation, thereby ensuring the aging of the metadata output. Sex; no additional computers are required, resulting in lower implementation costs. In addition, the memory usage of the current metadata server is rarely increased, and only the amount of memory modified by the parent process's write operation is allocated to the child process.
  • FIG. 1 is a schematic flowchart of a metadata output method according to an embodiment of the present application.
  • FIG. 2 is a schematic flowchart of receiving a call to a metadata service process according to an embodiment of the present application
  • FIG. 3 is a schematic flowchart of controlling metadata output by a sub-process according to an embodiment of the present application
  • FIG. 4 is a schematic flowchart diagram of a metadata output method according to an embodiment of the present application.
  • FIG. 5 is a schematic flowchart of uploading metadata according to an embodiment of the present application.
  • FIG. 6 is a schematic flowchart of a metadata output method according to an embodiment of the present application.
  • FIG. 7 is a schematic flowchart diagram of a metadata output method according to an embodiment of the present application.
  • FIG. 8 is a schematic flowchart diagram of a metadata output method according to an embodiment of the present application.
  • FIG. 9 is a block diagram showing an exemplary structure of a metadata server according to an embodiment of the present application.
  • FIG. 10 is a block diagram showing an exemplary structure of a metadata output client according to an embodiment of the present application.
  • FIG. 1 is a metadata output method provided by an embodiment of the present application, which is applicable to a server device, and the method includes the following steps.
  • step S10 a call to the MetaServer process is received, and an output configuration parameter including the metadata is called.
  • the metadata service process runs in the metadata server and outputs corresponding metadata based on the received call.
  • the call to the metadata service process includes the output configuration parameters of the metadata.
  • the output configuration parameters include an output destination, an output format, and an output content storage path.
  • the output target refers to the memory directory where the metadata to be output is currently located;
  • the output format refers to the saved form after the metadata is output, such as a file format or a memory directory format;
  • the output content storage path is the output metadata in the memory. Storage path.
  • the output destination is /var/log/abc
  • the output format is file format
  • the output content storage path is /tmp/msv/abc;
  • the output configuration parameter outputs the metadata in the /var/log/abc directory as a file. Go to the /tmp/msv/abc directory.
  • step S11 a child process of the metadata service process is created by a fork function.
  • the metadata server creates a metadata service through a fork function after receiving a call to the metadata service process
  • the child process of the process is used to output metadata corresponding to the output target at the moment of the fork operation; the parent process continues with other operations on the metadata server.
  • the child process shares the memory address space with the metadata service process (parent process), and the parent process uses copy-on-write technology to copy the corresponding change when the parent process has a write operation to the memory address space.
  • the memory address space is allocated to the child process. Ensure that the memory address space corresponding to the child process can be fixed at the time point of the fork operation after the fork operation of the fork function is completed, and the consistency of the metadata at a certain time point is achieved, and the output is configured according to the child process.
  • the parameter outputs the corresponding metadata.
  • step S12 the control sub-process outputs corresponding metadata according to the output configuration parameters.
  • the output of the metadata no longer needs to restore the memory state of all the files of the metadata server, and the child process is created by the fork function to obtain the memory state of the corresponding output target at the time of the fork operation, which can ensure the time limit of the metadata output. Sex.
  • the memory state is restored on the additionally equipped computer device in order to avoid competing with the metadata service process.
  • the metadata output method of the present application it is no longer necessary to equip the additional computer, so that the implementation cost is reduced.
  • the memory usage of the current metadata server is rarely increased, and only the amount of memory modified by the parent process's write operation is allocated to the child process.
  • step S10 receives a call to the metadata service process, which is a remote procedure call (RPC) from the client to the metadata service process. That is, a call to the metadata service process is initiated to the metadata server by a client computer other than the metadata server.
  • RPC remote procedure call
  • the output operation for all metadata in the distributed storage system is controlled by the client computer, through which the output configuration parameters are set and a remote procedure call is initiated to the metadata server process of the metadata server.
  • the output of the metadata can be remotely and centrally controlled.
  • centralized control by the client computer can effectively reduce the maintenance difficulty and cost of the system and improve the maintenance efficiency.
  • receiving the call to the metadata service process in step S10 further includes the following steps.
  • step S101 a call to the metadata service process is received.
  • step S102 it is determined whether the child process of the metadata service process created last time has completed execution.
  • step S103 is performed; when the child process of the last created metadata service process is not completed, step S104 is performed.
  • the child process created by each fork operation marks the corresponding state after the output of the metadata is completed. For example, the status of the child process that has completed execution is marked as "Done". In order to avoid taking up too much memory resources, determine whether the last created child process has completed execution.
  • step S103 the output configuration parameters are saved to the local configuration file.
  • the metadata server saves the output configuration parameters of this call in the local configuration file in preparation for the child process to complete the metadata output.
  • step S104 the message that the metadata service process is currently busy is returned, and the process returns to step S102.
  • the call to the metadata service process comes from a remote client computer, the current busy message of the metadata service process is returned to the remote client computer.
  • the memory space corresponding to the child process created by the fork function at different times has different time attributes, and the metadata state in memory will be different.
  • metadata of multiple output targets is required, in order to ensure the timeliness of the output metadata, multiple output targets that need to be output are simultaneously set in the output configuration parameters, and are completed by one call to the metadata service process.
  • the child process created by the fork operation will complete the metadata output of multiple output targets to ensure the timeliness of all output targets.
  • the output configuration parameters include multiple output targets such as /var/log/abc, /var/log/efg, /var/log/chb.
  • step S12 controls the sub-process to output corresponding metadata according to the output configuration parameter, and further includes the following steps.
  • step S121 the control sub-process writes the output metadata to the temporary file of the local memory according to the output configuration parameter.
  • the child process writes the metadata corresponding to the output target to a temporary file in the local memory.
  • a preset threshold such as 200M
  • the temporary file is named as a file name recognizable by the server upload process.
  • the identifiable file name is usually the file name corresponding to the output target. For example, if the output destination is /var/log/abc, the temporary file is named abc_tmp.
  • the output needs to be output or the metadata has been When the output needs to be output, change the file name from abc_tmp to abc. If the output metadata is divided into multiple temporary files according to the file size threshold, they are named: abc1, abc2, ...
  • the server upload process is used to upload the output metadata to a distributed storage system to implement other functions related to metadata, such as user data analysis, user behavior statistics, and the like.
  • the process of uploading metadata will be described in detail in subsequent embodiments.
  • step S122 when the metadata output is completed, the flag sub-process has completed execution.
  • the child process In the metadata output process, the child process is usually marked as "waiting". After the metadata is output, it is usually marked as "Done” to continue the next metadata output. Exporting metadata to local memory can reduce access to the metadata server's local disk and avoid affecting the performance of the metadata server.
  • step S12 controls the child process to output corresponding metadata according to the output configuration parameter, and also needs to determine whether the child process has a deadlock state.
  • the child process may wait for other processes to release, but the memory resources required by other processes may also be locked by the child process at the same time, need to wait for the child process to release, then the deadlock state occurs.
  • kill the child process After detecting that the child process enters the deadlock state, kill the child process and mark the child process as "failed".
  • it is necessary to kill the child process that has entered the deadlock state For example, use the process ID (PID) of the child process and the corresponding command to kill the child process entering the deadlock state.
  • PID process ID
  • FIG. 4 is a metadata output method provided by an embodiment of the present application, which is applicable to a server device, and the method includes the following steps.
  • step S20 a call to the metadata service process is received, and an output configuration parameter including the metadata and an upload configuration parameter are invoked.
  • Upload configuration parameters include upload rate, target directory, and compression parameters.
  • the upload rate is used to control the data transmission rate of the upload process. For example, if the upload rate is 20 Mb/S, the data transmission rate of the upload process may be less than or equal to 20 Mb/S.
  • the target directory is the directory to which the exported metadata is uploaded. The target directory can be located in a local cluster or in another remote cluster.
  • the compression parameter is used to control whether the output metadata needs to be compressed before uploading. When there is too much metadata to be uploaded, in order to improve efficiency, the compression parameter is used to compress the data before uploading.
  • step S21 a child process of the metadata service process is created by a fork function.
  • step S22 the control sub-process outputs corresponding metadata according to the output configuration parameter, and simultaneously uploads the output metadata to the distributed storage system according to the upload configuration parameter.
  • the server upload process is set in the metadata server, and the metadata for output is uploaded to the distributed storage system.
  • the server upload process is set to start every preset time to check if there is output metadata that needs to be uploaded. For example, use the Crontab command to set up a server upload process running every minute.
  • the server uploading process detects whether the local configuration file of the metadata server includes the output configuration parameter and the upload configuration parameter. When the configuration parameter is detected, the metadata upload operation is started according to the upload configuration parameter.
  • the output process and upload process of the metadata are parallel processes, and the output metadata is uploaded as soon as possible to reduce the occupation of the memory space and avoid affecting the performance of the metadata server.
  • a preset threshold for example, whether the number of un-transferred temporary files is reached. 5
  • the output metadata is continuously written to the temporary file in the local memory; when the number of the un-uploaded temporary files reaches the preset threshold, the sub-process is controlled to enter. Waiting state, waiting for the temporary file not uploaded to be uploaded. Therefore, the output speed of the metadata can be adaptively uploaded, and the memory is not excessively occupied as much as possible.
  • the metadata server starts the server upload process, it needs to open the executable file located on the local disk.
  • an exclusive lock is added to the execution file.
  • the metadata server can no longer start the second server upload process.
  • the exclusive lock release of the execution file will be released. In order to start the next server upload process again.
  • the process of preventing the upload of metadata occupies too much network bandwidth and affects the network data transmission of other processes of the metadata server.
  • the metadata server After the metadata server successfully starts the server upload process, it determines whether there is a need to perform the upload task according to the local configuration file. If not, exit the server upload process; if there is, start uploading metadata and check whether the task has completed execution (whether the metadata has been completely output), and when it is detected that the task has been executed, set this upload as the last of the task. Upload once, until the last upload of the task is detected and uploaded.
  • the process of uploading the output metadata to the distributed storage system according to the uploading configuration parameter in step S22 further includes:
  • step S221 the output metadata is uploaded to a temporary directory of the distributed storage system.
  • step S222 when the output metadata is completely uploaded, the temporary directory is renamed to the target directory, and the task is marked as completed.
  • the temporary directory is modified to the target directory in the upload configuration parameter, thereby saving the output metadata to the target directory of the distributed storage system, for example, modifying the temporary directory /tmp/upload to the target directory.
  • Directory /var/lib/abc After the metadata is uploaded, mark the task as "Upload completed.”
  • FIG. 6 is a metadata output method according to an embodiment of the present application, and the method includes the following steps.
  • step S301 a call to the metadata service process is received, and an output configuration parameter including the metadata and an upload configuration parameter are invoked.
  • step S302 it is determined whether the child process of the metadata service process created last time has completed execution.
  • step S304 is performed; when the child process of the last created metadata service process is not completed, step S303 is performed.
  • step S303 the message that the metadata service process is currently busy is returned, and the process returns to step S302.
  • step S304 the output configuration parameters and the upload configuration parameters are saved to the local configuration file.
  • step S305 a child process of the metadata service process is created by a fork function.
  • step S306 the child process is controlled to obtain an output configuration parameter from a local configuration file.
  • step S307 it is determined whether the number of temporary files that are not currently uploaded reaches a preset threshold. When not uploaded When the number of files does not reach the preset threshold, step S308 is performed; when the number of unuploaded files reaches the preset threshold, step S309 is performed.
  • step S308 the child process is controlled to enter a waiting state to wait for the un-uploaded temporary file to be uploaded.
  • step S309 the output metadata is written to a temporary file in the local memory.
  • step S310 when the size of the temporary file reaches a certain threshold or the task execution of the output metadata is completed, the temporary file is named as a file name recognizable by the server uploading process, waiting for the metadata to be output by the server uploading process. Upload to a distributed storage system.
  • Step S305 After the child process of the metadata service process is created by the fork function, when the metadata server controls the output of the metadata, the metadata server also detects whether the child process has a deadlock state in parallel, including the following related steps.
  • step S311 it is determined whether the child process enters a deadlock state.
  • step S312 is performed; when the child process does not enter the deadlock state, step S311 is continued.
  • step S312 the child process is killed and the current output task execution failure is marked.
  • the metadata output process executed by the metadata service process in the embodiment of the present application is parallel to the metadata upload process performed by the server upload process.
  • the metadata server starts the server upload process every preset time.
  • the metadata upload process includes the following steps.
  • step S313 the execution file of the server upload process located on the local disk is opened.
  • step S314 it is determined whether the execution file is successfully opened. When the execution file is unsuccessful, the process returns to step S313; when the execution file is successfully opened, step S315 is performed.
  • step S315 it is determined according to the local configuration file whether or not a task is running. When it is determined that no task is running, step S316 is performed; when it is determined that a task is running, step S317 is performed.
  • step S316 the server upload process is closed.
  • step S317 the output metadata is uploaded to a temporary directory of the distributed storage system.
  • step S318 it is determined whether the task of the child process output metadata has been completed.
  • step S319 is performed; when the task of the child process outputting the metadata is not completed, the process returns to step S317.
  • step S319 the current upload is set as the last upload of the task.
  • step S320 when the output metadata is completely uploaded, the temporary directory is renamed to the target directory, and the upload is marked as completed.
  • FIG. 7 is a metadata output method provided by an embodiment of the present application, which is applicable to a client device, and includes the following steps. Step.
  • step S40 the output configuration parameters of the metadata are configured.
  • the client computer configures the output configuration parameters of the metadata according to the user operation, including the output destination, the output format, and the output content storage path.
  • step S41 a call to the metadata service process is initiated to the metadata server to create a child process of the metadata service process through the fork function, and the child process outputs corresponding metadata according to the output configuration function.
  • the client computer initiates a call to the metadata server process to the metadata server, the call including the output configuration parameters configured in step S40.
  • the metadata server uses the fork function to create a child process of the metadata service process, and the child process outputs corresponding metadata according to the output configuration function.
  • the call can be a remote procedure call, thereby realizing remote centralized configuration and management of metadata output in a distributed storage system composed of multiple clusters, and multiple clusters can be operated in parallel, reducing system maintenance difficulty and cost, and improving efficiency. .
  • step S40 further configures an upload configuration parameter of the metadata while configuring the output configuration parameter of the metadata.
  • Upload configuration parameters include upload rate, target directory, and compression parameters.
  • FIG. 8 is a metadata output method provided by an embodiment of the present application, which is applicable to a client device, and includes the following steps.
  • step S50 the output configuration parameters and the upload configuration parameters of the metadata are configured.
  • step S51 a call to the metadata service process is initiated to the metadata server to create a child process of the metadata service process by the fork function, and the child process outputs the corresponding element according to the output configuration function.
  • the data is uploaded to the distributed storage system by the server upload process of the metadata server.
  • step S52 a progress inquiry request is sent to the metadata server.
  • a progress query request may be sent to the metadata server according to the configuration identifier to query the execution progress of the corresponding configuration task.
  • the sent progress query request may also not include the configuration identifier. In this case, the default is to query the progress of the currently executing configuration task or the last execution completion task status.
  • step S53 the task status information returned by the metadata server is received.
  • the output configuration parameters, upload configuration parameters, and task progress status of the configuration task can be queried. If it is a configuration task in progress, the task progress status includes "executing" and “waiting”; if it is a configuration task that has been completed, the task progress status includes "execution failure” and "execution failure”. When the progress status of the task is "execution failure", it is also possible to query the cause of the execution failure.
  • FIG. 9 is a metadata server provided by an embodiment of the present application, where the metadata server includes:
  • the receiving module 60 is configured to receive a call to a metadata service (MetaServer) process, and invoke an output configuration parameter including metadata;
  • a metadata service MetalServer
  • a creating module 61 configured to create a child process of the metadata service process by using a fork function
  • the processing module 62 is configured to control the sub-process to output corresponding metadata according to the output configuration parameter.
  • the receiving module 60 includes:
  • a receiving submodule for receiving a remote procedure call from a client to a metadata service process is provided.
  • the receiving module 60 includes:
  • a first determining submodule configured to determine whether a child process of the last created metadata service process has completed execution
  • the save submodule is configured to save the output configuration parameter to the local configuration file when the child process of the last created metadata service process has completed execution;
  • the processing module 62 includes:
  • the output configuration parameters include a plurality of output objects.
  • the processing module 62 includes:
  • a second determining submodule configured to determine whether the child process has a deadlock state
  • the first processing submodule is configured to kill the child process when the child process has a deadlock state.
  • the processing module 62 further includes:
  • a third determining sub-module configured to determine whether the number of non-uploaded temporary files reaches a preset threshold
  • the second processing sub-module is configured to enter a waiting state when the number of un-uploaded temporary files reaches a preset threshold, so as to wait for the un-uploaded temporary file to be uploaded.
  • the call further includes an upload configuration parameter of the metadata;
  • the metadata server further includes:
  • the uploading module is configured to upload the output metadata to the distributed storage system according to the uploading configuration parameter.
  • the uploading configuration parameter includes an uploading rate and a target directory, where the uploading rate is used to control the data transmission rate of the uploading process;
  • the uploading module includes:
  • Rename the submodule to rename the temporary directory to the target directory when the output metadata is completely uploaded, and mark that the task has been completed.
  • FIG. 10 is a metadata output client provided by an embodiment of the present application, including:
  • a first configuration module 70 configured to configure output configuration parameters of the metadata
  • the calling module 71 is configured to initiate a call to the metadata service process to the metadata server to create a child process of the metadata service process by the fork function, and the child process outputs the corresponding metadata according to the output configuration function.
  • the client further includes:
  • the second configuration module is configured to configure metadata upload configuration parameters.
  • the client further includes:
  • a query module configured to send a progress query request to the metadata server
  • the receiving module is configured to receive task status information returned by the metadata server.
  • the metadata output method, server and client provided by the embodiments of the present application can be used for a cloud computing platform of an Internet enterprise, and provide support for a distributed file system of the cloud computing platform.
  • the metadata server of each cluster can be centrally managed through the client, thereby controlling the metadata output of each cluster.
  • metadata of a file is usually used to record corresponding file attribute information, such as a file storage path, a space occupied, a number of accesses, a number of read and write operations, and the like, and outputting the metadata enables the cloud computing platform to be used by the user.
  • the behavior and the documents produced by the behavior are analyzed.
  • the metadata corresponding to the user file of the website in the distributed file system is output and uploaded to the specified directory, and every preset period ( For example, output and upload once a day or every week.
  • every preset period For example, output and upload once a day or every week.
  • the metadata output method, the server, and the client provided by the embodiments of the present application can also be applied to the ODPS (Open Data Processing Service) computing platform.
  • the ODPS provides data for TB/PB level, and the real-time requirement is not High distributed processing power, applied to data analysis, mining, business intelligence and other fields, allowing users to spend more time focusing on user data mining and analysis.
  • the metadata output method provided by the embodiment of the present application, the user can configure different output parameters according to actual user data analysis requirements. Use the output metadata to analyze changes in user files, the daily increase in user files, and the storage of user files in each cluster. It has been determined whether the corresponding cluster needs to be expanded.
  • the user's file access is analyzed according to the output metadata, but it has been determined that the user files that have become "hot spot" data need to be migrated, etc.; or the output metadata is used to analyze the distribution of users with different characteristics.
  • the metadata output method provided by the embodiment of the present application is a basic solution for providing a cloud computing platform and a big data service provided by an Internet enterprise, and brings great convenience for centralized management of multiple clusters and scenarios in which metadata needs to be frequently output. Improve the efficiency of cloud computing platforms and big data services in user data mining and analysis.
  • a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
  • processors CPUs
  • input/output interfaces network interfaces
  • memory volatile and non-volatile memory
  • the memory may include non-persistent memory, random access memory (RAM), and/or non-volatile memory in a computer readable medium, such as read only memory (ROM) or flash memory.
  • RAM random access memory
  • ROM read only memory
  • Memory is an example of a computer readable medium.
  • Computer readable media includes both permanent and non-persistent, removable and non-removable media.
  • Information storage can be implemented by any method or technology.
  • the information can be computer readable instructions, data structures, modules of programs, or other data.
  • Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory. (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disk read only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, Magnetic tape cartridges, magnetic tape storage or other magnetic storage devices or any other non-transportable media can be used to store information that can be accessed by a computing device.
  • computer readable media does not include non-transitory computer readable media, such as modulated data signals and carrier waves.
  • first device if a first device is coupled to a second device, the first device can be directly electrically coupled to the second device, or electrically coupled indirectly through other devices or coupling means. Connected to the second device.
  • the description of the present invention is intended to be illustrative of the preferred embodiments of the invention. The scope of protection of the present invention The definition is subject to the requirements.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Computer And Data Communications (AREA)

Abstract

本发明公开了一种元数据输出方法、客户端和元数据服务器，所述元数据输出方法包括：接收到对元数据服务进程的调用，所述调用包括元数据的输出配置参数；通过分叉函数创建所述元数据服务进程的子进程；控制所述子进程根据所述输出配置参数输出相应的元数据。不再需要恢复所有文件的内存状态，能够得到在分叉操作的时间点相应的输出目标的内存状态，从而保证元数据输出的时效性；不再需要配备额外的计算机，使实现成本降低。

Description

元数据输出方法、客户端和元数据服务器
本申请要求2015年08月19日递交的申请号为201510512514.X、发明名称为“元数据输出方法、客户端和元数据服务器”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本发明属于云计算领域，具体地说，涉及一种元数据输出方法、客户端和元数据服务器。
背景技术
文件系统的使用过程中，经常会对系统中的文件或者目录进行统计，以获取当前用户数据的特点，指导文件系统的后续开发；同时这些信息可以帮助文件系统的用户来分析数据使用，对不同的用户进行记账等用途。
目前互联网企业的生产系统上所采用的大规模分布式存储系统架构都采用了类似于GFS(谷歌文件系统)的架构。在这种架构下，所有的数据修改都会形成操作日志，在重启恢复的时候可以再次调用日志到内存中以达到进程重启前的内存状态。随着时间的积累，文件系统内的操作日志会越来越多，导致元数据服务(MetaServer)进程重启花费的时间也越来越长，同时需要保留所有操作日志会造成的存储空间不足的问题。为了提高元数据服务进程重启恢复速度和控制存储空间的无限使用，元数据服务进程会定期将内存中的内容输出到本地磁盘形成快照，然后将快照前的操作日志删除。在进程重启恢复内存状态时会先从磁盘将快照文件读取并恢复至内存，然后将快照后产生的操作日志应用到内存中来避免应用所有操作日志带来的长时间恢复问题。
在现有系统实现方案下，为了可以对部分或者全部文件进行分析，通常有如下方法来获取到与想要的时间点一致的元(meta)数据。
方法一：
1、在想要分析的时间点，触发元数据服务进程将内存信息输出为磁盘上的快照(snapshot)；
2、将快照拷贝到额外的一台空闲机器上，将快照加载到该机器的内存；
3、输出需要的元数据。
方法二：
1、拷贝元数据服务进程的任何一个快照和其后面产生的操作日志到额外的一台空闲机器上；
2、使用工具来加载快照并应用其后的操作日志到达某个时刻的内存状态；
3、输出需要的元数据。
现有的上述方法存在以下问题：当前无论需要的是部分还是全量的元数据，都需要将所有文件的内存状态恢复。在文件数量多造成元数据服务器内存使用率较高时，需要额外的闲置机器来保证在恢复内存的过程中不与原来的元数据服务进程竞争内存而导致系统不稳定，造成实现成本增加，尤其是在多个集群同时需要该功能的时候表现尤为严重。
发明内容
有鉴于此，本申请提供了一种元数据输出方法、客户端和元数据服务器，以解决现有技术中在输出特定时间的元数据时，需要恢复所有文件的内存状态的技术问题。
为了解决上述技术问题，本申请公开了一种元数据输出方法，包括：接收到对元数据服务(MetaServer)进程的调用，所述调用包括元数据的输出配置参数；通过分叉(fork)函数创建所述元数据服务进程的子进程；控制所述子进程根据所述输出配置参数输出相应的元数据。
所述接收到对元数据服务进程的调用，所述调用包括元数据的输出配置参数包括：接收到来自客户端的对所述元数据服务进程的远程过程调用。
所述接收到对元数据服务进程的调用，所述调用包括元数据的输出配置参数包括：判断上一次创建的所述元数据服务进程的子进程是否已完成执行；当上一次创建的所述元数据服务进程的子进程已完成执行时，保存所述输出配置参数至本地配置文件；当上一次创建的所述元数据服务进程的子进程未完成执行时，返回所述元数据服务进程当前繁忙的消息。
所述控制所述子进程根据所述输出配置参数输出相应的元数据包括：控制所述子进程根据所述输出配置参数将输出的元数据写入本地内存的临时文件；当所述元数据输出完毕时，标记所述子进程已完成执行。
所述输出配置参数包括多个输出对象。
所述控制所述子进程根据所述输出配置参数输出相应的元数据包括：判断所述子进 程是否出现死锁状态；当所述子进程出现死锁状态时，杀死所述子进程。
所述控制所述子进程根据所述输出配置参数将输出的元数据写入本地内存的临时文件之前，所述方法还包括：判断未上传的所述临时文件的个数是否达到预设阈值；当未上传的所述临时文件的个数达到预设阈值时，进入等待状态，以等待未上传的所述临时文件被上传完毕。
所述调用还包括元数据的上传配置参数；所述控制所述子进程根据所述输出配置参数输出相应的元数据的同时，所述方法还包括：根据所述上传配置参数将所述输出的元数据上传至分布式存储系统。
所述上传配置参数包括上传速率和目标目录，其中，所述上传速率用于控制上传过程的数据传输速率；所述根据所述上传配置参数将所述输出的元数据上传至分布式存储系统包括：上传所述输出的元数据至所述分布式存储系统的临时目录；当所述输出的元数据被完整的上传完毕时，将所述临时目录重命名为所述目标目录，并标记本次任务已完成。
为了解决上述技术问题，本申请还公开了一种元数据输出方法，包括：配置元数据的输出配置参数；向元数据服务器发起对元数据服务进程的调用，以通过分叉函数创建出所述元数据服务进程的子进程，由所述子进程根据所述输出配置函数输出相应的元数据。
所述配置元数据的输出配置参数的同时，所述方法还包括：配置元数据的上传配置参数。
所述方法还包括：发送进度查询请求至元数据服务器；接收所述元数据服务器返回的任务状态信息。
为了解决上述技术问题，本申请还公开了一种元数据服务器，所述元数据服务器包括：接收模块，用于接收到对元数据服务(MetaServer)进程的调用，所述调用包括元数据的输出配置参数；创建模块，用于通过分叉(fork)函数创建所述元数据服务进程的子进程；处理模块，用于控制所述子进程根据所述输出配置参数输出相应的元数据。
所述接收模块包括：接收子模块，用于接收到来自客户端的对所述元数据服务进程的远程过程调用。
所述接收模块包括：第一判断子模块，用于判断上一次创建的所述元数据服务进程的子进程是否已完成执行；保存子模块，用于当上一次创建的所述元数据服务进程的子进程已完成执行时，保存所述输出配置参数至本地配置文件；返回子模块，用于当上一 次创建的所述元数据服务进程的子进程未完成执行时，返回所述元数据服务进程当前繁忙的消息。
所述处理模块包括：写入子模块，用于控制所述子进程根据所述输出配置参数将输出的元数据写入本地内存的临时文件；标记子模块，用于当所述元数据输出完毕时，标记所述子进程已完成执行。
所述输出配置参数包括多个输出对象。
所述处理模块包括：第二判断子模块，用于判断所述子进程是否出现死锁状态；第一处理子模块，用于当所述子进程出现死锁状态时，杀死所述子进程。
所述处理模块还包括：第三判断子模块，用于判断未上传的所述临时文件的个数是否达到预设阈值；第二处理子模块，用于当未上传的所述临时文件的个数达到预设阈值时，进入等待状态，以等待未上传的所述临时文件被上传完毕。
所述调用还包括元数据的上传配置参数；所述元数据服务器还包括：上传模块，用于根据所述上传配置参数将所述输出的元数据上传至分布式存储系统。
所述上传配置参数包括上传速率和目标目录，其中，所述上传速率用于控制上传过程的数据传输速率；所述上传模块包括：上传子模块，用于上传所述输出的元数据至所述分布式存储系统的临时目录；重命名子模块，用于当所述输出的元数据被完整的上传完毕时，将所述临时目录重命名为所述目标目录，并标记本次任务已完成。
为了解决上述技术问题，本申请还公开了一种元数据输出客户端，包括：第一配置模块，用于配置元数据的输出配置参数；调用模块，用于向元数据服务器发起对元数据服务进程的调用，以通过分叉函数创建出所述元数据服务进程的子进程，由所述子进程根据所述输出配置函数输出相应的元数据。
所述客户端还包括：第二配置模块，用于配置元数据的上传配置参数。
所述客户端还包括：查询模块，用于发送进度查询请求至元数据服务器；接收模块，用于接收所述元数据服务器返回的任务状态信息。
与现有技术相比，本申请可以获得包括以下技术效果：不再需要恢复所有文件的内存状态，能够得到在分叉操作的时间点相应的输出目标的内存状态，从而保证元数据输出的时效性；不再需要配备额外的计算机，使实现成本降低。此外对当前元数据服务器的内存使用量的增加极少，只会为子进程分配由于父进程的写入操作而修改的内存量。
当然，实施本申请的任一产品必不一定需要同时达到以上所述的所有技术效果。
附图说明
此处所说明的附图用来提供对本申请的进一步理解，构成本申请的一部分，本申请的示意性实施例及其说明用于解释本申请，并不构成对本申请的不当限定。在附图中：
图1是本申请实施例的一种元数据输出方法的流程示意图；
图2是本申请实施例的接收到对元数据服务进程的调用的流程示意图；
图3是本申请实施例控制子进程输出元数据的流程示意图；
图4是本申请实施例的一种元数据输出方法的流程示意图；
图5是本申请实施例上传元数据的流程示意图；
图6是本申请实施例的一种元数据输出方法的流程示意图；
图7是本申请实施例的一种元数据输出方法的流程示意图；
图8是本申请实施例的一种元数据输出方法的流程示意图；
图9是本申请实施例的一种元数据服务器的示例性结构框图；
图10是本申请实施例的一种元数据输出客户端的示例性结构框图。
具体实施方式
以下将配合附图及实施例来详细说明本发明的实施方式，藉此对本发明如何应用技术手段来解决技术问题并达成技术功效的实现过程能充分理解并据以实施。
图1是本申请实施例提供的一种元数据输出方法，适用于服务器设备，该方法包括以下步骤。
在步骤S10中，接收到对元数据服务(MetaServer)进程的调用，调用包括元数据的输出配置参数。
该元数据服务进程运行在元数据服务器中，根据接收到的调用输出相应的元数据。对元数据服务进程的调用包括元数据的输出配置参数。
该输出配置参数包括输出目标、输出格式和输出内容存放路径。其中，输出目标是指需要输出的元数据当前所在的内存目录；输出格式是指元数据输出后的保存形式，如文件形式或者内存目录形式；输出内容存放路径是输出的元数据在内存中的存放路径。例如，输出目标为/var/log/abc，输出格式为文件形式，输出内容存放路径为/tmp/msv/abc；上述输出配置参数将/var/log/abc目录下的元数据以文件形式输出到/tmp/msv/abc目录。
在步骤S11中，通过分叉(fork)函数创建元数据服务进程的子进程。
元数据服务器接收到针对元数据服务进程的调用后，通过分叉函数创建元数据服务 进程的子进程。该子进程用于输出在分叉操作这一时刻与输出目标对应的元数据；父进程继续进行有关元数据服务器的其他操作。该子进程与元数据服务进程(父进程)共享内存地址空间，并且父进程采用写时拷贝技术(Copy-on-Write)，在父进程对内存地址空间有写入操作时才会复制相应变化的内存地址空间分配给子进程。保证子进程对应的内存地址空间在分叉函数的分叉操作完成后可以定格在该分叉操作的时间点来实现元数据在某个时间点上的一致性，并通过该子进程根据输出配置参数输出相应的元数据。
在步骤S12中，控制子进程根据输出配置参数输出相应的元数据。
使元数据的输出不再需要将元数据服务器所有文件的内存状态恢复，通过分叉函数创建子进程，得到在分叉操作的时间点相应的输出目标的内存状态，能够保证元数据输出的时效性。同时，现有技术中在额外配备的计算机设备上恢复内存状态是为了避免与元数据服务进程竞争内存，而采用本申请的元数据输出方法，不再需要配备额外的计算机，使实现成本降低。此外对当前元数据服务器的内存使用量的增加极少，只会为子进程分配由于父进程的写入操作而修改的内存量。
在一个实施例中，步骤S10接收到对元数据服务进程的调用，是来自于客户端的对元数据服务进程的远程过程调用(Remote Procedure Call，RPC)。即由元数据服务器之外的客户端计算机来向元数据服务器发起针对元数据服务进程的调用。此时，针对分布式存储系统中所有元数据的输出操作都由该客户端计算机进行控制操作，通过该客户端计算机来设置输出配置参数并向元数据服务器的元数据服务进程发起远程过程调用。此时能够对元数据的输出进行远程集中控制，在同时为多个集群提供元数据服务的情况下，通过客户端计算机集中控制，有效将降低了系统的维护难度和成本，提高维护效率。
在一个实施例中，如图2所示，步骤S10接收到对元数据服务进程的调用进一步包括以下步骤。
在步骤S101中，接收到对元数据服务进程的调用。
在步骤S102中，判断上一次创建的元数据服务进程的子进程是否已完成执行。当上一次创建的元数据服务进程的子进程已执行完成时，执行步骤S103；当上一次创建的元数据服务进程的子进程未执行完成时，执行步骤S104。
每次分叉操作创建的子进程，在完成元数据的输出之后都会标记相应的状态，例如已完成执行的子进程状态标记为“Done(完成)”。为了避免占用内存资源过多，判断上一次创建的子进程是否已完成执行。
在步骤S103中，保存输出配置参数至本地配置文件。
元数据服务器将本次调用的输出配置参数保存在本地配置文件，以备创建出的子进程来完成元数据输出。
在步骤S104中，返回元数据服务进程当前繁忙的消息，并返回步骤S102。
如果对元数据服务进程的调用来自远程客户端计算机，则将元数据服务进程当前繁忙的消息返回至远程客户端计算机。
在不同时间通过分叉函数创建的子进程所对应的内存空间具有不同的时间属性，内存中的元数据状态也会不同。在需要多个输出目标的元数据时，为了保证所输出的元数据的时效性，将需要输出的多个输出目标同时设置在输出配置参数中，通过对元数据服务进程的一次调用来完成。此时，分叉操作创建的子进程将完成多个输出目标的元数据输出，以保证所有输出目标的时效性。例如，输出配置参数共包括/var/log/abc、/var/log/efg、/var/log/chb等多个输出目标。
在一个实施例中，如图3所示，步骤S12控制子进程根据输出配置参数输出相应的元数据进一步包括以下步骤。
在步骤S121中，控制子进程根据输出配置参数将输出的元数据写入本地内存的临时文件。
子进程将与输出目标对应的元数据写入本地内存的临时文件。当该临时文件的大小达到预设阈值(如200M)或者相应的元数据已全部输出完毕时，将该临时文件命名为服务器上传进程可识别的文件名。可识别的文件名通常是与输出目标相对应的文件名，例如，输出目标为/var/log/abc，则该临时文件命名为abc_tmp，当该临时文件大小已达到阈值需要输出或者元数据已写入完毕需要输出时，将文件名从abc_tmp变为abc，如果按照文件大小的阈值将输出的元数据划分为多个临时文件，则分别命名为：abc1、abc2、……
服务器上传进程用于将输出的元数据上传到分布式存储系统，以实现与元数据相关的其他功能，例如用户数据分析，用户行为统计等等。上传元数据的过程将在后续实施例中详细说明。
在步骤S122中，当元数据输出完毕时，标记子进程已完成执行。
在元数据输出过程中，该子进程通常标记为“等待(waiting)”，元数据输出完毕后，通常标记为“完成(Done)”，以继续下一次元数据输出。将元数据输出到本地内存，能够减少对元数据服务器本地磁盘的访问，避免影响元数据服务器的性能。
在一个实施例中，步骤S12控制子进程根据输出配置参数输出相应的元数据的同时，还需要判断该子进程是否出现死锁状态。当子进程需要输出的元数据被其他进程锁定(如 排他锁)，则该子进程有可能会一直等待其他进程释放，但其他进程需要的内存资源也有可能同时被该子进程锁定，需要等待该子进程释放，此时即出现死锁状态。检测到子进程进入死锁状态后，杀死该子进程，将该子进程标记为“失败”，为了保证元数据输出功能的正常运行，需要杀死已进入死锁状态的子进程。例如，利用该子进程的进程标识(PID)和相应命令杀死进入死锁状态的子进程。
图4是本申请实施例提供的一种元数据输出方法，适用于服务器设备，该方法包括以下步骤。
在步骤S20中，接收到对元数据服务进程的调用，调用包括元数据的输出配置参数和上传配置参数。
在设置输出配置参数的同时，设置相应的上传配置参数。上传配置参数包括上传速率、目标目录和压缩参数。其中，上传速率用于控制上传过程的数据传输速率，例如该上传速率为20Mb/S，则上传过程的数据传输速率会小于或等于20Mb/S。该目标目录是将输出的元数据上传至的目录，该目标目录可位于本地集群，也可以位于其他远程集群。该压缩参数用于控制输出的元数据在上传之前是否需要压缩处理，当需要上传的元数据过多时，为了提高效率，通过该压缩参数设置在上传之前对数据进行压缩处理。
在步骤S21中，通过分叉函数创建元数据服务进程的子进程。
在步骤S22中，控制子进程根据输出配置参数输出相应的元数据，同时根据上传配置参数将输出的元数据上传至分布式存储系统。
在元数据服务器中设置服务器上传进程，用于输出的元数据上传至分布式存储系统。该服务器上传进程被设置为每隔预设时长启动一次，以检查是否有输出的元数据需要上传。例如，利用Crontab命令设置每分钟运行一次服务器上传进程。服务器上传进程检测元数据服务器的本地配置文件是否包括输出配置参数和上传配置参数，检测到上述配置参数时开始根据上传配置参数进行元数据的上传操作。
元数据的输出过程和上传过程是并行过程，将输出的元数据尽快上传以减少对内存空间的占用，避免影响元数据服务器的性能。在控制子进程根据输出配置参数将输出的元数据写入本地内存的临时文件之前，判断当前未上传的临时文件的个数是否达到预设阈值，例如判断未上传的临时文件的个数是否达到5个。当未上传的临时文件的个数未达到预设阈值时，继续将输出的元数据写入本地内存的临时文件；当未上传的临时文件的个数达到预设阈值时，控制该子进程进入等待状态，以等待未上传的临时文件被上传完毕。从而使元数据的输出速度能够自适应上传速度，尽量保证内存不会被过多占用。
元数据服务器每次启动该服务器上传进程时，需要打开位于本地磁盘的执行文件。在打开执行文件时，对该执行文件添加排他锁，此时元数据服务器无法再启动第二个服务器上传进程，需等待本次启动的服务器上传进程执行完成后，将对执行文件的排他锁释放，才能再次启动下一个服务器上传进程。以防止上传元数据的过程占用网络带宽过多，影响元数据服务器的其他进程的网络数据传输。
元数据服务器成功启动该服务器上传进程之后，根据本地配置文件判断是否有需要执行上传的任务。如果没有则退出服务器上传进程；如果有则开始上传元数据并检测该任务是否已完成执行(元数据是否已全部输出)，检测到该任务已完成执行时，设置本次上传为该任务的最后一次上传，直到检测到该任务的最后一次上传也已经上传完毕。
如图5所示，步骤S22根据上传配置参数将输出的元数据上传至分布式存储系统的过程中进一步包括：
在步骤S221中，上传输出的元数据至分布式存储系统的临时目录。
控制启动的服务器上传进程上传输出的元数据，并在上传的目标集群创建一临时目录以保存上传中的元数据。
在步骤S222中，当输出的元数据被完整的上传完毕时，将临时目录重命名为目标目录，并标记本次任务已完成。
即该任务的最后一次上传完毕之后，将临时目录修改为上传配置参数中的目标目录，从而将输出的元数据保存到分布式存储系统的目标目录，例如将临时目录/tmp/upload修改为目标目录/var/lib/abc。在元数据的上传完成后，将任务标记为“上传已完成”。
图6是本申请实施例提供的一种元数据输出方法，该方法包括以下步骤。
在步骤S301中，接收到对元数据服务进程的调用，调用包括元数据的输出配置参数和上传配置参数。
在步骤S302中，判断上一次创建的元数据服务进程的子进程是否已完成执行。当上一次创建的元数据服务进程的子进程已执行完成时，执行步骤S304；当上一次创建的元数据服务进程的子进程未执行完成时，执行步骤S303。
在步骤S303中，返回元数据服务进程当前繁忙的消息，并返回步骤S302。
在步骤S304中，保存输出配置参数和上传配置参数至本地配置文件。
在步骤S305中，通过分叉函数创建元数据服务进程的子进程。
在步骤S306中，控制该子进程从本地配置文件中获取输出配置参数。
在步骤S307中，判断当前未上传的临时文件的个数是否达到预设阈值。当未上传的 文件个数未达到预设阈值时，执行步骤S308；当未上传的文件个数达到预设阈值时，执行步骤S309。
在步骤S308中，控制该子进程进入等待状态，以等待未上传的临时文件被上传完毕。
在步骤S309中，将输出的元数据写入本地内存的临时文件。
在步骤S310中，当临时文件的大小到达了某个阈值或者该输出元数据的任务执行完成时，将临时文件命名为服务器上传进程可识别的文件名，以等待服务器上传进程将输出的元数据上传至分布式存储系统。
步骤S305通过分叉函数创建元数据服务进程的子进程之后，元数据服务器在控制该子进程输出元数据时，还要并行的检测该子进程是否出现死锁状态，包括以下相关步骤。
在步骤S311中，判断子进程是否进入死锁状态。当子进程进入死锁状态时，执行步骤S312；当子进程未进入死锁状态时，继续执行步骤S311。
在步骤S312中，杀死子进程并标记本次输出任务执行失败。
本申请实施例中元数据服务进程执行的元数据输出过程，与服务器上传进程执行的元数据上传过程是并行的。元数据服务器每隔预设时长启动一次服务器上传进程，元数据上传过程包括以下步骤。
在步骤S313中，打开位于本地磁盘的服务器上传进程的执行文件。
在步骤S314中，判断该执行文件是否被成功打开。当该执行文件打开不成功时，返回步骤S313；当该执行文件被成功打开时，执行步骤S315。
在步骤S315中，根据本地配置文件判断是否有任务在运行。当判断出没有任务在运行时，执行步骤S316；当判断出有任务在运行时，执行步骤S317。
在步骤S316中，关闭服务器上传进程。
在步骤S317中，将输出的元数据上传至分布式存储系统的临时目录。
在步骤S318中，判断子进程输出元数据的任务是否已完成。当子进程输出元数据的任务已完成时，执行步骤S319；当子进程输出元数据的任务未完成时，返回步骤S317。
在步骤S319中，将本次上传设置为本任务的最后一次上传。
在步骤S320中，当输出的元数据被完整的上传完毕时，将临时目录重命名为目标目录，并标记本次上传已完成。
在输出元数据的同时异步并发上传该输出的元数据到分布式存储系统，及时回收内存，以达到不限量的元数据输出并减少网络带宽争抢。
图7是本申请实施例提供的一种元数据输出方法，适用于客户端设备，包括以下步 骤。
在步骤S40中，配置元数据的输出配置参数。
客户端计算机根据用户操作配置元数据的输出配置参数，包括输出目标、输出格式和输出内容存放路径。
在步骤S41中，向元数据服务器发起对元数据服务进程的调用，以通过分叉函数创建出元数据服务进程的子进程，由子进程根据所述输出配置函数输出相应的元数据。
客户端计算机向元数据服务器发起对元数据服务进程的调用，该调用包括步骤S40中配置的输出配置参数。通过该调用使元数据服务器利用分叉函数创建出元数据服务进程的子进程，由子进程根据所述输出配置函数输出相应的元数据。
该调用可以是远程过程调用，从而在由多个集群组成的分布式存储系统中，实现对元数据输出的远程集中配置和管理，多个集群可以并行操作，降低系统维护难度和成本，提高效率。
在一个实施例中，步骤S40在配置元数据的输出配置参数的同时，还配置元数据的上传配置参数。上传配置参数包括上传速率、目标目录和压缩参数。使元数据服务器在输出相应的元数据后，根据目标目录将输出的元数据上传到分布式存储系统。
图8是本申请实施例提供的一种元数据输出方法，适用于客户端设备，包括以下步骤。
在步骤S50中，配置元数据的输出配置参数和上传配置参数。
在步骤S51中，向元数据服务器发起对元数据服务进程的调用，以通过分叉函数创建出所述元数据服务进程的子进程，由所述子进程根据所述输出配置函数输出相应的元数据，并由元数据服务器的服务器上传进程将输出的元数据上传到分布式存储系统。
在步骤S52中，发送进度查询请求至元数据服务器。
在配置元数据的输出配置参数和上传配置参数并向元数据服务器发起调用后，客户端计算机生成相应的配置标识。可根据该配置标识向元数据服务器发送进度查询请求，以查询相应的配置任务的执行进度。发送的进度查询请求也可以不包含该配置标识，此时则默认查询当前正在执行的配置任务的进度或者最后一个执行完成任务状态。
在步骤S53中，接收元数据服务器返回的任务状态信息。
通过该任务状态信息，可查询到该配置任务的输出配置参数、上传配置参数和任务进度状态。如果是正在执行中的配置任务，该任务进度状态包括“执行中”和“等待中”；如果是已经执行完成的配置任务，该任务进度状态包括“执行失败”和“执行失败”， 其中该任务进度状态为“执行失败”时还能够查询到造成执行失败的原因。
图9是本申请实施例提供的一种元数据服务器，所述元数据服务器包括：
接收模块60，用于接收到对元数据服务(MetaServer)进程的调用，调用包括元数据的输出配置参数；
创建模块61，用于通过分叉(fork)函数创建元数据服务进程的子进程；
处理模块62，用于控制子进程根据输出配置参数输出相应的元数据。
该接收模块60包括：
接收子模块，用于接收到来自客户端的对元数据服务进程的远程过程调用。
在一个实施例中，该接收模块60包括：
第一判断子模块，用于判断上一次创建的元数据服务进程的子进程是否已完成执行；
保存子模块，用于当上一次创建的元数据服务进程的子进程已完成执行时，保存输出配置参数至本地配置文件；
返回子模块，用于当上一次创建的元数据服务进程的子进程未完成执行时，返回元数据服务进程当前繁忙的消息。
在一个实施例中，该处理模块62包括：
写入子模块，用于控制子进程根据输出配置参数将输出的元数据写入本地内存的临时文件；
标记子模块，用于当元数据输出完毕时，标记子进程已完成执行。
在一个实施例中，输出配置参数包括多个输出对象。
在一个实施例中，该处理模块62包括：
第二判断子模块，用于判断子进程是否出现死锁状态；
第一处理子模块，用于当子进程出现死锁状态时，杀死子进程。
在一个实施例中，该处理模块62还包括：
第三判断子模块，用于判断未上传的临时文件的个数是否达到预设阈值；
第二处理子模块，用于当未上传的临时文件的个数达到预设阈值时，进入等待状态，以等待未上传的临时文件被上传完毕。
在一个实施例中，该调用还包括元数据的上传配置参数；元数据服务器还包括：
上传模块，用于根据上传配置参数将输出的元数据上传至分布式存储系统。
上传配置参数包括上传速率和目标目录，其中，上传速率用于控制上传过程的数据传输速率；该上传模块包括：
上传子模块，用于上传输出的元数据至分布式存储系统的临时目录；
重命名子模块，用于当输出的元数据被完整的上传完毕时，将临时目录重命名为目标目录，并标记本次任务已完成。
图10是本申请实施例提供的一种元数据输出客户端，包括：
第一配置模块70，用于配置元数据的输出配置参数；
调用模块71，用于向元数据服务器发起对元数据服务进程的调用，以通过分叉函数创建出元数据服务进程的子进程，由子进程根据输出配置函数输出相应的元数据。
在一个实施例中，该客户端还包括：
第二配置模块，用于配置元数据的上传配置参数。
在一个实施例中，该客户端还包括：
查询模块，用于发送进度查询请求至元数据服务器；
接收模块，用于接收元数据服务器返回的任务状态信息。
下面结合应用场景对本申请实施例做进一步说明。
本申请实施例提供的元数据输出方法、服务器和客户端可以用于互联网企业的云计算平台，为云计算平台的分布式文件系统提供支持。对于分布在不同地区的多个存储集群，可通过客户端对各个集群的元数据服务器进行集中管理，从而控制各个集群的元数据输出。在分布式文件系统中，文件的元数据通常用于记录相应文件属性信息，例如文件的存储路径、占用空间、访问次数、读写操作次数等等，输出这些元数据使云计算平台能够对用户行为和该行为产生的文件进行分析。例如，通过客户端计算机配置元数据的输出配置参数和上传配置参数，将分布式文件系统中与某网站的用户文件的对应的元数据输出并上传到指定的目录，并每隔预设周期(例如每天或者每周)完成一次输出和上传。从上传的目标目录中获取与不同时间点对应的元数据，利用不同时间点的元数据分析该网站的用户文件的变化，进一步分析用户行为变化和趋势，为云计算平台提供可靠的元数据支持。
此外，在ODPS(Open Data Processing Service，开放数据处理服务)计算平台中也可以应用本申请实施例提供的元数据输出方法、服务器和客户端，ODPS提供针对TB/PB级数据、实时性要求不高的分布式处理能力，应用于数据分析、挖掘、商业智能等领域，使用户可以将更多的时间来专注于用户数据挖掘和分析。利用本申请实施例提供的元数据输出方法，用户可以根据实际的用户数据分析需求配置不同的输出参数。利用输出的元数据来分析用户文件的变化，用户文件每天的增加量，各个集群的用户文件存储情况， 已确定是否需要对相应的集群进行扩容。根据输出的元数据分析用户文件的访问量，已却确定需要将已成为“热点”数据的用户文件进行迁移等等；或者利用输出的元数据分析不同特征用户的分布情况。本申请实施例提供的元数据输出方法是为互联网企业推出的云计算平台以及大数据服务提供支持的基础性方案，为多集群的集中管理以及元数据需要频繁输出的场景带来了极大便利，提高了云计算平台以及大数据服务在用户数据挖掘和分析方面的效率。
在一个典型的配置中，计算设备包括一个或多个处理器(CPU)、输入/输出接口、网络接口和内存。
内存可能包括计算机可读介质中的非永久性存储器，随机存取存储器(RAM)和/或非易失性内存等形式，如只读存储器(ROM)或闪存(flash RAM)。内存是计算机可读介质的示例。
计算机可读介质包括永久性和非永久性、可移动和非可移动媒体可以由任何方法或技术来实现信息存储。信息可以是计算机可读指令、数据结构、程序的模块或其他数据。计算机的存储介质的例子包括，但不限于相变内存(PRAM)、静态随机存取存储器(SRAM)、动态随机存取存储器(DRAM)、其他类型的随机存取存储器(RAM)、只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、快闪记忆体或其他内存技术、只读光盘只读存储器(CD-ROM)、数字多功能光盘(DVD)或其他光学存储、磁盒式磁带，磁带磁磁盘存储或其他磁性存储设备或任何其他非传输介质，可用于存储可以被计算设备访问的信息。按照本文中的界定，计算机可读介质不包括非暂存电脑可读媒体(transitory media)，如调制的数据信号和载波。
如在说明书及权利要求当中使用了某些词汇来指称特定组件。本领域技术人员应可理解，硬件制造商可能会用不同名词来称呼同一个组件。本说明书及权利要求并不以名称的差异来作为区分组件的方式，而是以组件在功能上的差异来作为区分的准则。如在通篇说明书及权利要求当中所提及的“包含”为一开放式用语，故应解释成“包含但不限定于”。“大致”是指在可接收的误差范围内，本领域技术人员能够在一定误差范围内解决所述技术问题，基本达到所述技术效果。此外，“耦接”一词在此包含任何直接及间接的电性耦接手段。因此，若文中描述一第一装置耦接于一第二装置，则代表所述第一装置可直接电性耦接于所述第二装置，或通过其他装置或耦接手段间接地电性耦接至所述第二装置。说明书后续描述为实施本发明的较佳实施方式，然所述描述乃以说明本发明的一般原则为目的，并非用以限定本发明的范围。本发明的保护范围当视所附权 利要求所界定者为准。
还需要说明的是，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的商品或者系统不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种商品或者系统所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括所述要素的商品或者系统中还存在另外的相同要素。
上述说明示出并描述了本发明的若干优选实施例，但如前所述，应当理解本发明并非局限于本文所披露的形式，不应看作是对其他实施例的排除，而可用于各种其他组合、修改和环境，并能够在本文所述发明构想范围内，通过上述教导或相关领域的技术或知识进行改动。而本领域人员所进行的改动和变化不脱离本发明的精神和范围，则都应在本发明所附权利要求的保护范围内。

Claims (24)

  1. 一种元数据输出方法，其特征在于，包括：
    接收到对元数据服务进程的调用，所述调用包括元数据的输出配置参数；
    通过分叉函数创建所述元数据服务进程的子进程；
    控制所述子进程根据所述输出配置参数输出相应的元数据。
  2. 如权利要求1所述的方法，其特征在于，所述接收到对元数据服务进程的调用，所述调用包括元数据的输出配置参数包括：
    接收到来自客户端的对所述元数据服务进程的远程过程调用。
  3. 如权利要求1所述的方法，其特征在于，所述接收到对元数据服务进程的调用，所述调用包括元数据的输出配置参数包括：
    判断上一次创建的所述元数据服务进程的子进程是否已完成执行；
    当上一次创建的所述元数据服务进程的子进程已完成执行时，保存所述输出配置参数至本地配置文件；
    当上一次创建的所述元数据服务进程的子进程未完成执行时，返回所述元数据服务进程当前繁忙的消息。
  4. 如权利要求1所述的方法，其特征在于，所述控制所述子进程根据所述输出配置参数输出相应的元数据包括：
    控制所述子进程根据所述输出配置参数将输出的元数据写入本地内存的临时文件；
    当所述元数据输出完毕时，标记所述子进程已完成执行。
  5. 如权利要求1所述的方法，其特征在于，所述输出配置参数包括多个输出对象。
  6. 如权利要求1所述的方法，其特征在于，所述控制所述子进程根据所述输出配置参数输出相应的元数据包括：
    判断所述子进程是否出现死锁状态；
    当所述子进程出现死锁状态时，杀死所述子进程。
  7. 如权利要求4所述的方法，其特征在于，所述控制所述子进程根据所述输出配置参数将输出的元数据写入本地内存的临时文件之前，所述方法还包括：
    判断未上传的所述临时文件的个数是否达到预设阈值；
    当未上传的所述临时文件的个数达到预设阈值时，进入等待状态，以等待未上传的所述临时文件被上传完毕。
  8. 如权利要求1-7任一所述的方法，其特征在于，所述调用还包括元数据的上传 配置参数；所述控制所述子进程根据所述输出配置参数输出相应的元数据的同时，所述方法还包括：
    根据所述上传配置参数将所述输出的元数据上传至分布式存储系统。
  9. 如权利要求8所述的方法，其特征在于，所述上传配置参数包括上传速率和目标目录，其中，所述上传速率用于控制上传过程的数据传输速率；
    所述根据所述上传配置参数将所述输出的元数据上传至分布式存储系统包括：
    上传所述输出的元数据至所述分布式存储系统的临时目录；
    当所述输出的元数据被完整的上传完毕时，将所述临时目录重命名为所述目标目录，并标记本次任务已完成。
  10. 一种元数据输出方法，其特征在于，包括：
    配置元数据的输出配置参数；
    向元数据服务器发起对元数据服务进程的调用，以通过分叉函数创建出所述元数据服务进程的子进程，由所述子进程根据所述输出配置函数输出相应的元数据。
  11. 如权利要求10所述的方法，其特征在于，所述配置元数据的输出配置参数的同时，所述方法还包括：
    配置元数据的上传配置参数。
  12. 如权利要求10或11所述的方法，其特征在于，所述方法还包括：
    发送进度查询请求至元数据服务器；
    接收所述元数据服务器返回的任务状态信息。
  13. 一种元数据服务器，其特征在于，所述元数据服务器包括：
    接收模块，用于接收到对元数据服务进程的调用，所述调用包括元数据的输出配置参数；
    创建模块，用于通过分叉函数创建所述元数据服务进程的子进程；
    处理模块，用于控制所述子进程根据所述输出配置参数输出相应的元数据。
  14. 如权利要求13所述的元数据服务器，其特征在于，所述接收模块包括：
    接收子模块，用于接收到来自客户端的对所述元数据服务进程的远程过程调用。
  15. 如权利要求13所述的元数据服务器，其特征在于，所述接收模块包括：
    第一判断子模块，用于判断上一次创建的所述元数据服务进程的子进程是否已完成执行；
    保存子模块，用于当上一次创建的所述元数据服务进程的子进程已完成执行时，保 存所述输出配置参数至本地配置文件；
    返回子模块，用于当上一次创建的所述元数据服务进程的子进程未完成执行时，返回所述元数据服务进程当前繁忙的消息。
  16. 如权利要求13所述的元数据服务器，其特征在于，所述处理模块包括：
    写入子模块，用于控制所述子进程根据所述输出配置参数将输出的元数据写入本地内存的临时文件；
    标记子模块，用于当所述元数据输出完毕时，标记所述子进程已完成执行。
  17. 如权利要求13所述的元数据服务器，其特征在于，所述输出配置参数包括多个输出对象。
  18. 如权利要求13所述的元数据服务器，其特征在于，所述处理模块包括：
    第二判断子模块，用于判断所述子进程是否出现死锁状态；
    第一处理子模块，用于当所述子进程出现死锁状态时，杀死所述子进程。
  19. 如权利要求16所述的元数据服务器，其特征在于，所述处理模块还包括：
    第三判断子模块，用于判断未上传的所述临时文件的个数是否达到预设阈值；
    第二处理子模块，用于当未上传的所述临时文件的个数达到预设阈值时，进入等待状态，以等待未上传的所述临时文件被上传完毕。
  20. 如权利要求13-19任一所述的元数据服务器，其特征在于，所述调用还包括元数据的上传配置参数；所述元数据服务器还包括：
    上传模块，用于根据所述上传配置参数将所述输出的元数据上传至分布式存储系统。
  21. 如权利要求20所述的元数据服务器，其特征在于，所述上传配置参数包括上传速率和目标目录，其中，所述上传速率用于控制上传过程的数据传输速率；
    所述上传模块包括：
    上传子模块，用于上传所述输出的元数据至所述分布式存储系统的临时目录；
    重命名子模块，用于当所述输出的元数据被完整的上传完毕时，将所述临时目录重命名为所述目标目录，并标记本次任务已完成。
  22. 一种元数据输出客户端，其特征在于，包括：
    第一配置模块，用于配置元数据的输出配置参数；
    调用模块，用于向元数据服务器发起对元数据服务进程的调用，以通过分叉函数创建出所述元数据服务进程的子进程，由所述子进程根据所述输出配置函数输出相应的元 数据。
  23. 如权利要求22所述的客户端，其特征在于，所述客户端还包括：
    第二配置模块，用于配置元数据的上传配置参数。
  24. 如权利要求22或23所述的客户端，其特征在于，所述客户端还包括：
    查询模块，用于发送进度查询请求至元数据服务器；
    接收模块，用于接收所述元数据服务器返回的任务状态信息。
PCT/CN2016/094320 2015-08-19 2016-08-10 元数据输出方法、客户端和元数据服务器 WO2017028719A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201510512514.XA CN106469087B (zh) 2015-08-19 2015-08-19 元数据输出方法、客户端和元数据服务器
CN201510512514.X 2015-08-19

Publications (1)

Publication Number Publication Date
WO2017028719A1 true WO2017028719A1 (zh) 2017-02-23

Family

ID=58050688

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/094320 WO2017028719A1 (zh) 2015-08-19 2016-08-10 元数据输出方法、客户端和元数据服务器

Country Status (2)

Country Link
CN (1) CN106469087B (zh)
WO (1) WO2017028719A1 (zh)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110209548A (zh) * 2018-04-19 2019-09-06 深圳市腾讯计算机系统有限公司 服务控制方法、系统、电子设备及计算机可读存储介质
CN111435299A (zh) * 2019-01-14 2020-07-21 阿里巴巴集团控股有限公司 一种应用程序的处理方法及装置
CN111600949A (zh) * 2020-05-14 2020-08-28 上海鸿翼软件技术股份有限公司 一种数据传输方法、装置、设备及计算机可读存储介质
CN113535695A (zh) * 2021-06-21 2021-10-22 中盾创新档案管理(北京)有限公司 一种基于进程调度的档案更新方法
CN115964353A (zh) * 2023-03-10 2023-04-14 阿里巴巴（中国）有限公司 一种分布式文件系统及其访问计量方法

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109165112B (zh) * 2018-08-16 2022-02-18 郑州云海信息技术有限公司 一种元数据集群的故障恢复方法、系统及相关组件
CN110286850B (zh) * 2019-05-15 2023-05-09 镕铭微电子(济南)有限公司 固态硬盘元数据的写入方法、恢复方法及固态硬盘
CN113448722A (zh) * 2020-03-28 2021-09-28 阿里巴巴集团控股有限公司 进程内存的映射方法和基于无服务器架构的实例处理方法
CN111984446A (zh) * 2020-08-07 2020-11-24 苏州浪潮智能科技有限公司 一种基于子进程的多控制器系统的运转方法和装置
CN112084058B (zh) * 2020-09-23 2024-08-27 北京金山云网络技术有限公司 进程处理方法、装置和电子设备
CN113687834B (zh) * 2021-10-27 2022-02-18 深圳华锐金融技术股份有限公司 分布式系统节点部署方法、装置、设备及介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CA2312597A1 (en) * 2000-07-05 2002-01-05 Benjamin Zhang A method for data retrieval using tree-structured query with returned result set in xml format
CN1834956A (zh) * 2005-03-18 2006-09-20 联想(北京)有限公司 多线程元数据的处理方法
CN101576912A (zh) * 2009-06-03 2009-11-11 中兴通讯股份有限公司 实现分布式文件系统异步输入输出接口的系统及读写方法
CN102521232A (zh) * 2011-11-09 2012-06-27 Ut斯达康通讯有限公司 一种互联网元数据的分布式采集处理系统及方法
CN103647666A (zh) * 2013-12-13 2014-03-19 北京中创信测科技股份有限公司 一种统计呼叫详细记录报文并实时输出结果的方法及装置

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN100440795C (zh) * 2004-01-05 2008-12-03 华为技术有限公司 一种系统日志实现方法和装置
US7353241B2 (en) * 2004-03-24 2008-04-01 Microsoft Corporation Method, medium and system for recovering data using a timeline-based computing environment
CN101467453B (zh) * 2006-06-15 2011-12-07 索尼株式会社 信息处理装置以及信息处理方法
CN100587692C (zh) * 2007-01-26 2010-02-03 华中科技大学 一种提高元数据服务可靠性的方法及其系统
CN101286127B (zh) * 2008-05-08 2010-06-02 华中科技大学 一种多叉日志存储的连续数据保护和恢复方法
CN101594252A (zh) * 2009-06-01 2009-12-02 中兴通讯股份有限公司 一种海量日志存储管理系统和方法
CN104156298B (zh) * 2014-08-19 2017-02-15 腾讯科技（深圳）有限公司 一种应用监控方法及装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CA2312597A1 (en) * 2000-07-05 2002-01-05 Benjamin Zhang A method for data retrieval using tree-structured query with returned result set in xml format
CN1834956A (zh) * 2005-03-18 2006-09-20 联想(北京)有限公司 多线程元数据的处理方法
CN101576912A (zh) * 2009-06-03 2009-11-11 中兴通讯股份有限公司 实现分布式文件系统异步输入输出接口的系统及读写方法
CN102521232A (zh) * 2011-11-09 2012-06-27 Ut斯达康通讯有限公司 一种互联网元数据的分布式采集处理系统及方法
CN103647666A (zh) * 2013-12-13 2014-03-19 北京中创信测科技股份有限公司 一种统计呼叫详细记录报文并实时输出结果的方法及装置

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110209548A (zh) * 2018-04-19 2019-09-06 深圳市腾讯计算机系统有限公司 服务控制方法、系统、电子设备及计算机可读存储介质
CN110209548B (zh) * 2018-04-19 2023-07-14 深圳市腾讯计算机系统有限公司 服务控制方法、系统、电子设备及计算机可读存储介质
CN111435299A (zh) * 2019-01-14 2020-07-21 阿里巴巴集团控股有限公司 一种应用程序的处理方法及装置
CN111435299B (zh) * 2019-01-14 2023-06-20 阿里巴巴集团控股有限公司 一种应用程序的处理方法及装置
CN111600949A (zh) * 2020-05-14 2020-08-28 上海鸿翼软件技术股份有限公司 一种数据传输方法、装置、设备及计算机可读存储介质
CN111600949B (zh) * 2020-05-14 2024-03-15 上海鸿翼软件技术股份有限公司 一种数据传输方法、装置、设备及计算机可读存储介质
CN113535695A (zh) * 2021-06-21 2021-10-22 中盾创新档案管理(北京)有限公司 一种基于进程调度的档案更新方法
CN115964353A (zh) * 2023-03-10 2023-04-14 阿里巴巴（中国）有限公司 一种分布式文件系统及其访问计量方法
CN115964353B (zh) * 2023-03-10 2023-08-22 阿里巴巴（中国）有限公司 一种分布式文件系统及其访问计量方法

Also Published As

Publication number Publication date
CN106469087A (zh) 2017-03-01
CN106469087B (zh) 2020-06-05

Similar Documents

Publication Publication Date Title
WO2017028719A1 (zh) 元数据输出方法、客户端和元数据服务器
US11422982B2 (en) Scaling stateful clusters while maintaining access
US10489422B2 (en) Reducing data volume durability state for block-based storage
US9983825B2 (en) Efficient data volume replication for block-based storage
US11711420B2 (en) Automated management of resource attributes across network-based services
US11042503B1 (en) Continuous data protection and restoration
US11860741B2 (en) Continuous data protection
US10621049B1 (en) Consistent backups based on local node clock
US10776174B2 (en) Managing hosted resources across different virtualization platforms
US12013764B2 (en) Past-state backup generator and interface for database systems
JP6381801B2 (ja) 格納クライアントファイルシステム用格納ボリュームの動的スケーリング
US11868324B2 (en) Remote durable logging for journaling file systems
US10509696B1 (en) Error detection and mitigation during data migrations
WO2019109854A1 (zh) 分布式数据库数据处理方法、装置、存储介质及电子装置
WO2017041649A1 (zh) 一种应用部署方法及设备
US9910881B1 (en) Maintaining versions of control plane data for a network-based service control plane
US11042454B1 (en) Restoration of a data source
US10452637B1 (en) Migration of mutable data sets between data stores
US11886439B1 (en) Asynchronous change data capture for direct external transmission
US12038946B1 (en) Minimizing connection loss when changing database query engine versions
CN114328376A (zh) 快照创建方法、装置、设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16836583

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16836583

Country of ref document: EP

Kind code of ref document: A1