CN109597689B - Distributed file system memory optimization method, device, equipment and medium - Google Patents
Distributed file system memory optimization method, device, equipment and medium Download PDFInfo
- Publication number
- CN109597689B CN109597689B CN201811504792.0A CN201811504792A CN109597689B CN 109597689 B CN109597689 B CN 109597689B CN 201811504792 A CN201811504792 A CN 201811504792A CN 109597689 B CN109597689 B CN 109597689B
- Authority
- CN
- China
- Prior art keywords
- osd
- array
- target array
- osdmanager
- file system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 71
- 238000005457 optimization Methods 0.000 title claims abstract description 26
- 230000006870 function Effects 0.000 claims description 15
- 238000004590 computer program Methods 0.000 claims description 12
- 238000004806 packaging method and process Methods 0.000 claims description 9
- 238000003780 insertion Methods 0.000 claims description 4
- 230000037431 insertion Effects 0.000 claims description 4
- 238000012217 deletion Methods 0.000 claims description 3
- 230000037430 deletion Effects 0.000 claims description 3
- 238000003491 array Methods 0.000 abstract description 7
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
Abstract
The application discloses a distributed file system memory optimization method, a device, equipment and a medium, comprising the following steps: determining an array related to OSDMap in each OSD instance from all OSD instances of the OSD module process to obtain a target array; removing the target array in all OSD examples of the OSD module process; and adding the target array in an OSDManager of the OSD module process, and controlling all OSD instances in the OSD module process to share the target array in the OSDManager. According to the method and the device, the target array in each OSD instance is removed, and only one target array which can be shared by all OSD instances is arranged in the OSDManager, so that a plurality of target arrays which originally need to consume memory resources can be reduced to one target array, and the memory resource consumption of the distributed file system is effectively reduced.
Description
Technical Field
The present application relates to the field of storage technologies, and in particular, to a method, an apparatus, a device, and a medium for optimizing a memory of a distributed file system.
Background
At present, the distributed file system is applied in many fields and industries, and has a very good market prospect. In a distributed file system, memory is a very critical factor affecting the performance of the distributed file system. However, the consumption of memory resources in the existing distributed file system is large, which affects the performance of the distributed file system.
Therefore, how to further reduce the memory resource consumption of the distributed file system is a problem to be solved at present.
Disclosure of Invention
In view of this, an object of the present application is to provide a method, an apparatus, a device, and a medium for optimizing a memory of a distributed file system, which can further reduce the memory resource consumption of the distributed file system. The specific scheme is as follows:
in a first aspect, the present application discloses a distributed file system memory optimization method, including:
determining an array related to OSDMap in each OSD instance from all OSD instances of the OSD module process to obtain a target array;
removing the target array in all OSD examples of the OSD module process;
and adding the target array in an OSDManager of the OSD module process, and controlling all OSD instances in the OSD module process to share the target array in the OSDManager.
Optionally, the target array includes any one or a combination of any several of a map _ cache array, a map _ bl _ cache array, and a map _ bl _ inc _ cache array.
Optionally, the method for optimizing a memory of a distributed file system further includes:
and packaging an operation function of a preset operation aiming at the target array into the OSDManager so that any OSD instance can finish the preset operation aiming at the target array by calling the operation function in the OSDManager.
Optionally, the preset operation includes any one or a combination of any several of an insert operation, a search operation, and a delete operation.
In a second aspect, the present application discloses a distributed file system memory optimization apparatus, including:
the data determining module is used for determining an array which is related to the OSDMap and exists in each OSD instance from all OSD instances of the OSD module process to obtain a target array;
the array eliminating module is used for eliminating the target array in all OSD examples of the OSD module process;
and the array adding module is used for adding the target array in an OSDManager of the OSD module process and controlling all OSD instances in the OSD module process to share the target array in the OSDManager.
Optionally, the target array includes any one or a combination of any several of a map _ cache array, a map _ bl _ cache array, and a map _ bl _ inc _ cache array.
Optionally, the distributed file system memory optimization apparatus further includes:
and the function packaging module is used for packaging an operation function of preset operation aiming at the target array into the OSDManager so that any OSD instance can finish the preset operation aiming at the target array by calling the operation function in the OSDManager.
Optionally, the preset operation includes any one or a combination of any several of an insert operation, a search operation, and a delete operation.
In a third aspect, the present application discloses a distributed file system memory optimization device, including:
a memory for storing a computer program;
and the processor is used for executing the computer program to realize the distributed file system memory optimization method disclosed in the foregoing.
In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the distributed file system memory optimization method disclosed above.
Therefore, the target array related to the OSDMap of each OSD instance in the OSD module process of the distributed file system is determined, then the target array in each OSD instance is removed, the target array is added to the OSDManager of the OSD module process, and then all OSD instances are controlled to share the target array added to the OSDManager. Compared with the prior art that target arrays which need to consume memory resources and are related to the OSDMap are independently arranged in each OSD example, the target arrays in each OSD example are removed, and only one target array which can be shared by all OSD examples is arranged in the OSDManager, so that a plurality of target arrays which need to consume memory resources originally can be reduced to one target array, specifically, because 36 OSD examples are included in one OSD module process under the normal condition, the method is equivalent to the reduction of 36 target arrays which need to consume memory resources originally to 1 target array. In conclusion, the memory resource consumption of the distributed file system is effectively reduced.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a flowchart of a distributed file system memory optimization method disclosed in the present application;
FIG. 2 is a flowchart of a specific distributed file system memory optimization method disclosed herein;
fig. 3 is a schematic structural diagram of a distributed file system memory optimization apparatus disclosed in the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only some embodiments of the present application, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The embodiment of the application discloses a distributed file system memory optimization method, which is shown in fig. 1 and comprises the following steps:
step S11: and determining an array related to the OSDMap in each OSD instance from all OSD instances of the OSD module process to obtain a target array.
It should be noted that the OSD module process of the existing distributed file system includes 1 OSDManager and 36 OSD instances, and the 36 OSD instances are managed by the OSDManager.
In this embodiment, the target array includes any one or combination of map _ cache array, map _ bl _ cache array, and map _ bl _ inc _ cache array.
Step S12: and eliminating the target array in all OSD examples of the OSD module process.
In the prior art, each OSD instance includes one target array, and in the embodiment of the present application, the target array in each OSD instance is eliminated, that is, the modified OSD instance does not include the target array inside any more.
Step S13: and adding the target array in an OSDManager of the OSD module process, and controlling all OSD instances in the OSD module process to share the target array in the OSDManager.
In the prior art, the OSDManager does not include the target array therein, and in the embodiment of the present application, after the target array is determined, the target array is added to the OSDManager, so that the changed OSDManager includes one target array, and subsequently, each OSD instance modified in the embodiment can share the target array in the changed OSDManager, thereby reducing the memory resource consumption.
As can be seen, in the embodiment of the present application, a target array related to OSDMap of each OSD instance in the OSD module process of the distributed file system is determined, then the target array in each OSD instance is removed, the target array is added to the OSDManager of the OSD module process, and then all OSD instances are controlled to share the target array added to the OSDManager. Compared with the prior art that a target array which needs to consume memory resources and is related to the OSDMap is independently arranged in each OSD example, the target array in each OSD example is removed, and only one target array which can be shared by all OSD examples is arranged in the OSDManager, so that a plurality of target arrays which originally need to consume memory resources can be reduced to one target array, specifically, because 36 OSD examples are included in one OSD module process under a common condition, the target array which originally needs to consume memory resources is reduced to 1 target array. In conclusion, the memory resource consumption of the distributed file system is effectively reduced.
Referring to fig. 2, an embodiment of the present application discloses a specific distributed file system memory optimization method, including:
Step S21: determining the array related to the OSDMap in each OSD instance from all OSD instances of the OSD module process to obtain a map _ cache array, a map _ bl _ cache array and a map _ bl _ inc _ cache array.
Step S22: and eliminating the map _ cache array, the map _ bl _ cache array and the map _ bl _ inc _ cache array in all OSD instances of the OSD module process.
Step S23: adding a map _ cache array, a map _ bl _ cache array and a map _ bl _ inc _ cache array in an OSDManager of the OSD module process, and controlling all OSD instances in the OSD module process to share the map _ cache array, the map _ bl _ cache array and the map _ bl _ inc _ cache array in the OSDManager.
Step S24: and packaging operation functions of preset operations aiming at the map _ cache array, the map _ bl _ cache array and the map _ bl _ inc _ cache array into the OSDManager so that any OSD instance can call the operation functions in the OSDManager to complete the preset operations aiming at the map _ cache array, the map _ bl _ cache array and the map _ bl _ inc _ cache array.
The preset operation comprises any one or any combination of an insertion operation, a search operation and a deletion operation.
Referring to fig. 3, an embodiment of the present application discloses a distributed file system memory optimization apparatus, including:
the data determining module 11 is configured to determine, from all OSD instances of the OSD module process, an array related to OSDMap existing in each OSD instance to obtain a target array;
an array eliminating module 12, configured to eliminate the target array in all OSD instances of the OSD module process;
and an array adding module 13, configured to add the target array to the OSDManager of the OSD module process, and control all OSD instances in the OSD module process to share the target array in the OSDManager.
It should be noted that the OSD module process of the existing distributed file system includes 1 OSDManager and 36 OSD instances, and the 36 OSD instances are managed by the OSDManager.
In this embodiment, the target array includes any one or a combination of any several of a map _ cache array, a map _ bl _ cache array, and a map _ bl _ inc _ cache array.
In the prior art, each OSD instance includes one target array, and in the embodiment of the present application, the target array in each OSD instance is eliminated, that is, the modified OSD instance does not include the target array inside any more.
In the prior art, the OSDManager does not include the target array therein, and in the embodiment of the present application, after the target array is determined, the target array is added to the OSDManager, so that the changed OSDManager includes one target array, and subsequently, each OSD instance modified in the embodiment can share the target array in the changed OSDManager, thereby reducing the memory resource consumption.
Further, the distributed file system memory optimization apparatus in the embodiment of the present application may further include:
and the function packaging module is used for packaging an operation function of preset operation aiming at the target array into the OSDManager so that any OSD instance can finish the preset operation aiming at the target array by calling the operation function in the OSDManager.
The preset operation comprises any one or any combination of an insertion operation, a searching operation and a deleting operation.
As can be seen, in the embodiment of the present application, a target array related to OSDMap of each OSD instance in the OSD module process of the distributed file system is determined, then the target array in each OSD instance is removed, the target array is added to the OSDManager of the OSD module process, and then all OSD instances are controlled to share the target array added to the OSDManager. Compared with the prior art that a target array which needs to consume memory resources and is related to the OSDMap is independently arranged in each OSD example, the target array in each OSD example is removed, and only one target array which can be shared by all OSD examples is arranged in the OSDManager, so that a plurality of target arrays which originally need to consume memory resources can be reduced to one target array, specifically, because 36 OSD examples are included in one OSD module process under a common condition, the target array which originally needs to consume memory resources is reduced to 1 target array. In conclusion, the memory resource consumption of the distributed file system is effectively reduced.
Further, the present application also discloses a distributed file system memory optimization device, which includes:
a memory for storing a computer program;
and the processor is used for executing the computer program to realize the distributed file system memory optimization method disclosed by the embodiment.
Further, the present application also discloses a computer readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the distributed file system memory optimization method disclosed in the foregoing embodiment.
For the specific steps of the method, reference may be made to corresponding contents disclosed in the foregoing embodiments, which are not described herein again.
In the present specification, the embodiments are described in a progressive manner, and each embodiment focuses on differences from other embodiments, and the same or similar parts between the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The method, the device, the equipment and the medium for optimizing the memory of the distributed file system provided by the application are introduced in detail, a specific example is applied in the description to explain the principle and the implementation mode of the application, and the description of the embodiment is only used for helping to understand the method and the core idea of the application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.
Claims (6)
1. A distributed file system memory optimization method is characterized by comprising the following steps:
determining an array related to OSDMap in each OSD instance from all OSD instances of the OSD module process to obtain a target array;
removing the target array in all OSD examples of the OSD module process;
adding the target array in an OSDManager of the OSD module process, and controlling all OSD instances in the OSD module process to share the target array in the OSDManager;
the target array comprises any one or combination of any several of a map _ cache array, a map _ bl _ cache array and a map _ bl _ inc _ cache array;
And, the method further comprises:
and packaging an operation function of a preset operation aiming at the target array into the OSDManager so that any OSD instance can finish the preset operation aiming at the target array by calling the operation function in the OSDManager.
2. The distributed file system memory optimization method of claim 1,
the preset operation comprises any one or any combination of an insertion operation, a search operation and a deletion operation.
3. A distributed file system memory optimization apparatus, comprising:
the data determining module is used for determining an array which is related to the OSDMap and exists in each OSD instance from all OSD instances of the OSD module process to obtain a target array;
the array eliminating module is used for eliminating the target array in all OSD examples of the OSD module process;
the array adding module is used for adding the target array in an OSDManager of the OSD module process and controlling all OSD instances in the OSD module process to share the target array in the OSDManager;
the target array comprises any one or combination of any several of a map _ cache array, a map _ bl _ cache array and a map _ bl _ inc _ cache array;
And, the apparatus further comprises:
and the function packaging module is used for packaging an operation function of preset operation aiming at the target array into the OSDManager so that any OSD instance can finish the preset operation aiming at the target array by calling the operation function in the OSDManager.
4. The distributed file system memory optimization device of claim 3,
the preset operation comprises any one or any combination of an insertion operation, a search operation and a deletion operation.
5. A distributed file system memory optimization device, comprising:
a memory for storing a computer program;
a processor for executing said computer program to implement the distributed file system memory optimization method of claim 1 or 2.
6. A computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the distributed file system memory optimization method of claim 1 or 2.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811504792.0A CN109597689B (en) | 2018-12-10 | 2018-12-10 | Distributed file system memory optimization method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811504792.0A CN109597689B (en) | 2018-12-10 | 2018-12-10 | Distributed file system memory optimization method, device, equipment and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109597689A CN109597689A (en) | 2019-04-09 |
| CN109597689B true CN109597689B (en) | 2022-06-10 |
Family
ID=65962321
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811504792.0A Active CN109597689B (en) | 2018-12-10 | 2018-12-10 | Distributed file system memory optimization method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109597689B (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1737752A (en) * | 2004-08-18 | 2006-02-22 | 华为技术有限公司 | Managing method for shared data |
| CN102736945A (en) * | 2011-03-31 | 2012-10-17 | 国际商业机器公司 | A method and a system of running a plurality of instances of an application program |
| CN102810070A (en) * | 2012-07-16 | 2012-12-05 | 福建富士通信息软件有限公司 | High-performance professional ability packaging process engine and process control method thereof |
| CN103297456A (en) * | 2012-02-24 | 2013-09-11 | 阿里巴巴集团控股有限公司 | Method for accessing sharing resources in distributed system and distributed system |
| CN104364759A (en) * | 2012-04-19 | 2015-02-18 | 加泰罗尼亚理工大学 | Method, system and an executable piece of code for controlling the use of hardware resources of a computer system |
| CN106951445A (en) * | 2017-02-17 | 2017-07-14 | 深圳市中博睿存信息技术有限公司 | A kind of distributed file system and its memory node loading method |
| CN107547252A (en) * | 2017-06-29 | 2018-01-05 | 新华三技术有限公司 | A kind of network failure processing method and device |
| CN107622199A (en) * | 2017-09-21 | 2018-01-23 | 中国科学院信息工程研究所 | Flush Reload cache side-channel attack defence method and device in a kind of cloud environment |
| WO2018077292A1 (en) * | 2016-10-28 | 2018-05-03 | 北京市商汤科技开发有限公司 | Data processing method and system, electronic device |
| CN108509157A (en) * | 2018-04-13 | 2018-09-07 | 郑州云海信息技术有限公司 | A kind of data balancing method and device applied to distributed file system |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7526617B2 (en) * | 2005-12-29 | 2009-04-28 | Sap Ag | System and method for memory management using memory windows |
| US20140040566A1 (en) * | 2012-08-01 | 2014-02-06 | Boris Burshteyn | Method and system for accessing c++ objects in shared memory |
-
2018
- 2018-12-10 CN CN201811504792.0A patent/CN109597689B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1737752A (en) * | 2004-08-18 | 2006-02-22 | 华为技术有限公司 | Managing method for shared data |
| CN102736945A (en) * | 2011-03-31 | 2012-10-17 | 国际商业机器公司 | A method and a system of running a plurality of instances of an application program |
| CN103297456A (en) * | 2012-02-24 | 2013-09-11 | 阿里巴巴集团控股有限公司 | Method for accessing sharing resources in distributed system and distributed system |
| CN104364759A (en) * | 2012-04-19 | 2015-02-18 | 加泰罗尼亚理工大学 | Method, system and an executable piece of code for controlling the use of hardware resources of a computer system |
| CN102810070A (en) * | 2012-07-16 | 2012-12-05 | 福建富士通信息软件有限公司 | High-performance professional ability packaging process engine and process control method thereof |
| WO2018077292A1 (en) * | 2016-10-28 | 2018-05-03 | 北京市商汤科技开发有限公司 | Data processing method and system, electronic device |
| CN106951445A (en) * | 2017-02-17 | 2017-07-14 | 深圳市中博睿存信息技术有限公司 | A kind of distributed file system and its memory node loading method |
| CN107547252A (en) * | 2017-06-29 | 2018-01-05 | 新华三技术有限公司 | A kind of network failure processing method and device |
| CN107622199A (en) * | 2017-09-21 | 2018-01-23 | 中国科学院信息工程研究所 | Flush Reload cache side-channel attack defence method and device in a kind of cloud environment |
| CN108509157A (en) * | 2018-04-13 | 2018-09-07 | 郑州云海信息技术有限公司 | A kind of data balancing method and device applied to distributed file system |
Non-Patent Citations (4)
| Title |
|---|
| DWC2: A dynamic weight-based cooperative caching scheme for object-based storage cluster;Qingsong Wei 等;《2008 IEEE International Conference on Cluster Computing》;20081031;第167-174页 * |
| Implementing and Evaluating Security Controls for an Object-Based Storage System;Zhongying Niu 等;《24th IEEE Conference on Mass Storage Systems and Technologies (MSST 2007)》;20071029;第1-13页 * |
| Visual C++中利用内存映射文件在进程之间共享数据;樊华;《软件导刊》;20051020(第20期);全文 * |
| 基于对象存储系统中安全机制的研究;周功业 等;《小型微型计算机系统》;20061130;第27卷(第11期);第2133-2137页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109597689A (en) | 2019-04-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107832100B (en) | APK plug-in loading method and terminal thereof | |
| EP3432157B1 (en) | Data table joining mode processing method and apparatus | |
| CN106682215B (en) | Data processing method and management node | |
| US9977598B2 (en) | Electronic device and a method for managing memory space thereof | |
| WO2016191610A1 (en) | Method and apparatus for real-time data migration | |
| CN107153643B (en) | Data table connection method and device | |
| CN110297810B (en) | Stream data processing method and device and electronic equipment | |
| US20240061712A1 (en) | Method, apparatus, and system for creating training task on ai training platform, and medium | |
| US20200252314A1 (en) | Method and apparatus for managing network connection, and storage medium | |
| CN110399101B (en) | Write operation processing method, device and system of disk and storage medium | |
| CN109213450B (en) | Associated metadata deleting method, device and equipment based on flash memory array | |
| CN104407990A (en) | Disk access method and device | |
| CN111338803B (en) | Thread processing method and device | |
| CN104021022A (en) | Partition adjusting method and electronic equipment | |
| US20230035761A1 (en) | Method, apparatus, device, and medium for creating thick-provisioned volume | |
| CN109597689B (en) | Distributed file system memory optimization method, device, equipment and medium | |
| US10241927B2 (en) | Linked-list-based method and device for application caching management | |
| CN110222046B (en) | List data processing method, device, server and storage medium | |
| US20180203875A1 (en) | Method for extending and shrinking volume for distributed file system based on torus network and apparatus using the same | |
| CN105204903A (en) | Process module loading interception method and device | |
| US9678860B2 (en) | Updating data fields of buffers | |
| CN111625500B (en) | File snapshot method and device, electronic equipment and storage medium | |
| US11681601B2 (en) | Application program management method and apparatus, and storage medium | |
| CA3142143A1 (en) | Method and apparatus for correlating data tables based on kv database | |
| US20150379548A1 (en) | Method and System for Data Processing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |