CN116755611A - Method and device for promoting reading and writing of multiple types of data based on ceph storage - Google Patents
Method and device for promoting reading and writing of multiple types of data based on ceph storage Download PDFInfo
- Publication number
- CN116755611A CN116755611A CN202310546938.2A CN202310546938A CN116755611A CN 116755611 A CN116755611 A CN 116755611A CN 202310546938 A CN202310546938 A CN 202310546938A CN 116755611 A CN116755611 A CN 116755611A
- Authority
- CN
- China
- Prior art keywords
- data
- ceph
- types
- writing
- disk
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 76
- 230000001737 promoting effect Effects 0.000 title claims description 16
- 238000007635 classification algorithm Methods 0.000 claims abstract description 9
- 230000003068 static effect Effects 0.000 claims description 17
- 238000004364 calculation method Methods 0.000 claims description 13
- 238000013507 mapping Methods 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 10
- 238000004422 calculation algorithm Methods 0.000 claims description 8
- 238000004519 manufacturing process Methods 0.000 claims description 5
- 230000002688 persistence Effects 0.000 claims description 5
- 238000012706 support-vector machine Methods 0.000 claims description 4
- 239000013598 vector Substances 0.000 claims description 4
- 230000001133 acceleration Effects 0.000 claims description 3
- 230000006870 function Effects 0.000 description 16
- 238000010586 diagram Methods 0.000 description 10
- 238000004891 communication Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000002708 enhancing effect Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000000354 decomposition reaction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0644—Management of space entities, e.g. partitions, extents, pools
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0674—Disk device
- G06F3/0676—Magnetic disk device
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/068—Hybrid storage device
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a method and a device for improving reading and writing of various types of data based on ceph storage, wherein the method comprises the following steps: s1: various disk types are brought in according to the data structure type, ceph is correspondingly deployed, and various disks are taken over and uniformly divided into OSD disk characters; s2: the data is subjected to class classification by using a classification algorithm in the data uploading process, the data classification process is accelerated, and the data is subjected to label assignment so as to meet the requirement of subsequent pool class second classification; s3: dividing pool for storing various data in ceph, and configuring node disk writing rules; s4: and continuously resetting and reading and writing the stored ceph data to classify different data in a second level. The application aims to provide a ceph-based distributed storage system for classifying and storing write-in data, wherein different types of disks are divided according to different types of data for classifying and storing the data, and the high availability and the strong consistency of the data are ensured under the condition of maximally utilizing the performance of the disks.
Description
Technical Field
The application relates to the technical field of distributed storage, in particular to a method and a device for improving reading and writing of various types of data based on ceph storage.
Background
With the increase of the internet traffic, the rapid increase of the access amount and the metadata traffic, the processing intensity of each core part of the distributed system is relatively increased, the workload of the system is increased, the ceph is split into multiple objects in the process of uploading files to the ceph, the objects are randomly scattered and enter pgs, the pgs marks and calculates the objects by using a credit algorithm, pgid is produced, the random scattering only corresponds to osd after being random, and after entering osd, the pool classifies and records the file data to obtain complete file indexes and corresponding decomposition log lines for data addressing.
Then there is a large amount of static block data, dynamic block data, and hot file data during the discrete writing of large amounts of data. Under the condition that a large amount of data with different structure types are handled, how to ensure the maximized read-write performance of the data with different types becomes the defect of the distributed storage capacity of the ceph, and the ceph does not have the condition of carrying out isolation division on the data with different types, so that the overall comprehensive performance of the ceph can be improved to a greater extent by solving the problem.
Disclosure of Invention
Aiming at the technical problems in the background technology, the application provides a method and a device for improving reading and writing of various types of data based on ceph storage.
In a first aspect, the present application provides a method for promoting reading and writing of multiple types of data based on ceph storage, including the following steps:
s1: various disk types are brought in according to the data structure type, ceph is correspondingly deployed, and various disks are taken over and uniformly divided into OSD disk characters;
s2: in the data uploading process, judging the data attribute, and classifying the data according to the judging result; in the uploading process, the data is subjected to class classification by using a classification algorithm, the data classification process is accelerated, and the data is subjected to label assignment so as to meet the requirement of subsequent pool class second classification;
s3: dividing pool for storing various data in ceph, and configuring node disk writing rules;
s4: and continuously resetting and reading and writing the stored ceph data to classify different data in a second level.
By adopting the technical scheme, the application provides the ceph-based distributed storage system for classifying and storing the write-in data, and classifying and storing the data by dividing different types of magnetic disks according to different types of data, so that the high availability and the strong consistency of the data are ensured under the condition of maximally utilizing the performance of the magnetic disks.
Preferably, in the step S1, three types of disk types, namely HDD, SSD, NAS types of disk types, are included according to the data structure type, ceph is deployed correspondingly, and the three types of disk types are taken over and divided into OSD disk characters in a unified manner.
Preferably, in the step S1, multiple disks are distinguished on a single physical server, a system disk uses SSD for installing Linux operating system, the remaining disks are respectively inserted into HDD, SSD, NAS three types of different disks, the sizes of the disks are not distinguished under the same scale, the physical disks are not raid, and bare disks are used in a full-scale mode; the ceph distributed system deployment then takes over all physical server bare volume disks, labeled (osd.0) & gt (osd.n).
Preferably, in the step S2, the classifying the data according to the determination result specifically includes: the judgment result is that static block data is divided into mirror images, the judgment result is that dynamic block data is divided into virtual machines, and the judgment result is that hot static file data is divided into file data.
Preferably, in the step S2, the classifying the data using the classifying algorithm in the uploading process, accelerating the data classifying process, and designating the tag of the data specifically includes:
marking the data with accounting eigenvalue, distinguishing file data structure as [0,1,2,3] corresponding to different attribute types of the file, wherein the sample data structure can be described as (0, 0), (1, 0), (2, 0), (3, 0);
for all sample points to be reloaded from the original undivided space to a new separable special space, a mapping is defined: phi: x-phi (X); at this time, a kernel function calculation structure is set, and defined as follows:
K(X 1 ，X 2 )＝<φ(X 1 )，φ(X 2 )>；
the prediction of the support vector machine is determined by the inner product of the support vector, and the function mode is expressed as follows:
the kernel function is equivalent to computing the inner product of the feature space after mapping. The calculation can directly calculate the inner product in a low-dimensional space, and space mapping is carried out without display;
after the calculation is finished, the unified data result is classified by the function structure, the data is classified into class pools, the data is classified into ceph in an acceleration way, the ceph can directly hit the attribute of the ceph, and the data is merged into the pool at high speed.
Preferably, in the step S3, a rule-rule is created, all the disk buckets of the nodes are allocated, and disks with different structure types are distinguished for different rule rules; after the cluster point disk bucket is built, all disk bucket marks are arranged into rule rules, and corresponding rule codes are set for subsequent production; all rule rules are then stored, integrated into a cruhmap file, transcoded using a cruhtool command, and re-formulated into ceph.
Preferably, in the step S3, a pool copy structure is designed, and pg_num and pgs_num are set for all pool pools, and the NUM algorithm principle is pg_num= (Target PGs per OSD) x (0 sd#) x (% Data)/(Size), and the replied copy Data is designed to be dual copy to maintain Data reliability and high availability for reading and writing under the condition of maintaining h.a. (High Availability).
Preferably, in the step S3, three types of pool pools are divided in ceph, wherein each pool is a pool of volume, vms, files types, and the corresponding pool is marked as a pool for storing static block data, dynamic block data, and hot static file data, and the following rule configuration is performed:
configuring rule to set HDD as rule0, writing all node HDD discs into rule0, and configuring 3 copies;
configuring rule rules to set SSD as rule1, writing all node SSD discs into rule1 rules, and configuring 3 copies;
configuring rule to set NAS as rule2, writing all node NAS discs into rule2, and configuring 3 copies.
In a second aspect, the present application further provides a device for promoting reading and writing of multiple types of data based on ceph storage, which is characterized in that: the device comprises:
the ceph deployment module is configured to incorporate various disk types according to the data structure types, correspondingly deploy the ceph, and take over various disks to be uniformly divided into OSD disk characters;
the data classifying module is configured to judge the data attribute in the data uploading process and classify the data according to the judging result; in the uploading process, the data is subjected to class classification by using a classification algorithm, the data classification process is accelerated, and the data is subjected to label assignment so as to meet the requirement of subsequent pool class second classification;
the disk write rule configuration module is configured to divide pool pools for storing various data in ceph and configure node disk write rules;
and the persistence resetting module is configured for carrying out persistence resetting on the stored ceph data and reading and writing so as to carry out second-level classification on different data.
In a third aspect, the application also proposes a computer-readable storage medium, on which a computer program is stored, which program, when being executed by a processor, implements the method according to the first aspect.
The application relates to a method and a device for improving reading and writing of various types of data based on ceph storage, wherein the method comprises the following steps: s1: various disk types are brought in according to the data structure type, ceph is correspondingly deployed, and various disks are taken over and uniformly divided into OSD disk characters; s2: in the data uploading process, judging the data attribute, and classifying the data according to the judging result; in the uploading process, the data is subjected to class classification by using a classification algorithm, the data classification process is accelerated, and the data is subjected to label assignment so as to meet the requirement of subsequent pool class second classification; s3: dividing pool for storing various data in ceph, and configuring node disk writing rules; s4: and continuously resetting and reading and writing the stored ceph data to classify different data in a second level. The application aims to provide a ceph-based distributed storage system for classifying and storing write-in data, wherein different types of disks are divided according to different types of data for classifying and storing the data, and the high availability and the strong consistency of the data are ensured under the condition of maximally utilizing the performance of the disks.
Drawings
The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and together with the description serve to explain the principles of the application. Many of the intended advantages of other embodiments and embodiments will be readily appreciated as they become better understood by reference to the following detailed description. The elements of the drawings are not necessarily to scale relative to each other. Like reference numerals designate corresponding similar parts.
FIG. 1 is a flow chart of a method for promoting multiple types of data read-write based on ceph storage according to the present application.
FIG. 2 is a schematic diagram of a method for enhancing reading and writing of multiple types of data based on ceph storage according to an embodiment of the present application.
FIG. 3 is a schematic diagram of one embodiment of a ceph-based storage promotion multi-type data read-write method that may be used with the present application.
FIG. 4 is a diagram showing the IOPS index rise after applying the ceph-based memory promotion method of the present application.
FIG. 5 is a schematic block diagram of an apparatus for enhancing reading and writing of multiple types of data based on ceph storage according to an embodiment of the present application.
Fig. 6 is a schematic diagram of a computer system suitable for use in implementing an embodiment of the application.
Detailed Description
The application is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the application and are not limiting of the application. It should be noted that, for convenience of description, only the portions related to the present application are shown in the drawings.
It should be noted that, without conflict, the embodiments of the present application and features of the embodiments may be combined with each other. The application will be described in detail below with reference to the drawings in connection with embodiments.
FIG. 1 is a flow chart showing a method for promoting reading and writing of multiple types of data based on ceph storage according to the present application, and referring to FIG. 1, the method specifically includes the following steps:
s1: various disk types are brought in according to the data structure type, ceph is correspondingly deployed, and various disks are taken over and uniformly divided into OSD disk characters;
in the S1, three types of disk types are included according to the data structure type, namely HDD, SSD, NAS types of disk types are respectively allocated, ceph is allocated correspondingly, and the three types of disk types are taken over and uniformly divided into OSD disk characters;
in the S1, distinguishing a plurality of disks on a single physical server, using SSD for installing a Linux operating system by a system disk, respectively inserting HDD, SSD, NAS three types of different disks into the rest disks, not distinguishing the sizes of the disks under the same scale, not making raid by the physical disks, and using bare disks in a full-scale mode; the ceph distributed system deployment then takes over all physical server bare volume disks, labeled (osd.0) & gt (osd.n).
S2: in the data uploading process, judging the data attribute, and classifying the data according to the judging result; in the uploading process, the data is subjected to class classification by using a classification algorithm, the data classification process is accelerated, and the data is subjected to label assignment so as to meet the requirement of subsequent pool class second classification;
in the step S2, the classifying the data according to the determination result specifically includes: the judgment result is that static block data is divided into mirror images, the judgment result is that dynamic block data is divided into virtual machines, and the judgment result is that hot static file data is divided into file data.
In the step S2, the data classifying process is accelerated by using a classifying algorithm in the uploading process, and the label specification of the data specifically includes:
marking the data with accounting eigenvalue, distinguishing file data structure as [0,1,2,3] corresponding to different attribute types of the file, wherein the sample data structure can be described as (0, 0), (1, 0), (2, 0), (3, 0);
for all sample points to be reloaded from the original undivided space to a new separable special space, a mapping is defined: phi: x-phi (X); at this time, a kernel function calculation structure is set, and defined as follows:
K(X 1 ，X 2 )＝<φ(x 1 )，φ(X 2 )>；
the prediction of the support vector machine is determined by the inner product of the support vector, and the function mode is expressed as follows:
the kernel function is equivalent to computing the inner product of the feature space after mapping. The calculation can directly calculate the inner product in a low-dimensional space, and space mapping is carried out without display;
after the calculation is finished, the unified data result is classified by the function structure, the data is classified into class pools, the data is classified into ceph in an acceleration way, the ceph can directly hit the attribute of the ceph, and the data is merged into the pool at high speed.
S3: dividing pool for storing various data in ceph, and configuring node disk writing rules;
in the S3, creating a rule-rule, arranging all node disk barrels, and distinguishing disks of different structure types for different rule rules; after the cluster point disk bucket is built, all disk bucket marks are arranged into rule rules, and corresponding rule codes are set for subsequent production; all rule rules are then stored, integrated into a cruhmap file, transcoded using a cruhtool command, and re-formulated into ceph.
In the step S3, a pool copy structure is designed, and pg_num and pgs_num are set for all pool pools, and the rule of the NUM algorithm is pg_num= (Target PGs per OSD) x (0 sd#) x (% Data)/(size), and the replied copy Data is designed to be double-copy to keep Data reliability and high availability for reading and writing under the condition of keeping h.a. (High Availability).
In the step S3, three types of pool pools with structures are divided into three types of pool pools volume, vms, files in ceph, the corresponding pool pools are marked as static block data, dynamic block data and thermal static file data, and the following rule configuration is performed:
configuring rule to set HDD as rule0, writing all node HDD discs into rule0, and configuring 3 copies:
configuring rule rules to set SSD as rule1, writing all node SSD discs into rule1 rules, and configuring 3 copies;
configuring rule to set NAS as rule2, writing all node NAS discs into rule2, and configuring 3 copies.
S4: and continuously resetting and reading and writing the stored ceph data to classify different data in a second level.
In a specific embodiment, a method for promoting reading and writing of multiple types of data based on ceph storage according to the present application will be described in detail below:
the application aims to provide a ceph-based distributed storage system for classifying and storing write-in data, wherein different types of disks are divided according to different types of data for classifying and storing the data, and the high availability and the strong consistency of the data are ensured under the condition of maximally utilizing the performance of the disks.
In this embodiment, to achieve the above technical purpose, the ceph storage-based method for accelerating reading and writing of multiple types of data according to the present application includes the following steps:
step 1: three types of magnetic disk types are included according to the data structure type, namely HDD, SSD, NAS types of magnetic disk types; corresponding to deployed ceph, three types of disks are taken over and uniformly divided into OSD characters
Step 2: in the data uploading process, judging the data attribute, wherein the judging result is that static block data are divided into mirror images, dynamic block data are divided into virtual machines, and hot static file data are divided into file data;
step 3: in the uploading process, the data is subjected to class classification by using a classification algorithm, the data classification process is accelerated, and the data is subjected to label assignment so as to meet the second classification of the subsequent pool;
step 4: dividing three structural pool pools in ceph into volume, vms, files types of pool pools respectively, wherein the corresponding pool pools are marked as static block data, dynamic block data and thermal static file data;
step 5: configuring rule to set HDD as rule0, writing all node HDD discs into rule0, and configuring 3 copies;
step 6: configuring rule rules to set SSD as rule1, writing all node SSD discs into rule1 rules, and configuring 3 copies;
step 7: configuring rule configuration NAS as rule2, writing all node NAS discs into rule2, and configuring 3 copies;
step 8: and (3) continuously resetting the stored ceph data and reading and writing based on the steps 2-7 so as to achieve the purposes of classifying different types of different data and effectively carrying out copy hot standby in real time.
In a further embodiment, multiple disks are distinguished on a single physical server, the system disk uses SSD for installing a Linux operating system, the remaining disks are respectively inserted into HDD, SSD, NAS three different types of disks, the sizes of the disks are not distinguished under the same scale, the physical disks are not raid, and the bare disks are used in a full-scale mode.
In a further embodiment, the ceph distributed system deploys taking over all physical server bare volume disks, labeled (osd.0) & gt (osd.n). At this time, the disk structure can be seen using the ceph command, and the ceph-osd specific tag and the distributed node name already belonging to the up state.
In a further embodiment, create a rule-rule, assign all node disk buckets, distinguish between disks of different structure types for different rule, and target for example:
in a further embodiment, after the cluster point disk bucket is built, all disk bucket marks are built into rule rules, and corresponding rule codes are set for subsequent production:
in a further embodiment, all rule rules are stored, integrated into a cruhmap file, transcoded using a cruhtool command, and re-formulated into ceph use commands as follows:
crushtool-c crushmap.dump-o newcrushmap；
ceph osd setcrushmap-i newcrushmap；
in a further embodiment, a pool copy structure is designed, and pg_num are set for all pool pools, and the NUM algorithm principle is pg_num= (Target PGs per OSD) x (osd#) x (%data)/(Size), and the replied copy Data is designed to be double-copy Data reliability and high availability in reading and writing under the condition of maintaining the h.a. (High Availability);
in a further embodiment, the post-production pool corresponds to a detail structure, and the volumes correspond to HDD disk drive-drive: 0; vms corresponds to SSD disk drive-rule 1; files correspond to NAS disk drive-rule: 2; the details are as follows:
pool 1'files'replicated size 2 min_size 2 crush_rule 2 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode offlast_change 5916 lfor 0/630/1297 flags hashpspool，selfmanaged_snaps stripe width 0 application rbd；
pool 2'volumes'replicated size 2 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode offlast_change 2093 lfor 0/1275/1307 flags hashpspool，selfmanaged_snaps stripe width 0 application rbd；
pool 3'vms'replicated size 2 min_size 2 crush_rule 1 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode off last_change 5913。
in a further embodiment, the data is marked with accounting eigenvalues, and the file data structure is distinguished as [0,1,2,3] corresponding to different attribute types of the file, and at this time, the sample data structure can be described as (0, 0), (1, 0), (2, 0), (3, 0);
in a further embodiment, for all sample points to be swapped from the original undivided space to a new separable special space, a mapping is defined here: phi: x-phi (X); at this time, a kernel function calculation structure is set, and defined as follows:
K(X 1 ，X 2 )＝<φ(X1)，φ(X 2 )>；
in a further embodiment, the prediction of the support vector machine is determined by the inner product of the support vectors, and the function is expressed as:
the kernel function is equivalent to computing the inner product of the feature space after mapping. The calculation can directly calculate the inner product in a low-dimensional space, the space mapping is carried out without display, and the calculation rate can be more efficient;
in a further embodiment, after the calculation is finished, the unified data result is classified by the function structure to classify the data into class pools, so that the classification of the data to the ceph is accelerated, the ceph can hit the attribute of the ceph directly, and the data can be merged into the pool at high speed;
in a further embodiment, different data writing is tested, various files can be read and written continuously, high-performance bottleneck breaking is highlighted instantaneously, and ceph-H.A. capability is improved integrally.
With further reference to fig. 5, as an implementation of the method described above, the present application provides an embodiment of an apparatus for promoting reading and writing of multiple types of data based on ceph storage, where an embodiment of the apparatus corresponds to the embodiment of the method shown in fig. 1, and the apparatus may be specifically applied to various electronic devices.
Referring to fig. 5, an apparatus for promoting reading and writing of multiple types of data based on ceph storage includes:
the ceph deployment module 101 is configured to incorporate various disk types according to the data structure types, deploy ceph correspondingly, and take over various disks to divide uniformly into OSD disk characters;
the data classifying module 102 is configured to judge the data attribute in the data uploading process, and classify the data according to the judging result; in the uploading process, the data is subjected to class classification by using a classification algorithm, the data classification process is accelerated, and the data is subjected to label assignment so as to meet the requirement of subsequent pool class second classification;
the disk write rule configuration module 103 is configured to divide pool pools for storing various data in ceph and configure node disk write rules;
the persistence-homing module 104 is configured to perform persistence homing and read-write on the stored ceph data, so as to classify different data in seconds.
Referring now to FIG. 6, a schematic diagram of a computer system 200 suitable for use in implementing an electronic device of an embodiment of the present application is shown. The electronic device shown in fig. 6 is only an example and should not be construed as limiting the functionality and scope of use of the embodiments of the application.
As shown in fig. 6, the computer system 200 includes a Central Processing Unit (CPU) 201, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 202 or a program loaded from a storage section 208 into a Random Access Memory (RAM) 203. In the RAM 203, various programs and data required for the operation of the system 200 are also stored. The CPU 201, ROM 202, and RAM 203 are connected to each other through a bus 204. An input/output (I/0) interface 205 is also connected to bus 204.
The following components are connected to the I/0 interface 205: an input section 206 including a keyboard, a mouse, and the like; an output section 207 including a Liquid Crystal Display (LCD) or the like, a speaker or the like; a storage section 208 including a hard disk or the like; and a communication section 209 including a network interface card such as a LAN card, a modem, and the like. The communication section 209 performs communication processing via a network such as the internet. The driver 220 is also connected to the I/0 interface 205 as needed. A removable medium 211 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is installed as needed on the drive 220, so that a computer program read therefrom is installed as needed into the storage section 208.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 209, and/or installed from the removable medium 211. The above-described functions defined in the method of the present application are performed when the computer program is executed by a Central Processing Unit (CPU) 201.
As another aspect, the present application also provides a computer-readable storage medium that may be contained in the electronic device described in the above embodiment; or may exist alone without being incorporated into the electronic device. The computer-readable storage medium carries one or more programs which, when executed by the electronic device, cause the electronic device to perform the method as shown in fig. 1.
The computer readable storage medium according to the present application may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present application, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable storage medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations of the present application may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of remote computers, the remote computer may be connected to the user computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (e.g., connected through the internet using an internet service provider).
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
While the application has been described with reference to specific embodiments, the scope of the application is not limited thereto, and any changes or substitutions can be easily made by those skilled in the art within the scope of the application disclosed herein, and are intended to be covered by the scope of the application. Therefore, the protection scope of the application is subject to the protection scope of the claims.
In the description of the present application, it should be understood that the directions or positional relationships indicated by the terms "upper", "lower", "inner", "outer", etc. are based on the directions or positional relationships shown in the drawings, are merely for convenience of description and simplification of the description, and do not indicate or imply that the apparatus or element in question must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as limiting the present application. The word 'comprising' does not exclude the presence of elements or steps not listed in a claim. The word 'a' or 'an' preceding an element does not exclude the presence of a plurality of such elements. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims shall not be construed as limiting the scope.
Claims (10)
1. A method for improving reading and writing of various types of data based on ceph storage is characterized by comprising the following steps: the method comprises the following steps:
s1: various disk types are brought in according to the data structure type, ceph is correspondingly deployed, and various disks are taken over and uniformly divided into OSD disk characters;
s2: in the data uploading process, judging the data attribute, and classifying the data according to the judging result; in the uploading process, the data is subjected to class classification by using a classification algorithm, the data classification process is accelerated, and the data is subjected to label assignment so as to meet the requirement of subsequent pool class second classification;
s3: dividing pool for storing various data in ceph, and configuring node disk writing rules;
s4: and continuously resetting and reading and writing the stored ceph data to classify different data in a second level.
2. The method for promoting the reading and writing of multiple types of data based on ceph storage according to claim 1, wherein the method comprises the following steps: in the step S1, three types of disk types are included according to the data structure type, namely HDD, SSD, NAS types of disk types, ceph is correspondingly deployed, and the three types of disk types are taken over and uniformly divided into OSD disk characters.
3. The method for promoting the reading and writing of multiple types of data based on ceph storage according to claim 2, wherein the method comprises the following steps: in the S1, distinguishing a plurality of disks on a single physical server, using SSD for installing a Linux operating system by a system disk, respectively inserting HDD, SSD, NAS three types of different disks into the rest disks, not distinguishing the sizes of the disks under the same scale, not making raid by the physical disks, and using bare disks in a full-scale mode; the ceph distributed system deployment then takes over all physical server bare volume disks, labeled (osd.0) & gt (osd.n).
4. The method for promoting the reading and writing of multiple types of data based on ceph storage according to claim 1, wherein the method comprises the following steps: in the step S2, the classifying the data according to the determination result specifically includes: the judgment result is that static block data is divided into mirror images, the judgment result is that dynamic block data is divided into virtual machines, and the judgment result is that hot static file data is divided into file data.
5. The method for promoting the reading and writing of multiple types of data based on ceph storage according to claim 4, wherein the method comprises the following steps: in the step S2, the data classifying process is accelerated by using a classifying algorithm in the uploading process, and the label specification of the data specifically includes:
marking the data with accounting eigenvalue, distinguishing file data structure as [0,1,2,3] corresponding to different attribute types of the file, wherein the sample data structure can be described as (0, 0), (1, 0), (2, 0), (3, 0);
for all sample points to be reloaded from the original undivided space to a new separable special space, a mapping is defined: phi: x-phi (X); at this time, a kernel function calculation structure is set, and defined as follows:
K(X 1 ,X 2 )＝＜φ(X 1 )，φ(X 2 )＞；
the prediction of the support vector machine is determined by the inner product of the support vector, and the function mode is expressed as follows:
the kernel function is equivalent to the inner product calculated in the mapped feature space, and the inner product can be calculated in the low-dimensional space directly by the calculation, and the space mapping is carried out without displaying:
after the calculation is finished, the unified data result is classified by the function structure, the data is classified into class pools, the data is classified into ceph in an acceleration way, the ceph can directly hit the attribute of the ceph, and the data is merged into the pool at high speed.
6. The method for promoting the reading and writing of multiple types of data based on ceph storage according to claim 1, wherein the method comprises the following steps: in the S3, creating a rule-rule, arranging all node disk barrels, and distinguishing disks of different structure types for different rule rules; after the cluster point disk bucket is built, all disk bucket marks are arranged into rule rules, and corresponding rule codes are set for subsequent production; all rule rules are then stored, integrated into a cruhmap file, transcoded using a cruhtool command, and re-formulated into ceph.
7. The method for promoting the reading and writing of multiple types of data based on ceph storage according to claim 6, wherein the method comprises the following steps: in the step S3, a pool copy structure is designed, and pg_num and pgs_num are set for all pool pools, and the rule of the NUM algorithm is pg_num= (Target PGs per OSD) x (osd#) x (%data)/(Size), and the replied copy Data is designed to be double copy to keep Data reliability and high availability of reading and writing under the condition of keeping h.a. (High Availability).
8. The method for promoting the reading and writing of multiple types of data based on ceph storage according to claim 7, wherein the method comprises the following steps: in the step S3, three types of pool pools with structures are divided into three types of pool pools volume, vms, files in ceph, the corresponding pool pools are marked as static block data, dynamic block data and thermal static file data, and the following rule configuration is performed:
configuring rule to set HDD as rule0, writing all node HDD discs into rule0, and configuring 3 copies;
configuring rule rules to set SSD as rule1, writing all node SSD discs into rule1 rules, and configuring 3 copies;
configuring rule to set NAS as rule2, writing all node NAS discs into rule2, and configuring 3 copies.
9. The utility model provides a device based on ceph stores promotes multiple type data read-write which characterized in that: the device comprises:
the ceph deployment module is configured to incorporate various disk types according to the data structure types, correspondingly deploy the ceph, and take over various disks to be uniformly divided into OSD disk characters;
the data classifying module is configured to judge the data attribute in the data uploading process and classify the data according to the judging result; in the uploading process, the data is subjected to class classification by using a classification algorithm, the data classification process is accelerated, and the data is subjected to label assignment so as to meet the requirement of subsequent pool class second classification;
the disk write rule configuration module is configured to divide pool pools for storing various data in ceph and configure node disk write rules;
and the persistence resetting module is configured for carrying out persistence resetting on the stored ceph data and reading and writing so as to carry out second-level classification on different data.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310546938.2A CN116755611A (en) | 2023-05-16 | 2023-05-16 | Method and device for promoting reading and writing of multiple types of data based on ceph storage |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310546938.2A CN116755611A (en) | 2023-05-16 | 2023-05-16 | Method and device for promoting reading and writing of multiple types of data based on ceph storage |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116755611A true CN116755611A (en) | 2023-09-15 |
Family
ID=87957978
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310546938.2A Pending CN116755611A (en) | 2023-05-16 | 2023-05-16 | Method and device for promoting reading and writing of multiple types of data based on ceph storage |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116755611A (en) |
-
2023
- 2023-05-16 CN CN202310546938.2A patent/CN116755611A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107924323B (en) | Dependency-based container deployment | |
| CN102754077B (en) | The safety virtualization environment that can guide from external media device | |
| US11537652B2 (en) | Sharing filesystems between containers | |
| US8578370B2 (en) | Managing memory in multiple virtual machines | |
| US9971783B2 (en) | Data de-duplication for disk image files | |
| TWI736912B (en) | Computer program product, computer system and computer implement method of sorting and merging instruction for a general-purpose processor | |
| US9792075B1 (en) | Systems and methods for synthesizing virtual hard drives | |
| US9881351B2 (en) | Remote translation, aggregation and distribution of computer program resources in graphics processing unit emulation | |
| US9766954B2 (en) | Configuring resources used by a graphics processing unit | |
| CN109416621B (en) | Utilizing computer storage systems supporting shared objects to restore free space in non-volatile storage | |
| US20180365428A1 (en) | Configuration of a memory controller for copy-on-write with a resource controller | |
| US9128746B2 (en) | Asynchronous unmap of thinly provisioned storage for virtual machines | |
| TWI718563B (en) | Computer program product, computer system and computer-implemented method for saving and restoring machine state between multiple executions of an instruction | |
| US20180157605A1 (en) | Configuration of a memory controller for copy-on-write | |
| JP2009087282A (en) | Parallel computation system and parallel computation method | |
| US9779535B2 (en) | Configuring resources used by a graphics processing unit | |
| AU2020382999B2 (en) | Intelligent data pool | |
| US9535713B2 (en) | Manipulating rules for adding new devices | |
| US9952984B2 (en) | Erasing a storage block before writing partial data | |
| CN116755611A (en) | Method and device for promoting reading and writing of multiple types of data based on ceph storage | |
| CN108334453B (en) | File debugging method and device, terminal equipment and storage medium | |
| TWI806603B (en) | Computer-implemented method, computer program product and computer system for accessing topological mapping of cores | |
| US11789774B2 (en) | Optimization of workload scheduling in a distributed shared resource environment | |
| CN115080242A (en) | Method, device and medium for unified scheduling of PCI equipment resources | |
| KR20220018499A (en) | Non-volatile storage partition identifier |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |