CN111124291A - Data storage processing method and device of distributed storage system and electronic equipment - Google Patents
Data storage processing method and device of distributed storage system and electronic equipment Download PDFInfo
- Publication number
- CN111124291A CN111124291A CN201911252196.2A CN201911252196A CN111124291A CN 111124291 A CN111124291 A CN 111124291A CN 201911252196 A CN201911252196 A CN 201911252196A CN 111124291 A CN111124291 A CN 111124291A
- Authority
- CN
- China
- Prior art keywords
- data
- storage space
- storage
- data type
- threshold corresponding
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0608—Saving storage space on storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
- G06F3/0631—Configuration or reconfiguration of storage systems by allocating resources to storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0644—Management of space entities, e.g. partitions, extents, pools
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention discloses a data storage processing method for a distributed storage system, wherein the distributed storage system comprises at least one storage server, and the method is applied to the storage server and comprises the following steps: judging whether the used storage space of the storage server exceeds a storage space threshold corresponding to a first data type; stopping receiving the data of the first data type when the used storage space exceeds a storage space threshold corresponding to the first data type; judging whether the used storage space of the storage server exceeds a storage space threshold corresponding to a second data type; stopping receiving the data of the second data type when the used storage space exceeds the storage space corresponding to the second data type; converting at least a portion of the received data of the first data type to data of a second data type. The invention can reduce the complexity of the pre-allocation flow of the storage space during data storage, simplify the management of the free space and reduce abnormal branches.
Description
Technical Field
The present invention relates to the field of distributed storage systems, and in particular, to a data storage processing method and apparatus for a distributed storage system, an electronic device, and a computer-readable storage medium.
Background
When the distributed storage system receives the original data, the original data can be further converted, for example, the original data is converted into the data occupying less storage space by using a compression or erasure code technology, so that the storage efficiency of the system is improved. However, when data reception and conversion are performed, there may be insufficient space due to contention for limited remaining space resources, thereby causing a deadlock problem.
The prior art approach proposes a way to address the above problems in a distributed storage system by pre-allocating storage space quotas. Specifically, when the storage server needs to receive the original data or perform data conversion, the storage server applies for all or part of the required space to the metadata server, and receives the original data or performs data conversion in the allocated space after the metadata server successfully responds.
However, the prior art has the following defects: 1. the participation of a metadata server is needed, and the metadata server needs to dynamically redistribute and plan storage resources; 2. when the remaining space of the storage server where the multiple data copies are located is inconsistent, how to determine how much storage space to reapply for receiving the original data or performing data conversion faces certain difficulties. In summary, the related art approaches also fail to solve the problem of memory deadlock.
Disclosure of Invention
The invention aims to provide a data storage processing method and device for a distributed storage system, an electronic device and a computer-readable storage medium, so as to solve the problem of space deadlock on the premise of simplifying a space management process.
According to a first aspect of the present invention, there is provided a data storage processing method for a distributed storage system, the distributed storage system including at least one storage server, the method being applied to the storage server, the method including:
judging whether the used storage space of the storage server exceeds a storage space threshold corresponding to a first data type;
stopping receiving the data of the first data type when the used storage space exceeds a storage space threshold corresponding to the first data type;
judging whether the used storage space of the storage server exceeds a storage space threshold corresponding to a second data type;
stopping receiving the data of the second data type when the used storage space exceeds a storage space threshold corresponding to the second data type;
converting at least part of the received data of the first data type into data of a second data type;
the storage space threshold corresponding to the first data type is smaller than the storage space threshold corresponding to the second data type, the storage space threshold corresponding to the second data type is smaller than the total storage space of the storage server, and after the data of the first data type is converted into the data of the second data type, the occupied storage space is reduced.
Optionally, the method further includes:
and after converting at least part of the received data of the first data type into data of a second data type, sending the converted data of at least part of the second data type to other storage servers.
Optionally, the determining whether the used storage space of the storage server exceeds a storage space threshold corresponding to the first data type includes:
when a write-in request of data of a first data type is received, judging whether the used storage space of the storage server exceeds a storage space threshold corresponding to the first data type;
alternatively, the first and second electrodes may be,
and judging whether the used storage space of the storage server exceeds a storage space threshold corresponding to the first data type according to a preset first judgment period.
Optionally, the determining whether the used storage space of the storage server exceeds the storage space threshold corresponding to the second data type includes:
after stopping receiving the data of the first data type, when receiving a write request of the data of the second data type, judging whether the used storage space of the storage server exceeds a storage space threshold corresponding to the second data type;
alternatively, the first and second electrodes may be,
and after stopping receiving the data of the first data type, judging whether the used storage space of the storage server exceeds a storage space threshold corresponding to the second data type according to a preset second judgment period.
Optionally, the method further includes:
receiving a data modification request of a client;
and acquiring and storing a modified version data copy of the original data based on the pre-stored original data corresponding to the data modification request.
Optionally, after receiving a data modification request from a client, before acquiring and storing a modified version data copy of original data based on pre-stored original data corresponding to the data modification request, the method further includes:
determining whether an available storage space of the storage server is greater than a first storage threshold;
the obtaining and storing a modified version data copy of the original data based on pre-stored original data corresponding to the data modification request comprises:
if the available storage space of the storage server is larger than a first storage threshold value, acquiring and storing a modified version data copy of original data based on the pre-stored original data corresponding to the data modification request;
and, the method further comprises:
and deleting the original data after storing the modified version data copy of the original data.
Optionally, the method further includes:
determining whether an available storage space of the storage server is less than a second storage threshold;
and if the available storage space of the storage server is smaller than a second storage threshold value, locally executing the data recovery service to delete the original data corresponding to the modified version data copy and release the storage space.
According to a second aspect of the present invention, there is provided a data storage processing method for a distributed storage system, the distributed storage system including at least one storage server, the method being applied to the storage server, the method including:
judging whether the available storage space of the storage server is smaller than a storage space threshold corresponding to a first data type;
when the available storage space is smaller than a storage space threshold corresponding to the first data type, stopping receiving the data of the first data type;
judging whether the available storage space of the storage server is smaller than a storage space threshold corresponding to a second data type;
when the available storage space is smaller than a storage space threshold corresponding to the second data type, stopping receiving the data of the second data type;
converting at least part of the received data of the first data type into data of a second data type;
the storage space threshold corresponding to the first data type is smaller than the storage space threshold corresponding to the second data type, the storage space threshold corresponding to the first data type is smaller than the total storage space of the storage server, and after the data of the first data type is converted into the data of the second data type, the occupied storage space is reduced.
According to a third aspect of the present invention, there is provided a data storage processing apparatus for a distributed storage system including at least one storage server, the apparatus being applied to the storage server, the apparatus comprising:
the first judgment module is used for judging whether the used storage space of the storage server exceeds a storage space threshold corresponding to a first data type;
the first stopping module is used for stopping receiving the data of the first data type when the used storage space exceeds a storage space threshold corresponding to the first data type;
the second judgment module is used for judging whether the used storage space of the storage server exceeds a storage space threshold corresponding to a second data type;
the second stopping module is used for stopping receiving the data of the second data type when the used storage space exceeds a storage space threshold corresponding to the second data type;
the conversion module is used for converting at least part of the received data of the first data type into data of a second data type;
the storage space threshold corresponding to the first data type is smaller than the storage space threshold corresponding to the second data type, the storage space threshold corresponding to the second data type is smaller than the total storage space of the storage server, and after the data of the first data type is converted into the data of the second data type, the occupied storage space is reduced.
Optionally, the apparatus further comprises:
and the sending module is used for converting at least part of the received data of the first data type into data of a second data type and then sending the converted data of at least part of the second data type to other storage servers.
Optionally, the first determining module includes:
the first judging unit is used for judging whether the used storage space of the storage server exceeds a storage space threshold corresponding to a first data type or not when a writing request of data of the first data type is received;
alternatively, the first and second electrodes may be,
and the second judging unit is used for judging whether the used storage space of the storage server exceeds a storage space threshold corresponding to the first data type according to a preset first judging period.
Optionally, the second determining module includes:
a third judging unit, configured to, after stopping receiving the data of the first data type, judge whether an already-used storage space of the storage server exceeds a storage space threshold corresponding to the second data type when receiving a write request for the data of the second data type;
alternatively, the first and second electrodes may be,
and the fourth judging unit is used for judging whether the used storage space of the storage server exceeds the storage space threshold corresponding to the second data type according to a preset second judging period after the data of the first data type stops being received.
Optionally, the apparatus further comprises:
the receiving module is used for receiving a data modification request of a client;
and the storage module acquires and stores a modified version data copy of the original data based on the pre-stored original data corresponding to the data modification request.
Optionally, the apparatus further comprises:
a first determination module to determine whether an available storage space of the storage server is greater than a first storage threshold;
the memory module includes:
the storage unit is used for acquiring and storing a modified version data copy of original data based on the pre-stored original data corresponding to the data modification request if the available storage space of the storage server is larger than a first storage threshold;
and, the apparatus further comprises:
and the deleting module is used for deleting the original data after the modified version data copy of the original data is stored.
Optionally, the apparatus further comprises:
a second determination module to determine whether the available storage space of the storage server is less than a second storage threshold;
and the recovery module locally executes the data recovery service to delete the original data corresponding to the modified version data copy and release the storage space if the available storage space of the storage server is smaller than a second storage threshold.
According to a fourth aspect of the present invention, there is provided a data storage processing apparatus for a distributed storage system including at least one storage server, the apparatus being applied to the storage server, the apparatus comprising:
the first judging module is used for judging whether the available storage space of the storage server is smaller than a storage space threshold corresponding to a first data type;
the first stopping module is used for stopping receiving the data of the first data type when the available storage space is smaller than a storage space threshold corresponding to the first data type;
the second judging module is used for judging whether the available storage space of the storage server is smaller than a storage space threshold corresponding to a second data type;
the second stopping module is used for stopping receiving the data of the second data type when the available storage space is smaller than the storage space threshold corresponding to the second data type;
the conversion module is used for converting at least part of the received data of the first data type into data of a second data type;
the storage space threshold corresponding to the first data type is smaller than the storage space threshold corresponding to the second data type, the storage space threshold corresponding to the first data type is smaller than the total storage space of the storage server, and after the data of the first data type is converted into the data of the second data type, the occupied storage space is reduced.
According to a fifth aspect of the present invention, there is provided an electronic apparatus comprising:
a data storage processing apparatus for a distributed storage system according to the third or fourth aspect of the present invention; alternatively, the first and second electrodes may be,
a processor and a memory for storing executable instructions for controlling the processor to perform the data storage processing method for a distributed storage system according to the first or second aspect of the invention.
According to a sixth aspect of the present invention, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the data storage processing method for a distributed storage system according to the first or second aspect of the present invention.
According to the invention, different storage space thresholds are provided in the storage server aiming at different received data types, and corresponding data services are executed in different storage space threshold ranges. Therefore, the problem that different data services compete for limited residual space resources to cause deadlock of the storage space under the condition that the available storage space is insufficient can be avoided. And the data occupying larger storage space is converted into the data occupying smaller storage space to release the storage space, so that the storage server can adaptively control the available storage space or reduce the used storage space, the utilization rate of the storage space is improved, and the deadlock of the storage space is avoided.
By the invention, the storage server adaptively controls the available storage space or reduces the used storage space, the complexity of the pre-allocation flow of the storage space during data storage can be reduced, the free space management is simplified, the abnormal branches are reduced, the local end of the storage server allocates and manages the storage space, no additional storage space is required to be applied by the metadata server, the network communication times between different servers (for example, between the metadata server and the storage server) during data storage are reduced, and the efficiency of data storage and the timeliness of data storage are improved.
Other features of the present invention and advantages thereof will become apparent from the following detailed description of exemplary embodiments thereof, which proceeds with reference to the accompanying drawings.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description, serve to explain the principles of the invention.
Fig. 1 is a block diagram of a hardware configuration structure of a server that can be used to implement an embodiment of the present invention.
Fig. 2 is a flowchart of a data storage processing method for a distributed storage system according to an embodiment of the present invention.
Fig. 3 is a flowchart illustrating specific steps of a data storage processing method for a distributed storage system according to a first embodiment of the present invention.
Fig. 4 is a flowchart illustrating specific steps of a data storage processing method for a distributed storage system according to a second embodiment of the present invention.
Fig. 5 is a flowchart illustrating specific steps of a data storage processing method for a distributed storage system according to a third embodiment of the present invention.
Fig. 6 is a flowchart illustrating specific steps of a data storage processing method for a distributed storage system according to a fourth embodiment of the present invention.
Fig. 7 is a block diagram showing a specific structure of a data storage processing apparatus for a distributed storage system according to a first embodiment of the present invention.
Fig. 8 is a block diagram showing a specific structure of a data storage processing apparatus for a distributed storage system according to a second embodiment of the present invention.
Fig. 9 is a block diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that: the relative arrangement of the components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.
The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.
Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.
In all examples shown and discussed herein, any particular value should be construed as merely illustrative, and not limiting. Thus, other examples of the exemplary embodiments may have different values.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.
Fig. 1 is a block diagram showing a hardware configuration of a server 1000 that can implement an embodiment of the present invention. The server 1000 may be a unitary server or a distributed server across multiple computers or computer data centers. The server may be of various types, such as, but not limited to, a storage server, a cloud database server, a cloud computing server, a cloud management server, a web server, a news server, a mail server, a message server, an advertisement server, a file server, an application server, an interaction server, a storage server, a database server, a proxy server, or the like of a distributed system. In some embodiments, each server may include hardware, software, or embedded logic components or a combination of two or more such components for performing the appropriate functions supported or implemented by the server. For example, a server, such as a blade server, a cloud server, etc., or may be a server group consisting of a plurality of servers, which may include one or more of the above types of servers, etc.
As shown in fig. 1, the server 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, or may further include a display device 1500, an input device 1600, and the like. The processor 1100 may be a central processing unit CPU, a microprocessor MCU, or the like, and is configured to execute a computer program. The computer program may be written in an instruction set of an architecture such as x86, Arm, RISC, MIPS, SSE, etc. The memory 1200 includes, for example, a ROM (read only memory), a RAM (random access memory), a nonvolatile memory such as a hard disk, and the like. The interface device 1300 includes, for example, a USB interface, a headphone interface, and the like. The communication device 1400 is capable of wired communication using an optical fiber or a cable, or wireless communication, for example, and may specifically include WiFi communication, bluetooth communication, 2G/3G/4G/5G communication, and the like. The display device 1500 is, for example, a liquid crystal display panel, a touch panel, or the like. The input device 1600 may include, for example, a touch screen, a keyboard, a somatosensory input, and the like.
The servers shown in fig. 1 are merely illustrative and are in no way meant to limit the invention, its application, or uses. In an embodiment of the present invention, the server 1000 may be, for example, a storage server according to an embodiment of the present invention, and may include a memory 1200 and a processor 1100, where the memory 1200 is configured to store instructions for controlling the processor 1100 to operate to execute any one of the data storage processing methods for a distributed storage system according to an embodiment of the present invention. It should be understood by those skilled in the art that although a plurality of devices are shown for the server 1000 in fig. 1, the present invention may only relate to some of the devices, for example, the server 1000 may only relate to the processor 1100 and the memory 1200. The skilled person can design the instructions according to the disclosed solution. How the instructions control the operation of the processor is well known in the art and will not be described in detail herein.
In one embodiment of the invention, a data storage processing method for a distributed storage system is provided, wherein the distributed storage system comprises at least one storage server, and the method is applied to the storage server.
Referring to fig. 2, a flowchart of a data storage processing method for a distributed storage system according to an embodiment of the present invention is shown. The data storage processing method for the distributed storage system may be implemented by a server, which may be, for example, the server 1000 shown in fig. 1.
As shown in fig. 2, the data storage processing method for a distributed storage system according to an embodiment of the present invention includes the following steps:
and 102, judging whether the used storage space of the storage server exceeds a storage space threshold corresponding to the first data type.
In this step, the data of the first data type may be raw data, for example, data generated by a client. The storage server establishes a corresponding communication connection with the client through which data of the first data type is received from the client.
In an embodiment of the present invention, the storage server includes a storage space threshold corresponding to the first data type, and the storage space threshold may be less than a total storage space of the storage server. In this step, the storage server may compare the used storage space threshold with the storage space threshold corresponding to the first data type, so as to determine whether the used storage space exceeds the storage space threshold corresponding to the first data type. For example, if the storage space of the storage server is 100T, the used storage space is 60T, and the storage space threshold corresponding to the first data type is 50T, it may be determined that the used storage space has exceeded the storage space threshold corresponding to the first data type.
The storage space threshold corresponding to the first data type may be set in the storage server by a technical engineer according to experience, may be determined according to a historical storage space record of the data of the first data type, and may also be determined according to a historical application record/usage record of a user corresponding to the data of the first data type on the storage space for the data of the type. No further description is given here, and the determination method of the storage space threshold corresponding to the first data type is not limited in the present invention.
And 104, stopping receiving the data of the first data type when the used storage space exceeds the storage space threshold corresponding to the first data type, namely stopping receiving the original data.
In this step, when the used storage space exceeds the storage space threshold corresponding to the first data type, the storage server will no longer accept the data of the first data type or no longer provide the data service corresponding to the first data type. The method and the device avoid the problem that under the condition that the storage space of the storage server is limited, the data service process of the first data type is operated all the time to apply for data service, but the process cannot complete the service, and the rest storage space is occupied all the time by the process and cannot be provided for other processes, so that the storage space is deadlocked. And 106, judging whether the used storage space of the storage server exceeds a storage space threshold corresponding to the second data type.
In this step, the second data type may be conversion data, and the conversion data may be data transmitted from other storage servers.
In this embodiment of the present invention, the storage server includes a storage space threshold corresponding to the second data type. The storage space occupied by the data of the second data type is smaller than the storage space occupied by the data of the first data type, and therefore, the storage space threshold corresponding to the second data type is smaller than the storage space threshold of the first data type. The second data type may be, for example, compressed data, erasure code data, and the like, and is not limited herein.
The storage space threshold corresponding to the second data type may be set in the storage server by a technical engineer according to experience, may be determined according to a historical storage space record of the data of the second data type, and may also be determined according to a historical application record/usage record of a user corresponding to the data of the first data type on the storage space for the data of the type. No further description is given here, and the determination method of the storage space threshold corresponding to the second data type is not limited in the present invention.
And 108, stopping receiving the data of the second data type, namely stopping receiving the conversion data when the used storage space exceeds the storage space threshold corresponding to the second data type.
When the used storage space exceeds the corresponding storage space threshold of the second data type, the available storage space of the storage server may be already small, and if the data of the second data type continues to be received, the remaining available storage space of the storage server may be deadlocked, and local services such as data backup and data recovery cannot be performed in the storage server. In this step, when it is determined that the used storage space exceeds the storage space threshold corresponding to the second data type, the data of the second data type is not received any more.
It is understood that, in the distributed storage system, in order to ensure the reliability of the data, the storage server locally converts the data (original data) sent by the client into data of the second data type, i.e. converted data, and then sends the converted data to other storage servers. In this step, the storage server will stop receiving data of the second data type sent by other storage servers.
At step 110, at least a portion of the received data of the first data type is converted into data of the second data type.
In this step, the storage server may convert the data of the first data type into the data of the second data type according to a preset rule. For example, the data of the second data type may be divided into batches and then converted into data in batches, or the data of the second data type whose storage time exceeds a set time threshold may be converted into data in a time sequence in which the data of the second data type is received, which is not limited herein. After the data conversion is carried out, at least part of the storage space occupied by the data of the first data type can be released, so that the storage space of the storage server can be adaptively adjusted.
It should be noted that, the step and all the steps in the foregoing steps 102-108 may not have a restriction on the execution order, and the step may be executed as long as the storage server includes data of the first data type.
The storage space threshold corresponding to the first data type is smaller than the storage space threshold corresponding to the second data type, the storage space threshold corresponding to the second data type is smaller than the total storage space of the storage server, and after the data of the first data type is converted into the data of the second data type, the occupied storage space is reduced, namely the occupied storage space of the converted original data is reduced.
In one example, as shown in fig. 2, the method may further send the converted data (converted data) of at least a part of the second data type to another storage server after step 110 (step 112).
However, it should be noted that the conversion data in step 110 shown in fig. 2 may operate synchronously with the foregoing steps 102 to 108, and step 110 is not in chronological order with steps 102 to 108.
In step 102, determining whether the used storage space of the storage server exceeds a storage space threshold corresponding to a first data type includes:
when a write-in request of data of a first data type is received, judging whether the used storage space of the storage server exceeds a storage space threshold corresponding to the first data type; or judging whether the used storage space of the storage server exceeds a storage space threshold corresponding to the first data type according to a preset first judgment cycle.
In step 106, determining whether the used storage space of the storage server exceeds a storage space threshold corresponding to the second data type includes: after stopping receiving the data of the first data type, when receiving a write request of the data of the second data type, judging whether the used storage space of the storage server exceeds a storage space threshold corresponding to the second data type; or after stopping receiving the data of the first data type, judging whether the used storage space of the storage server exceeds the storage space threshold corresponding to the second data type according to a preset second judgment period.
The storage server in the distributed system executes various processes when receiving and storing data, and the data types include, for example, original data and converted data, wherein the storage space occupied by the data after the original data conversion processing is less than the original data space.
The data storage processing method of the present invention will be explained with reference to the embodiments of fig. 3 to fig. 4, where fig. 3 is a flowchart illustrating specific steps of the data storage processing method for a distributed storage system according to a first embodiment of the present invention, and fig. 4 is a flowchart illustrating specific steps of the data storage processing method for a distributed storage system according to a second embodiment of the present invention.
As shown in fig. 3, the data storage processing method for the distributed storage system includes:
In other words, it is determined whether the currently used storage space Ht of the storage server when the data reception service is executed exceeds the first storage space threshold H1 corresponding to the original data reception.
In step 206, when Ht does not exceed the first storage space threshold H1, the original data and the converted data continue to be received. And continuously returns to the step 204 to continue the threshold judgment in the receiving process.
In step 208, when Ht exceeds the first storage space threshold H1, the reception of the original data is stopped, but the reception of the converted data is continued.
Since the second storage space threshold H2 is greater than the first storage space threshold H1, during the process of receiving the converted data, it is necessary to further determine whether the currently used storage space Ht of the storage server exceeds the second storage space threshold H2 corresponding to the received converted data (step 210).
In step 212, when Ht does not exceed the second storage space threshold H2, the converted data continues to be received. And continuously returns to the step 210 to continue the threshold judgment during the execution of the receiving operation.
That is, when there is H1< Ht < H2, the storage server does not receive the original data any more, but can continue to receive the converted data.
In step 214, when Ht exceeds the threshold H2 for the second storage space, reception of the transition data is stopped.
In step 216, at least a portion of the original data received from the storage server may be transformed to obtain transformed data. Step 216 may be performed synchronously with the data reception and subsequent storage space threshold determination of step 202, without distinguishing the chronological order.
Referring now to fig. 4, the embodiment of fig. 4 differs from fig. 3 in that fig. 3 compares and judges the used storage space Ht with the corresponding storage space thresholds H1, H2, and fig. 4 compares and judges the available storage space Hs with the corresponding storage space thresholds H1, H2, wherein the sum of the used storage space Ht and the available storage space Hs is the total storage space HZ of the storage server.
As shown in fig. 4, the data storage processing method for a distributed storage system of the present embodiment includes:
In step 224, it is determined whether the available storage space of the storage server is smaller than a first storage space threshold, i.e. Hs < H1?
In other words, it is determined whether the currently available storage space Ht of the storage server when the data reception service is executed is smaller than the first storage space threshold H1 corresponding to the original data reception.
In step 226, when Hs is not less than the first storage space threshold H1, the original data and the converted data are continuously received. And continuously returns to the step 224 to continue the threshold judgment in the receiving process.
In step 228, when Hs is smaller than the first storage space threshold H1, the receiving of the original data is stopped, but the converted data is still received.
Since the second storage space threshold H2 is greater than the first storage space threshold H1, during the process of receiving the converted data, it needs to be further determined whether the currently available storage space Hs of the storage server is less than the second storage space threshold H2 corresponding to the received converted data (step 230).
In step 232, when Hs is not less than the second storage space threshold H2, the conversion data continues to be received. And continuously returns to step 230 to continue the threshold determination during the execution of the receiving operation.
That is, when there is H1> Hs > H2, then the storage server no longer receives the original data, but may continue to receive the converted data.
In step 234, when Ht is less than the second storage space threshold H2, the reception of the conversion data is stopped.
In step 236, at least a portion of the raw data received from the storage server may be converted to obtain converted data. Step 236 may be performed synchronously with the data reception and subsequent storage space threshold determination of step 222, without time sequencing.
The data storage processing method for the distributed storage system provides different storage space threshold values aiming at different data types in the storage server, and executes corresponding data services within different storage space threshold values. Therefore, the problem that different data services compete for limited residual space resources to cause deadlock of the storage space under the condition that the available storage space is insufficient can be avoided. And because the newly received original data can be continuously changed into converted data with smaller occupied space after conversion, the invention utilizes the characteristic that the space of the converted data occupies smaller space than the original data, and converts the data occupying larger storage space into the data occupying smaller storage space to release the storage space, so that the storage server can adaptively control the available storage space or reduce the used storage space, improve the utilization rate of the storage space and avoid the deadlock of the storage space.
Meanwhile, the storage server adaptively controls the available storage space or reduces the used storage space, can also reduce the complexity of the pre-allocation process of the storage space during data storage, simplifies the management of the free space, reduces abnormal branches without applying for an additional storage space through the metadata server, reduces the times of network communication between different servers (for example, between the metadata server and the storage server) during data storage, and improves the efficiency of data storage and the timeliness of data storage.
In one example, the data storage processing method for the distributed storage system of the present invention may further perform an operation of data reclamation to delete the original data and release the storage space occupied by the original data. Here, the original data may be received original data or converted data, and the data recovery may be performed when the client performs an access operation on data stored in the storage server. In the following, a description will be given in conjunction with the embodiments of fig. 5 and 6.
First, reference is made to the embodiment of fig. 5, which is a flowchart illustrating specific steps of a data storage processing method for a distributed storage system according to a third embodiment of the present invention.
As shown in fig. 5, the data storage processing method for a distributed storage system of this embodiment includes the steps of:
In this way, by deleting the original data, the data is recycled to release part of the storage space of the storage server.
Referring to fig. 6, fig. 6 is a flowchart illustrating specific steps of a data storage processing method for a distributed storage system according to a fourth embodiment of the present invention.
In the above embodiment of the present invention, when a data modification request from a client to a storage server is received, after the original data modification version data copy corresponding to the request is stored, the corresponding original data is deleted, and data recovery is performed, so that a space occupied by the original data is released.
The storage space is released through the recovery of the original data, the storage space is vacated to the maximum extent, and the deadlock of the storage space among different services in the recovery process is avoided. Therefore, the invention can automatically convert the data by the storage server without applying for an additional space process to the metadata server and ensure that the problem of space deadlock does not exist.
According to another embodiment of the present invention, there is also provided a data storage processing apparatus for a distributed storage system, wherein fig. 7 is a specific structural block diagram of the data storage processing apparatus for a distributed storage system according to the first embodiment of the present invention, and fig. 8 is a specific structural block diagram of the data storage processing apparatus for a distributed storage system according to the second embodiment of the present invention. The data storage processing device for the distributed storage system is applied to the storage server.
As shown in fig. 7, the data storage processing apparatus 2000 includes a first determining module 2200, a first stopping module 2400, a second determining module 2600, a second stopping module 2800, and a converting module 2900.
A first determining module 2200, configured to determine whether the used storage space of the storage server exceeds a storage space threshold corresponding to the first data type;
a first stopping module 2400, configured to stop receiving data of a first data type when the used storage space exceeds a storage space threshold corresponding to the first data type;
a second determining module 2600, configured to determine whether an used storage space of the storage server exceeds a storage space threshold corresponding to a second data type;
a second stopping module 2800, configured to stop receiving data of the second data type when the used storage space exceeds a storage space threshold corresponding to the second data type;
a conversion module 2900 configured to convert at least a portion of the received data of the first data type to data of a second data type;
the storage space threshold corresponding to the first data type is smaller than the storage space threshold corresponding to the second data type, the storage space threshold corresponding to the second data type is smaller than the total storage space of the storage server, and after the data of the first data type is converted into the data of the second data type, the occupied storage space is reduced.
In one example, the first data type is original data, and the second data type is converted data which occupies a smaller storage space than the original data after being subjected to conversion processing of the original data.
In one example, the data storage processing apparatus 2000 further comprises:
and a sending module (not shown in the figure) configured to convert at least part of the received data of the first data type into data of a second data type, and then send the converted data of at least part of the second data type to another storage server.
In one example, the determining by the first determining module 2200 whether the used storage space of the storage server exceeds a storage space threshold corresponding to a first data type includes: when a write-in request of data of a first data type is received, judging whether the used storage space of the storage server exceeds a storage space threshold corresponding to the first data type; or judging whether the used storage space of the storage server exceeds a storage space threshold corresponding to the first data type according to a preset first judgment cycle.
In one example, the second determining module 2600 of determining whether the used storage space of the storage server exceeds a storage space threshold corresponding to a second data type includes: after stopping receiving the data of the first data type, when receiving a write request of the data of the second data type, judging whether the used storage space of the storage server exceeds a storage space threshold corresponding to the second data type; or after stopping receiving the data of the first data type, judging whether the used storage space of the storage server exceeds the storage space threshold corresponding to the second data type according to a preset second judgment period.
In one example, the data storage processing apparatus 2000 further comprises:
a receiving module (not shown in the figure) for receiving a data modification request of a client; and a storage module (not shown in the figure), which acquires and stores a modified version data copy of the original data based on the pre-stored original data corresponding to the data modification request.
In one example, the data storage processing apparatus 2000 further comprises:
a first determining module (not shown in the figure) for determining whether the available storage space of the storage server is greater than a first storage threshold;
the step of acquiring and storing a modified version data copy of the original data by the storage module based on the pre-stored original data corresponding to the data modification request comprises: if the available storage space of the storage server is larger than a first storage threshold value, acquiring and storing a modified version data copy of original data based on the pre-stored original data corresponding to the data modification request;
and, the data storage processing apparatus 2000 further includes:
a deleting module (not shown in the figure) for deleting the original data after storing the modified version data copy of the original data.
In one example, the apparatus 2000 further comprises:
a second determining module (not shown in the figure) for determining whether the available storage space of the storage server is less than a second storage threshold;
and a recovery module (not shown in the figure), if the available storage space of the storage server is smaller than a second storage threshold, the recovery module locally executes the data recovery service to delete the original data corresponding to the modified version data copy, and release the storage space.
Referring now to fig. 8, the data storage processing apparatus 3000 of the embodiment of fig. 8 differs from the data storage processing apparatus 2000 of the embodiment of fig. 7 in that the data storage processing apparatus 2000 performs comparison and judgment according to the used storage space Ht and the storage space thresholds H1, H2 corresponding to the first and second data type data, and the data storage processing apparatus 3000 performs comparison and judgment according to the available storage space Hs and the storage space thresholds H1, H2 corresponding to the first and second data type data.
As shown in fig. 8, the data storage processing apparatus 3000 includes a first determining module 3200, a first stopping module 3400, a second determining module 3600, a second stopping module 3800, and a converting module 3900.
A first determining module 3200, configured to determine whether an available storage space of the storage server is smaller than a storage space threshold corresponding to a first data type;
a first stopping module 3400, configured to stop receiving data of the first data type when the available storage space is smaller than a storage space threshold corresponding to the first data type;
a second determining module 3600, configured to determine whether an available storage space of the storage server is smaller than a storage space threshold corresponding to a second data type;
a second stopping module 3800, configured to stop receiving data of the second data type when the available storage space is smaller than a storage space threshold corresponding to the second data type;
a conversion module 3900 that converts at least a portion of the received data of the first data type to data of a second data type;
the storage space threshold corresponding to the first data type is smaller than the storage space threshold corresponding to the second data type, the storage space threshold corresponding to the first data type is smaller than the total storage space of the storage server, and after the data of the first data type is converted into the data of the second data type, the occupied storage space is reduced.
In one example, the first data type is original data, and the second data type is converted data which occupies a smaller storage space than the original data after being subjected to conversion processing of the original data.
According to still another embodiment of the present invention, there is also provided an electronic device, and the electronic device 4000 may be the server 1000 shown in fig. 1. Fig. 9 is a block diagram of an electronic device according to an embodiment of the present invention.
In one aspect, the electronic device 4000 may include the aforementioned data storage processing apparatus 2000 or 3000 for a distributed storage system, so as to implement the data storage processing method for a distributed storage system according to any embodiment of the present invention.
On the other hand, as shown in fig. 9, the electronic device 4000 may include a memory 4200 and a processor 4400, the memory 4200 being configured to store executable instructions; the instructions are used to control the processor 4400 to perform the methods described above for space management for a distributed storage system.
In this embodiment, the electronic device 4000 may be the server 1000 in fig. 1, or may be another terminal device, which is not limited herein.
Finally, in the present embodiment, there is also provided a computer-readable storage medium on which a computer program is stored, the computer program, when being executed by a processor, implementing the data storage processing method for a distributed storage system according to any embodiment of the present invention.
The present invention may be a system, method and/or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions embodied therewith for causing a processor to implement various aspects of the present invention.
The computer readable storage medium may be a tangible device that can hold and store the instructions for use by the instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as punch cards or in-groove projection structures having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or electrical signals transmitted through electrical wires.
The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device via a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing/processing device.
The computer program instructions for carrying out operations of the present invention may be assembler instructions, Instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, aspects of the present invention are implemented by personalizing an electronic circuit, such as a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA), with state information of computer-readable program instructions, which can execute the computer-readable program instructions.
Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.
These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are equivalent.
Having described embodiments of the present invention, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen in order to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.
Claims (12)
1. A data storage processing method for a distributed storage system, the distributed storage system including at least one storage server, the method being applied to the storage server, the method comprising:
judging whether the used storage space of the storage server exceeds a storage space threshold corresponding to a first data type;
stopping receiving the data of the first data type when the used storage space exceeds a storage space threshold corresponding to the first data type;
judging whether the used storage space of the storage server exceeds a storage space threshold corresponding to a second data type;
stopping receiving the data of the second data type when the used storage space exceeds a storage space threshold corresponding to the second data type;
converting at least part of the received data of the first data type into data of a second data type;
the storage space threshold corresponding to the first data type is smaller than the storage space threshold corresponding to the second data type, the storage space threshold corresponding to the second data type is smaller than the total storage space of the storage server, and after the data of the first data type is converted into the data of the second data type, the occupied storage space is reduced.
2. The method of claim 1, further comprising:
and after converting at least part of the received data of the first data type into data of a second data type, sending the converted data of at least part of the second data type to other storage servers.
3. The method of claim 1, wherein determining whether the used storage space of the storage server exceeds a storage space threshold corresponding to a first data type comprises:
when a write-in request of data of a first data type is received, judging whether the used storage space of the storage server exceeds a storage space threshold corresponding to the first data type;
alternatively, the first and second electrodes may be,
and judging whether the used storage space of the storage server exceeds a storage space threshold corresponding to the first data type according to a preset first judgment period.
4. The method of claim 1, wherein determining whether the used storage space of the storage server exceeds a storage space threshold corresponding to a second data type comprises:
after stopping receiving the data of the first data type, when receiving a write request of the data of the second data type, judging whether the used storage space of the storage server exceeds a storage space threshold corresponding to the second data type;
alternatively, the first and second electrodes may be,
and after stopping receiving the data of the first data type, judging whether the used storage space of the storage server exceeds a storage space threshold corresponding to the second data type according to a preset second judgment period.
5. The method of claim 1, further comprising:
receiving a data modification request of a client;
and acquiring and storing a modified version data copy of the original data based on the pre-stored original data corresponding to the data modification request.
6. The method according to claim 5, wherein after receiving a data modification request from a client, before obtaining and storing a modified version data copy of original data based on pre-stored original data corresponding to the data modification request, the method further comprises:
determining whether an available storage space of the storage server is greater than a first storage threshold;
the obtaining and storing a modified version data copy of the original data based on pre-stored original data corresponding to the data modification request comprises:
if the available storage space of the storage server is larger than a first storage threshold value, acquiring and storing a modified version data copy of original data based on the pre-stored original data corresponding to the data modification request;
and, the method further comprises:
and deleting the original data after storing the modified version data copy of the original data.
7. The method of claim 5, further comprising:
determining whether an available storage space of the storage server is less than a second storage threshold;
and if the available storage space of the storage server is smaller than a second storage threshold value, locally executing the data recovery service to delete the original data corresponding to the modified version data copy and release the storage space.
8. A data storage processing method for a distributed storage system, the distributed storage system including at least one storage server, the method being applied to the storage server, the method comprising:
judging whether the available storage space of the storage server is smaller than a storage space threshold corresponding to a first data type;
when the available storage space is smaller than a storage space threshold corresponding to the first data type, stopping receiving the data of the first data type;
judging whether the available storage space of the storage server is smaller than a storage space threshold corresponding to a second data type;
when the available storage space is smaller than a storage space threshold corresponding to the second data type, stopping receiving the data of the second data type;
converting at least part of the received data of the first data type into data of a second data type;
the storage space threshold corresponding to the first data type is smaller than the storage space threshold corresponding to the second data type, the storage space threshold corresponding to the first data type is smaller than the total storage space of the storage server, and after the data of the first data type is converted into the data of the second data type, the occupied storage space is reduced.
9. A data storage processing apparatus for a distributed storage system including at least one storage server, the apparatus being applied to the storage server, the apparatus comprising:
the first judgment module is used for judging whether the used storage space of the storage server exceeds a storage space threshold corresponding to a first data type;
the first stopping module is used for stopping receiving the data of the first data type when the used storage space exceeds a storage space threshold corresponding to the first data type;
the second judgment module is used for judging whether the used storage space of the storage server exceeds a storage space threshold corresponding to a second data type;
the second stopping module is used for stopping receiving the data of the second data type when the used storage space exceeds a storage space threshold corresponding to the second data type;
the conversion module is used for converting at least part of the received data of the first data type into data of a second data type;
the storage space threshold corresponding to the first data type is smaller than the storage space threshold corresponding to the second data type, the storage space threshold corresponding to the second data type is smaller than the total storage space of the storage server, and after the data of the first data type is converted into the data of the second data type, the occupied storage space is reduced.
10. A data storage processing apparatus for a distributed storage system including at least one storage server, the apparatus being applied to the storage server, the apparatus comprising:
the first judging module is used for judging whether the available storage space of the storage server is smaller than a storage space threshold corresponding to a first data type;
the first stopping module is used for stopping receiving the data of the first data type when the available storage space is smaller than a storage space threshold corresponding to the first data type;
the second judging module is used for judging whether the available storage space of the storage server is smaller than a storage space threshold corresponding to a second data type;
the second stopping module is used for stopping receiving the data of the second data type when the available storage space is smaller than the storage space threshold corresponding to the second data type;
the conversion module is used for converting at least part of the received data of the first data type into data of a second data type;
the storage space threshold corresponding to the first data type is smaller than the storage space threshold corresponding to the second data type, the storage space threshold corresponding to the first data type is smaller than the total storage space of the storage server, and after the data of the first data type is converted into the data of the second data type, the occupied storage space is reduced.
11. An electronic device, comprising:
a data storage processing apparatus for a distributed storage system according to claim 9 or 10; alternatively, the first and second electrodes may be,
a processor and a memory for storing executable instructions for controlling the processor to perform the data storage processing method for a distributed storage system according to any one of claims 1 to 8.
12. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the data storage processing method for a distributed storage system according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911252196.2A CN111124291B (en) | 2019-12-09 | 2019-12-09 | Data storage processing method and device of distributed storage system and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911252196.2A CN111124291B (en) | 2019-12-09 | 2019-12-09 | Data storage processing method and device of distributed storage system and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111124291A true CN111124291A (en) | 2020-05-08 |
| CN111124291B CN111124291B (en) | 2023-05-30 |
Family
ID=70497957
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911252196.2A Active CN111124291B (en) | 2019-12-09 | 2019-12-09 | Data storage processing method and device of distributed storage system and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111124291B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112699124A (en) * | 2020-12-30 | 2021-04-23 | 罗普特科技集团股份有限公司 | Circulating storage method and system based on distributed multi-storage service |
| CN112737977A (en) * | 2020-12-28 | 2021-04-30 | 苏州浪潮智能科技有限公司 | Data packet processing method and device |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150161148A1 (en) * | 2013-12-11 | 2015-06-11 | Jdsu Uk Limited | Method and apparatus for managing data |
| US20160055097A1 (en) * | 2014-08-19 | 2016-02-25 | Yang Seok KI | Heterogeneous unified memory |
| CN108256087A (en) * | 2018-01-22 | 2018-07-06 | 北京腾云天下科技有限公司 | A kind of data importing, inquiry and processing method based on bitmap structure |
| CN109085999A (en) * | 2018-06-15 | 2018-12-25 | 华为技术有限公司 | data processing method and processing system |
| CN109213745A (en) * | 2018-08-27 | 2019-01-15 | 郑州云海信息技术有限公司 | A kind of distributed document storage method, device, processor and storage medium |
| CN109284073A (en) * | 2018-09-30 | 2019-01-29 | 北京金山云网络技术有限公司 | Date storage method, device, system, server, control node and medium |
| CN109753227A (en) * | 2017-11-08 | 2019-05-14 | 北京嘀嘀无限科技发展有限公司 | Storage method, device, mobile terminal, server and readable storage medium storing program for executing |
| CN110471614A (en) * | 2018-05-09 | 2019-11-19 | 杭州海康威视数字技术股份有限公司 | A kind of method of storing data, the method and device for detecting terminal |
-
2019
- 2019-12-09 CN CN201911252196.2A patent/CN111124291B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150161148A1 (en) * | 2013-12-11 | 2015-06-11 | Jdsu Uk Limited | Method and apparatus for managing data |
| US20160055097A1 (en) * | 2014-08-19 | 2016-02-25 | Yang Seok KI | Heterogeneous unified memory |
| CN109753227A (en) * | 2017-11-08 | 2019-05-14 | 北京嘀嘀无限科技发展有限公司 | Storage method, device, mobile terminal, server and readable storage medium storing program for executing |
| CN108256087A (en) * | 2018-01-22 | 2018-07-06 | 北京腾云天下科技有限公司 | A kind of data importing, inquiry and processing method based on bitmap structure |
| CN110471614A (en) * | 2018-05-09 | 2019-11-19 | 杭州海康威视数字技术股份有限公司 | A kind of method of storing data, the method and device for detecting terminal |
| CN109085999A (en) * | 2018-06-15 | 2018-12-25 | 华为技术有限公司 | data processing method and processing system |
| CN109213745A (en) * | 2018-08-27 | 2019-01-15 | 郑州云海信息技术有限公司 | A kind of distributed document storage method, device, processor and storage medium |
| CN109284073A (en) * | 2018-09-30 | 2019-01-29 | 北京金山云网络技术有限公司 | Date storage method, device, system, server, control node and medium |
Non-Patent Citations (1)
| Title |
|---|
| 吴程熙: "混合型分布式存储系统中的存储策略研究与实现" * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112737977A (en) * | 2020-12-28 | 2021-04-30 | 苏州浪潮智能科技有限公司 | Data packet processing method and device |
| CN112699124A (en) * | 2020-12-30 | 2021-04-23 | 罗普特科技集团股份有限公司 | Circulating storage method and system based on distributed multi-storage service |
| CN112699124B (en) * | 2020-12-30 | 2022-05-17 | 罗普特科技集团股份有限公司 | Circulating storage method and system based on distributed multi-storage service |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111124291B (en) | 2023-05-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109274731B (en) | Method and device for deploying and calling web service based on multi-tenant technology | |
| CN109213611B (en) | Cross-process communication method, device, terminal and storage medium | |
| CN111880967A (en) | File backup method, device, medium and electronic equipment in cloud scene | |
| CN111124291B (en) | Data storage processing method and device of distributed storage system and electronic equipment | |
| CN113791917A (en) | Method, device and system for processing delay message | |
| CN111290871A (en) | Method and device for acquiring crash information of application program | |
| CN111586177B (en) | Cluster session loss prevention method and system | |
| CN115904761A (en) | System on chip, vehicle and video processing unit virtualization method | |
| CN115220876A (en) | Virtual resource creating method, device, program product, medium and electronic equipment | |
| US10884818B2 (en) | Increasing processing capacity of virtual machines | |
| CN113656193A (en) | Application communication method, system, electronic equipment and storage medium | |
| US11303587B2 (en) | Chatbot information processing | |
| CN116521639A (en) | Log data processing method, electronic equipment and computer readable medium | |
| CN113076175B (en) | Memory sharing method and device for virtual machine | |
| CN113779122B (en) | Method and device for exporting data | |
| US11386043B2 (en) | Method, device, and computer program product for managing snapshot in application environment | |
| CN114070889A (en) | Configuration method, traffic forwarding method, device, storage medium, and program product | |
| CN112925796A (en) | Write consistency control method, device, equipment and storage medium | |
| CN113919310A (en) | Short message content determination method and device, electronic equipment and storage medium | |
| CN106886373B (en) | Physical machine and magnetic disk operation method and device thereof | |
| CN111158649A (en) | Method and device for configuring multi-level parameters | |
| CN113448960A (en) | Method and device for importing form file | |
| CN114996169B (en) | Device diagnosis method, device, electronic device, and storage medium | |
| CN112559001A (en) | Method and device for updating application | |
| CN111385334A (en) | Data distribution method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |