CN111143364A - Data block storage method and device, electronic equipment and storage medium - Google Patents
Data block storage method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN111143364A CN111143364A CN201911350188.1A CN201911350188A CN111143364A CN 111143364 A CN111143364 A CN 111143364A CN 201911350188 A CN201911350188 A CN 201911350188A CN 111143364 A CN111143364 A CN 111143364A
- Authority
- CN
- China
- Prior art keywords
- block storage
- data
- stored
- data stream
- service system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24568—Data stream processing; Continuous queries
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a data block storage method, a data block storage device, electronic equipment and a storage medium, which are applied to a cloud storage system, wherein the method comprises the following steps: receiving a data stream to be stored sent by a service system; inquiring a service system grouping directory, and determining a block storage list to which a service system belongs; acquiring a block storage list from a storage address of the block storage list; the block storage list is used for recording the mapping relation between the data characteristics of the data stream generated by the service system and the block storage nodes corresponding to the data characteristics of the data stream generated by the service system; determining data characteristics of a data stream to be stored; querying a block storage list by using the data characteristics, and determining a block storage node corresponding to the data characteristics; and storing the data stream to be stored to the storage node. The method and the device avoid the additional purchase of large-capacity storage equipment, and save the cost; corresponding data can be quickly inquired; and the data sharing of the field, the service system and the department can be realized.
Description
Technical Field
The present application relates to the field of cloud computing technologies, and in particular, to a data block storage method and apparatus, an electronic device, and a storage medium.
Background
The smart city fully applies a new generation of information technology to various industries in the city, realizes informatization and industrialization based on the urban informatization advanced form of the knowledge society, is deeply integrated with urbanization, contributes to improving the urbanization quality, realizes fine and dynamic management, improves the urban management effect and improves the quality of life of citizens.
The smart city needs to cover all fields in one city, including city management, social and civil life, resource environment, industrial economy, characteristic service and other fields, and each field includes a plurality of business systems with various obvious levels. As can be seen, the amount of data for a smart city is enormous.
In the related art, the storage modes of data across fields, systems and departments are dispersed and fixed, the fusion of the data cannot be realized, and the storage efficiency is low.
Disclosure of Invention
The present application provides a data block storage method, apparatus, electronic device and storage medium to solve the above-mentioned problems.
The application provides a data block storage method in a first aspect, which is applied to a cloud storage system, and the method comprises the following steps:
receiving a data stream to be stored sent by a service system;
inquiring a service system grouping directory, and determining a block storage list to which the service system belongs; the service system grouping directory is used for recording the mapping relation between the identity information of the service system and a block storage list to which the service system belongs and the storage address of the block storage list;
acquiring the block storage list from the storage address of the block storage list; the block storage list is used for recording the mapping relation between the data characteristics of the data stream generated by the service system and the block storage nodes corresponding to the data characteristics of the data stream generated by the service system;
determining data characteristics of the data stream to be stored;
querying the block storage list by using the data characteristics, and determining a block storage node corresponding to the data characteristics;
and storing the data stream to be stored to the storage node.
Further, determining the block storage node corresponding to the data feature specifically includes:
judging whether the use frequency of the data stream to be stored in a preset time exceeds a preset threshold value or not;
if the use frequency of the data stream to be stored in a preset time exceeds a preset threshold value, inquiring the block storage list, determining a cache level block storage node corresponding to the data characteristics, and taking the cache level block storage node as the block storage node corresponding to the data characteristics;
and if the use frequency of the data stream to be stored in the preset time does not exceed a preset threshold value, inquiring the block storage list, determining a disk block storage node corresponding to the data characteristic, and taking the disk block storage node as the block storage node corresponding to the data characteristic.
Further, after receiving the data stream to be stored sent by the service system, the method further includes:
when the block storage list of the service system cannot be acquired according to a service system grouping directory, determining a storage space required by the data stream to be stored and determining data characteristics of the data stream to be stored;
configuring corresponding block storage nodes for the data characteristics according to the storage space required by the data stream to be stored and the data characteristics of the data stream to be stored, and storing the data stream to be stored in the block storage nodes corresponding to the data characteristics;
establishing a block storage list to which the service system belongs according to the mapping relation between the data characteristics of the data stream to be stored and the block storage nodes corresponding to the data characteristics of the data stream to be stored, and storing the block storage list;
and updating the mapping relation between the service system and the block storage list to which the service system belongs and the storage address of the block storage list into the service system grouping directory.
Further, after determining the data characteristics of the data stream to be stored, the method further comprises:
when the block storage node corresponding to the data feature cannot be determined in the block storage list according to the data feature of the data stream to be stored, configuring the corresponding block storage node for the data feature according to the storage space required by the data stream to be stored and the data feature of the data stream to be stored, and storing the data stream to be stored in the block storage node corresponding to the data feature;
and storing the mapping relation between the data characteristics of the data stream to be stored and the block storage nodes corresponding to the data characteristics of the data stream to be stored into a block storage list to which the service system belongs.
A second aspect of the present application provides a data block storage device, the device comprising:
the receiving module is used for receiving the data stream to be stored sent by the service system;
the query module is used for querying a service system grouping directory and determining a block storage list to which the service system belongs; the service system grouping directory is used for recording the mapping relation between the service system and a block storage list to which the service system belongs and the storage address of the block storage list;
an obtaining module, configured to obtain the block storage list from a storage address of the block storage list; the block storage list is used for recording the mapping relation between the data characteristics of the data stream generated by the service system and the block storage nodes corresponding to the data characteristics of the data stream generated by the service system;
the first determining module is used for determining the data characteristics of the data stream to be stored;
a second determining module, configured to determine, according to the data feature of the data stream to be stored and the block storage list, a block storage node corresponding to the data feature;
and the first storage module is used for storing the data stream to be stored to the storage node.
Further, the second determining module includes:
the judging submodule is used for judging whether the use frequency of the data stream to be stored in the preset time exceeds a preset threshold value or not;
the first determining submodule is used for querying the block storage list, determining a cache level block storage node corresponding to the data characteristic and taking the cache level block storage node as the block storage node corresponding to the data characteristic if the use frequency of the data stream to be stored in a preset time exceeds a preset threshold value;
and the second determining submodule is used for querying the block storage list, determining a disk block storage node corresponding to the data characteristic and taking the disk block storage node as the block storage node corresponding to the data characteristic if the use frequency of the data stream to be stored in a preset time does not exceed a preset threshold value.
Further, the apparatus further comprises:
a third determining module, configured to determine, when the block storage list of the service system cannot be obtained according to a service system grouping directory, a storage space required by the data stream to be stored, and determine a data feature of the data stream to be stored;
the second storage module is used for configuring corresponding block storage nodes for the data characteristics according to the storage space required by the data stream to be stored and the data characteristics of the data stream to be stored, and storing the data stream to be stored in the block storage nodes corresponding to the data characteristics;
the first building module is used for building a block storage list to which the service system belongs according to the mapping relation between the data characteristics of the data stream to be stored and the block storage nodes corresponding to the data characteristics of the data stream to be stored, and storing the block storage list;
and the first updating module is used for updating the business system, the mapping relation of the block storage list to which the business system belongs and the storage address of the block storage list into the business system grouping directory.
Further, the apparatus further comprises:
a fourth determining module, configured to, when a block storage node corresponding to the data feature cannot be determined in the block storage list according to the data feature of the data stream to be stored, configure a corresponding block storage node for the data feature according to a storage space required by the data stream to be stored and the data feature of the data stream to be stored, and store the data stream to be stored in the block storage node corresponding to the data feature;
and the second updating module is used for storing the mapping relation between the data characteristics of the data stream to be stored and the block storage nodes corresponding to the data characteristics of the data stream to be stored to the block storage list to which the service system belongs.
A third aspect of the present application provides an electronic device comprising:
a processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute to implement the one data block storage method.
A fourth aspect of the present application provides a non-transitory computer-readable storage medium, wherein instructions of the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform a data block storage method as described.
Compared with the prior art, the method has the following advantages:
according to the method and the system, the cloud storage system is arranged, and the original storage devices of all service systems in all the existing fields are combined to obtain the storage node cluster, so that the additional purchase of large-capacity storage devices is avoided, and the cost is saved; the management server is used for storing the service system grouping directory and the block storage list, namely, the storage node cluster and each data stored in the storage node cluster are managed, and when a client needs to access some data, the management server can inquire the service system grouping directory and the block storage list so as to quickly inquire the corresponding data; because the management server manages the data stored in all the service systems, the data of the collapse field, the collapse service system and even the collapse department can be inquired, and the data sharing of the collapse field, the collapse service system and the collapse department is further realized.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments of the present application will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.
FIG. 1 is a flow chart illustrating the steps of a data storage method provided herein;
FIG. 2 is a schematic diagram of a data storage device provided herein;
fig. 3 is a schematic structural diagram of an electronic device provided in the present application.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present application more comprehensible, the present application is described in further detail with reference to the accompanying drawings and the detailed description.
The smart city aims to build a large data center by taking data concentration and sharing as a way, promote technical fusion, business fusion and data fusion, and realize cross-level, cross-region, cross-system, cross-department and cross-business cooperative management and service.
In the related technology, the storage modes of data across fields, systems and departments are dispersed and fixed, namely, each service system is also stored in respective storage equipment, when the data of different service systems need to be associated, a communication channel is established between the two storage equipment needing to be associated, so that the data of the two service systems are obtained, and when the data do not need to be associated, the communication channel between the two storage equipment is disconnected, so that the data are dispersed, the storage resources of each service system cannot be shared, and the storage efficiency is low. In addition, each storage device is independent, and the stored data is data inside a department or inside a system, which cannot avoid the occurrence of the following situations, for example, the data of some departments is not enough, so that the utilization rate of storage resources is low, and the waste of the storage resources is caused; and certain parts of data are abundant, so that the storage resources are insufficient, and a storage device needs to be added periodically, so that the cost is high.
In order to solve the above problems, the present application provides a data block storage method as shown in fig. 1, which is applied to a cloud storage system, where the cloud storage system includes a client, a management server, and a storage node cluster, where the client faces a service system and receives a data stream to be stored, which is sent by the service system; the management server is used for storing a service system grouping directory and a block storage list; the storage node cluster is used for providing mass storage space. The acquisition mode of the storage node cluster in the application is as follows: combining original storage equipment of each service system in each field to obtain a disk array; and then determining a virtual logical disk according to the disk array, and using the virtual logical disk as a storage node cluster in the application. Because additional storage equipment does not need to be added independently, the cost can be effectively reduced. Meanwhile, all the storage devices are used in a virtual logic disk mode, so that the use efficiency of each storage device can be greatly improved, and the condition that the storage resources are wasted due to low utilization rate of the storage resources caused by low data of certain departments can be avoided; the situation that the storage resources are insufficient and storage equipment needs to be added regularly due to the fact that a large amount of data exists in some parts can be avoided, and the cost is high.
The method comprises the following steps:
step S11, receiving data flow to be stored sent by the service system;
the method is mainly applied to the storage of cross-domain, system and department of smart city, and is applied to the application environment with wide domain, multiple systems and multiple departments. For example, the system comprises various fields of city management, social livelihood, resource environment, industrial economy, special services and the like, and more specifically, the business system in the city management field can be a business system of government affairs, traffic, city management, security, fire fighting and the like; the business system in the social and civil field can be a medical, community, social security, education and other business system; the service system in the resource environment field can be an energy, environment-friendly, ecological and other service system; the business system in the industrial economic field can be business systems of tourism, finance, port, park, enterprise service and the like; the business system in the characteristic service field can be a city ecological service platform, a postdoctor management service platform, a scientific and technological enterprise management service platform, a self-service enterprise management platform and other business systems. More specifically, each business system further includes each department.
In the application, each service system has its own identity information, and the identity information of each service system is unique. The identity information is carried in the data stream to be stored sent by the service system, and after receiving the data stream to be stored sent by the service system, the client acquires the identity information of the service system sending the data stream to be stored from the data stream to be stored.
Step S12, inquiring the grouping directory of the service system, and determining the block storage list to which the service system belongs; the service system grouping directory is used for recording the mapping relation between the identity information of the service system and a block storage list to which the service system belongs and the storage address of the block storage list;
and the management server inquires a service system grouping directory according to the identity information of the service system which sends the data stream to be stored and is acquired from the data stream to be stored by the client. The service system grouping directory stores the mapping relation between the identity information of the service system and the block storage list to which the service system belongs, and the storage address of the block storage list. And the management server determines a block storage list corresponding to the service system according to the identity information of the service system.
Specifically, after receiving a data stream to be stored sent by a service system, the method further includes:
step S1201, when the block storage list of the service system cannot be obtained according to the service system grouping directory, determining the storage space required by the data stream to be stored and determining the data characteristics of the data stream to be stored;
that is, when the service system stores a data stream to the cloud storage system for the first time, the service system does not exist in the service system grouping directory, and a mapping relationship between the identity information of the service system and the block storage list to which the service system belongs does not exist. At this time, it is necessary to determine the storage space required by the data stream to be stored, and determine the data characteristics of the data stream to be stored (the data characteristics may be already specified by the service system when generating the data stream to be stored).
Step S1202, configuring corresponding block storage nodes for the data characteristics according to the storage space required by the data stream to be stored and the data characteristics of the data stream to be stored, and storing the data stream to be stored in the block storage nodes corresponding to the data characteristics;
after receiving the data stream to be stored, in order to avoid data congestion, a corresponding block storage node is configured for the data stream to be stored, and the data stream to be stored is stored in the block storage node.
Step S1203, establishing a block storage list to which the service system belongs according to the mapping relation between the data characteristics of the data stream to be stored and the block storage nodes corresponding to the data characteristics of the data stream to be stored, and storing the block storage list;
after the data stream to be stored is stored, a corresponding block storage list is created for the service system according to the mapping relation between the data characteristics of the data stream to be stored and the block storage nodes corresponding to the data characteristics of the data stream to be stored, and the block storage list is stored.
Step S1204, updating the mapping relationship between the service system and the block storage list to which the service system belongs, and the storage address of the block storage list into the service system grouping directory.
And finally, updating the mapping relation of the service system and the block storage list to which the service system belongs and the storage address of the block storage list into a service system grouping directory. When the service system sends the data stream to be stored again next time, the data stream to be stored can be stored according to the mode of inquiring the service system packet directory.
Step S13, obtaining the block storage list from the storage address of the block storage list; the block storage list is used for recording the mapping relation between the data characteristics of the data stream generated by the service system and the block storage nodes corresponding to the data characteristics of the data stream generated by the service system;
since the number of business systems is large, and as cities develop, the number of business systems is gradually increased. If the mapping relationship between the identity information of the service system and the storage node of the service system is directly passed through, the data in the service system may only be stored in the storage device in a messy manner, and the next-level classification cannot be performed any more. For example, the social security service system may include social security information of a plurality of urban areas and a plurality of towns of a city. Over time, and as urban and rural populations move, the social security business system will have more and more data. If the mapping relationship between the identity information of the service system and the storage node of the service system is directly passed, that means that the storage area of the storage device corresponding to the service system is an integral area, all data of the social security service system is stored in the storage area, and the storage area cannot be finely divided according to the urban area and the village and the town of the social security, so that the effective management cannot be performed under the condition that the data volume is more and more.
Therefore, the mapping relation between the identity information of the service system and the block storage list is stored in the service system grouping directory by establishing the service system grouping directory, and the data characteristics are divided in the block storage list of the service system according to the characteristics of the service system, so that the data volume generated by the service system is effectively managed. The data feature division mentioned in the present application may be performed according to data features in a business system (for example, pictures, documents, videos, audios, etc.), or may be performed according to various functional departments in the business system (for example, finance department, personnel department, technical department, management department, etc.), or may be performed according to a geographic area related to the business system (for example, a district a, a district B, and a district C in a downtown, a township 1, a township 2, etc.).
Step S14, determining the data characteristics of the data stream to be stored;
the data stream to be stored may carry an identifier for identifying the data characteristics of the data stream to be stored, and the data characteristics of the data stream to be stored are determined.
Specifically, after determining the data characteristics of the data stream to be stored, the method further includes:
step 1401, when a block storage node corresponding to the data feature cannot be determined in the block storage list according to the data feature of the data stream to be stored, configuring a corresponding block storage node for the data feature according to a storage space required by the data stream to be stored and the data feature of the data stream to be stored, and storing the data stream to be stored in the block storage node corresponding to the data feature;
step 1402, storing the mapping relationship between the data characteristics of the data stream to be stored and the block storage nodes corresponding to the data characteristics of the data stream to be stored to the block storage list to which the service system belongs.
The service system may adjust the data type, department setting, and the like according to actual requirements, that is, the service system may generate new data characteristics, and when the transmitted data stream to be stored has the new data characteristics, a corresponding mapping relationship does not exist in the block storage list.
After receiving the data stream to be stored with the new data characteristics, in order to avoid data congestion, determining a storage space and data characteristics required by the data stream to be stored, configuring block storage nodes for the data stream to be stored, storing the block storage nodes in the block storage nodes, and updating a mapping relation between the data characteristics of the data stream to be stored and the block storage nodes corresponding to the data characteristics of the data stream to be stored to a block storage list to which the service system belongs. When the service system generates the data stream to be stored with the data characteristics again, the data stream can be stored in a mode of inquiring the block storage list.
Step S15, inquiring the block storage list of data characteristics, and determining the block storage nodes corresponding to the data characteristics; specifically, determining the block storage node corresponding to the data feature specifically includes:
step S1501, judging whether the use frequency of the data stream to be stored in the preset time exceeds a preset threshold value; step S1502, if the usage frequency of the data stream to be stored in a preset time exceeds a preset threshold, querying the block storage list, determining a cache-level block storage node corresponding to the data feature, and using the cache-level block storage node as the block storage node corresponding to the data feature;
step S1503, if the frequency of use of the data stream to be stored in a preset time does not exceed a preset threshold, querying the block storage list, determining a disk block storage node corresponding to the data feature, and using the disk block storage node as a block storage node corresponding to the data feature.
In order to improve the acquisition speed of the data stream, the use frequency of the data stream to be stored is monitored so as to judge whether the data stream to be stored is a common data stream, the common data stream can be placed in a block storage node which is accessed quickly, and an uncommon data stream is placed in a common block storage node. The cache level block storage node may be a block storage node dedicated to data commonly used by all service systems.
And step S16, storing the data stream to be stored to the storage node.
The storage node is composed of a plurality of storage blocks with fixed storage space, the data stream to be stored is divided into a plurality of data blocks according to the size of the storage space, and the plurality of data blocks are stored in the corresponding storage blocks, so that the storage of the data stream to be stored is realized.
According to the method and the system, the cloud storage system is arranged, and the original storage devices of all service systems in all the existing fields are combined to obtain the storage node cluster, so that the additional purchase of large-capacity storage devices is avoided, and the cost is saved; the management server is used for storing the service system grouping directory and the block storage list, namely, the storage node cluster and each data stored in the storage node cluster are managed, and when a client needs to access some data, the management server can inquire the service system grouping directory and the block storage list so as to quickly inquire the corresponding data; because the management server manages the data stored in all the service systems, the data of the collapse field, the collapse service system and even the collapse department can be inquired, and the data sharing of the collapse field, the collapse service system and the collapse department is further realized.
The present application provides a data block storage apparatus as shown in fig. 2 based on the same technical concept, the apparatus includes:
the receiving module is used for receiving the data stream to be stored sent by the service system;
the query module is used for querying a service system grouping directory and determining a block storage list to which the service system belongs; the service system grouping directory is used for recording the mapping relation between the service system and a block storage list to which the service system belongs and the storage address of the block storage list;
an obtaining module, configured to obtain the block storage list from a storage address of the block storage list; the block storage list is used for recording the mapping relation between the data characteristics of the data stream generated by the service system and the block storage nodes corresponding to the data characteristics of the data stream generated by the service system;
the first determining module is used for determining the data characteristics of the data stream to be stored;
a second determining module, configured to determine, according to the data feature of the data stream to be stored and the block storage list, a block storage node corresponding to the data feature;
and the first storage module is used for storing the data stream to be stored to the storage node.
Specifically, the second determining module includes:
the judging submodule is used for judging whether the use frequency of the data stream to be stored in the preset time exceeds a preset threshold value or not;
the first determining submodule is used for querying the block storage list, determining a cache level block storage node corresponding to the data characteristic and taking the cache level block storage node as the block storage node corresponding to the data characteristic if the use frequency of the data stream to be stored in a preset time exceeds a preset threshold value;
and the second determining submodule is used for querying the block storage list, determining a disk block storage node corresponding to the data characteristic and taking the disk block storage node as the block storage node corresponding to the data characteristic if the use frequency of the data stream to be stored in a preset time does not exceed a preset threshold value.
Specifically, the apparatus further comprises:
a third determining module, configured to determine, when the block storage list of the service system cannot be obtained according to a service system grouping directory, a storage space required by the data stream to be stored, and determine a data feature of the data stream to be stored;
the second storage module is used for configuring corresponding block storage nodes for the data characteristics according to the storage space required by the data stream to be stored and the data characteristics of the data stream to be stored, and storing the data stream to be stored in the block storage nodes corresponding to the data characteristics;
the first building module is used for building a block storage list to which the service system belongs according to the mapping relation between the data characteristics of the data stream to be stored and the block storage nodes corresponding to the data characteristics of the data stream to be stored, and storing the block storage list;
and the first updating module is used for updating the business system, the mapping relation of the block storage list to which the business system belongs and the storage address of the block storage list into the business system grouping directory.
Specifically, the apparatus further comprises:
a fourth determining module, configured to, when a block storage node corresponding to the data feature cannot be determined in the block storage list according to the data feature of the data stream to be stored, configure a corresponding block storage node for the data feature according to a storage space required by the data stream to be stored and the data feature of the data stream to be stored, and store the data stream to be stored in the block storage node corresponding to the data feature;
and the second updating module is used for storing the mapping relation between the data characteristics of the data stream to be stored and the block storage nodes corresponding to the data characteristics of the data stream to be stored to the block storage list to which the service system belongs.
The present application also provides an electronic device as shown in fig. 3, including:
a processor 31;
a memory 32 for storing instructions executable by the processor 31;
wherein the processor 31 is configured to execute to implement the data block storage method.
The present application also provides a non-transitory computer readable storage medium, wherein instructions of the storage medium, when executed by a processor 31 of an electronic device, enable the electronic device to perform a data block storage method as described.
For the system embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, refer to the partial description of the method embodiment.
The embodiments in the present specification are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other.
The foregoing describes in detail a data block storage method, apparatus, electronic device, and storage medium provided by the present application, and specific examples are applied herein to explain the principles and implementations of the present application, and the descriptions of the foregoing examples are only used to help understand the method and core ideas of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.
Claims (10)
1. A data block storage method is applied to a cloud storage system, and comprises the following steps:
receiving a data stream to be stored sent by a service system;
inquiring a service system grouping directory, and determining a block storage list to which the service system belongs; the service system grouping directory is used for recording the mapping relation between the identity information of the service system and a block storage list to which the service system belongs and the storage address of the block storage list;
acquiring the block storage list from the storage address of the block storage list; the block storage list is used for recording the mapping relation between the data characteristics of the data stream generated by the service system and the block storage nodes corresponding to the data characteristics of the data stream generated by the service system;
determining data characteristics of the data stream to be stored;
inquiring the block storage list, and determining a block storage node corresponding to the data characteristic;
and storing the data stream to be stored to the storage node.
2. The method according to claim 1, wherein determining the block storage node corresponding to the data feature specifically comprises:
judging whether the use frequency of the data stream to be stored in a preset time exceeds a preset threshold value or not;
if the use frequency of the data stream to be stored in a preset time exceeds a preset threshold value, inquiring the block storage list, determining a cache level block storage node corresponding to the data characteristics, and taking the cache level block storage node as the block storage node corresponding to the data characteristics;
and if the use frequency of the data stream to be stored in the preset time does not exceed a preset threshold value, inquiring the block storage list, determining a disk block storage node corresponding to the data characteristic, and taking the disk block storage node as the block storage node corresponding to the data characteristic.
3. The method of claim 1, wherein after receiving the data stream to be stored sent by the service system, the method further comprises:
when the block storage list of the service system cannot be acquired according to a service system grouping directory, determining a storage space required by the data stream to be stored and determining data characteristics of the data stream to be stored;
configuring corresponding block storage nodes for the data characteristics according to the storage space required by the data stream to be stored and the data characteristics of the data stream to be stored, and storing the data stream to be stored in the block storage nodes corresponding to the data characteristics;
establishing a block storage list to which the service system belongs according to the mapping relation between the data characteristics of the data stream to be stored and the block storage nodes corresponding to the data characteristics of the data stream to be stored, and storing the block storage list;
and updating the mapping relation between the service system and the block storage list to which the service system belongs and the storage address of the block storage list into the service system grouping directory.
4. The method of claim 1, wherein after determining the data characteristics of the data stream to be stored, the method further comprises:
when the block storage node corresponding to the data feature cannot be determined in the block storage list according to the data feature of the data stream to be stored, configuring the corresponding block storage node for the data feature according to the storage space required by the data stream to be stored and the data feature of the data stream to be stored, and storing the data stream to be stored in the block storage node corresponding to the data feature;
and storing the mapping relation between the data characteristics of the data stream to be stored and the block storage nodes corresponding to the data characteristics of the data stream to be stored into a block storage list to which the service system belongs.
5. A data block storage device, the device comprising:
the receiving module is used for receiving the data stream to be stored sent by the service system;
the query module is used for querying a service system grouping directory and determining a block storage list to which the service system belongs; the service system grouping directory is used for recording the mapping relation between the service system and a block storage list to which the service system belongs and the storage address of the block storage list;
an obtaining module, configured to obtain the block storage list from a storage address of the block storage list; the block storage list is used for recording the mapping relation between the data characteristics of the data stream generated by the service system and the block storage nodes corresponding to the data characteristics of the data stream generated by the service system;
the first determining module is used for determining the data characteristics of the data stream to be stored;
a second determining module, configured to determine, according to the data feature of the data stream to be stored and the block storage list, a block storage node corresponding to the data feature;
and the first storage module is used for storing the data stream to be stored to the storage node.
6. The apparatus of claim 5, wherein the second determining module comprises:
the judging submodule is used for judging whether the use frequency of the data stream to be stored in the preset time exceeds a preset threshold value or not;
the first determining submodule is used for querying the block storage list, determining a cache level block storage node corresponding to the data characteristic and taking the cache level block storage node as the block storage node corresponding to the data characteristic if the use frequency of the data stream to be stored in a preset time exceeds a preset threshold value;
and the second determining submodule is used for querying the block storage list, determining a disk block storage node corresponding to the data characteristic and taking the disk block storage node as the block storage node corresponding to the data characteristic if the use frequency of the data stream to be stored in a preset time does not exceed a preset threshold value.
7. The apparatus of claim 5, further comprising:
a third determining module, configured to determine, when the block storage list of the service system cannot be obtained according to a service system grouping directory, a storage space required by the data stream to be stored, and determine a data feature of the data stream to be stored;
the second storage module is used for configuring corresponding block storage nodes for the data characteristics according to the storage space required by the data stream to be stored and the data characteristics of the data stream to be stored, and storing the data stream to be stored in the block storage nodes corresponding to the data characteristics;
the first building module is used for building a block storage list to which the service system belongs according to the mapping relation between the data characteristics of the data stream to be stored and the block storage nodes corresponding to the data characteristics of the data stream to be stored, and storing the block storage list;
and the first updating module is used for updating the business system, the mapping relation of the block storage list to which the business system belongs and the storage address of the block storage list into the business system grouping directory.
8. The apparatus of claim 5, further comprising:
a fourth determining module, configured to, when a block storage node corresponding to the data feature cannot be determined in the block storage list according to the data feature of the data stream to be stored, configure a corresponding block storage node for the data feature according to a storage space required by the data stream to be stored and the data feature of the data stream to be stored, and store the data stream to be stored in the block storage node corresponding to the data feature;
and the second updating module is used for storing the mapping relation between the data characteristics of the data stream to be stored and the block storage nodes corresponding to the data characteristics of the data stream to be stored to the block storage list to which the service system belongs.
9. An electronic device, comprising:
a processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute to implement a data block storage method as claimed in any one of claims 1 to 4.
10. A non-transitory computer readable storage medium, wherein instructions, when executed by a processor of an electronic device, enable the electronic device to perform a method of storing data blocks that implements any one of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911350188.1A CN111143364A (en) | 2019-12-24 | 2019-12-24 | Data block storage method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911350188.1A CN111143364A (en) | 2019-12-24 | 2019-12-24 | Data block storage method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111143364A true CN111143364A (en) | 2020-05-12 |
Family
ID=70519741
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911350188.1A Pending CN111143364A (en) | 2019-12-24 | 2019-12-24 | Data block storage method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111143364A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030028509A1 (en) * | 2001-08-06 | 2003-02-06 | Adam Sah | Storage of row-column data |
| CN105335513A (en) * | 2015-10-30 | 2016-02-17 | 迈普通信技术股份有限公司 | Distributed file system and file storage method |
| CN106331075A (en) * | 2016-08-18 | 2017-01-11 | 华为技术有限公司 | Method for storing files, metadata server and manager |
| CN107967121A (en) * | 2017-10-25 | 2018-04-27 | 华为技术有限公司 | Method for writing data and storage device |
| CN108319598A (en) * | 2017-01-16 | 2018-07-24 | 腾讯科技（北京）有限公司 | data cache method, device and system |
| CN108388599A (en) * | 2018-02-01 | 2018-08-10 | 平安科技（深圳）有限公司 | Electronic device, Data Migration and call method and storage medium |
| CN109992210A (en) * | 2019-03-29 | 2019-07-09 | 重庆紫光华山智安科技有限公司 | Date storage method, device and electronic equipment |
-
2019
- 2019-12-24 CN CN201911350188.1A patent/CN111143364A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030028509A1 (en) * | 2001-08-06 | 2003-02-06 | Adam Sah | Storage of row-column data |
| CN105335513A (en) * | 2015-10-30 | 2016-02-17 | 迈普通信技术股份有限公司 | Distributed file system and file storage method |
| CN106331075A (en) * | 2016-08-18 | 2017-01-11 | 华为技术有限公司 | Method for storing files, metadata server and manager |
| CN108319598A (en) * | 2017-01-16 | 2018-07-24 | 腾讯科技（北京）有限公司 | data cache method, device and system |
| CN107967121A (en) * | 2017-10-25 | 2018-04-27 | 华为技术有限公司 | Method for writing data and storage device |
| CN108388599A (en) * | 2018-02-01 | 2018-08-10 | 平安科技（深圳）有限公司 | Electronic device, Data Migration and call method and storage medium |
| CN109992210A (en) * | 2019-03-29 | 2019-07-09 | 重庆紫光华山智安科技有限公司 | Date storage method, device and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105933376B (en) | A kind of data manipulation method, server and storage system | |
| CN111901367A (en) | Network data analysis method and device | |
| CN108282514B (en) | Distributed service establishing method and device | |
| US11646939B2 (en) | Network function NF management method and NF management device | |
| CN112261094B (en) | Message processing method and proxy server | |
| US20180227139A1 (en) | Method of Terminal-Based Conference Load-Balancing, and Device and System Utilizing Same | |
| CN112953982B (en) | Service processing method, service configuration method and related device | |
| CN110727738B (en) | Global routing system based on data fragmentation, electronic equipment and storage medium | |
| CN105045762A (en) | Management method and apparatus for configuration file | |
| CN104796880A (en) | Subscriber identity module (SIM) card resource allocation method, related device and system | |
| CN112566154A (en) | Network entity and method for identifier allocation and/or identifier mapping for network services | |
| CN112925607A (en) | System capacity expansion and contraction method and device and electronic equipment | |
| CN113301079B (en) | Data acquisition method, system, computing device and storage medium | |
| CN111414361A (en) | Label data storage method, device, equipment and readable storage medium | |
| CN114036031B (en) | Scheduling system and method for resource service application in enterprise digital middleboxes | |
| CN109525633B (en) | Block chain network, message sending method and message receiving method based on block chain network | |
| CN111049927B (en) | File storage method and device, electronic equipment and storage medium | |
| CN111143364A (en) | Data block storage method and device, electronic equipment and storage medium | |
| CN109450798B (en) | Method for managing routing table information and computer-readable storage medium | |
| CN107948226A (en) | A kind of license management method and system | |
| CN107277126B (en) | Cloud computing resource management method and device | |
| CN112532666A (en) | Reverse proxy method, apparatus, storage medium, and device | |
| CN108023920B (en) | Data packet transmission method, equipment and application interface | |
| CN102185768A (en) | Configuration business deploying method and device | |
| CN115987990A (en) | Multi-cluster load balancing method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200512 |