CN114338718B - Distributed storage method, device and medium for massive remote sensing data - Google Patents

Abstract

The invention discloses a distributed storage method, device and medium for huge amount of remote sensing data, belonging to the field of distributed storage. The invention optimizes the data division rules of the metadata of the huge amount of remote sensing data, and uses the data blocks containing multiple spatial elements as the unit of sharding, thereby realizing the relative balance of the calculation amount required by the remote sensing data in different sharding nodes, and taking into account Distributed storage of huge amounts of remote sensing data and massive computing requirements for subsequent remote sensing data. The invention can improve the stability and load balance of the distributed engine, and is of great significance to the application of the global comprehensive observation results.

Description

Distributed storage method, device and medium for massive remote sensing data

technical field

The invention belongs to the technical field of distributed storage, and in particular relates to a distributed storage method, device and medium for huge amounts of remote sensing data.

Background technique

Remote sensing data is usually stored in raster, vector and other formats. The model of raster data is relatively simple, but the amount of data is large. Therefore, most of the existing research on raster data focuses on the storage and management of raster data. In the storage of huge amounts of remote sensing data, domestic and foreign experts, scholars, research institutions and commercial companies have carried out a lot of related work. The existing remote sensing data storage methods can be mainly divided into the following five categories: remote sensing data storage based on centralized file system, remote sensing data storage based on database, network storage, file storage based on distributed file system and remote sensing based on distributed database. data storage.

Data storage based on a centralized file system refers to the centralized storage of remote sensing data on a single server, and the use of a file system to store and manage the file body of remote sensing data. The storage capacity and performance of centralized file storage depend almost entirely on a single server. Hardware performance is strongly dependent, and it is not only difficult to provide high reading speed, but also difficult to use hardware to perform data operations.

By combining relational database and geoscience data middleware, the storage of large-scale remote sensing data can be supported. However, there are some problems in using database for remote sensing data storage. For example, relational database has limited support for single large files. The storage of unstructured data also has many limitations, and the performance will drop significantly when the amount of data increases.

In order to solve the problem of the capacity limit of a single storage device, the overall storage cluster capacity can be increased by expanding nodes. The horizontal series connection of a single node through the network can integrate some nodes with relatively small capacity to become a large-scale centralized management data storage server. However, due to the different situations of individual nodes, it is difficult to centrally manage and maintain all nodes.

A distributed file system refers to a network architecture that associates a large number of spatially and physically dispersed computing nodes or storage nodes through logical relationships to form a storage relationship that is physically dispersed, logically unified, and data flows through the network. However, its data retrieval is not customized according to the characteristics of remote sensing data, and the data access delay will increase with the growth of data storage, resulting in a decrease in the overall running rate.

Distributed database architecture is a spatial data storage form that provides a unified access interface by combining relational database technology with spatial data middleware. However, since all service requests need to be processed and forwarded by the network server, the proxy server is easily blocked due to excessive network requests, thus becoming a performance bottleneck of the entire system.

In addition, the traditional distributed storage technology only considers storage and access requirements, so the fragmentation mechanism during storage is mainly based on file size load balancing. However, for remote sensing data, it often involves a large amount of computing requirements, and there is no correlation between the computing amount of remote sensing data and the file size, which leads to the imbalance of computing resources in the computing process of distributed remote sensing data. , resulting in additional performance bottlenecks.

Therefore, the present invention proposes a distributed and efficient storage method for huge amounts of remote sensing data, and constructs a storage method that meets the organization and computing requirements of huge amounts of remote sensing data.

SUMMARY OF THE INVENTION

The purpose of the present invention is to solve the problems existing in the prior art, and to provide a distributed storage method, device and medium for huge amounts of remote sensing data.

The concrete technical scheme adopted in the present invention is as follows:

In a first aspect, the present invention provides a distributed storage method for huge amounts of remote sensing data, which includes:

S1. After receiving the remote sensing data writing request submitted by the client, the routing node in the distributed database cluster reads all the spatial elements involved in the writing request to form a spatial element set;

S2. Count the total size of the ground object area of all the spatial elements in the spatial element set, and calculate the ratio of the total size of the ground object area to the total number of the data blocks according to the preset total number of data blocks, as the value of each data block Allocating area threshold;

S3, converting the geometric center point coordinates of all spatial elements in the spatial element set into spatial coding values through a spatial filling curve, and sorting all spatial elements according to the spatial coding values to form a spatial element sequence;

S4. All the empty data blocks are given unique identifiers and arranged in order, and the spatial elements in the spatial element sequence are taken out one by one in sequence and filled into the first data block that is not currently filled. When the total size of the ground object area of the spatial elements exceeds the allocated area threshold, the data block is considered to be full, and the next empty data block continues to be filled with data until all the spatial elements in the spatial element sequence are filled. Take out and fill in the data block;

S5. The routing node converts the unique identifier of each data block into a hash value through a hash algorithm, establishes a corresponding relationship between each data block and the sharding node according to the hash value, and executes writing according to the corresponding relationship operation to store each data block on the hard disk of the physical machine corresponding to the sharding node.

As a preferred option of the first aspect, the space filling curve is a Hilbert curve.

As a preferred option of the first aspect, the types of spatial elements in the spatial element set include image data and vector data, the image data includes entire images and image tiles, and the vector data includes point elements, line elements and area elements .

As a preferred aspect of the first aspect, the geometric center point of the point element is the point element itself.

As a preferred aspect of the first aspect, the preset total number of data blocks is a positive integer multiple of the total number of all shard nodes in the distributed database cluster.

As a preference of the first aspect, the hash algorithm is a consistent hash algorithm.

As a preferred aspect of the first aspect, each shard node in the distributed database cluster stores a mapping relationship table between the shard node and the physical machine hard disk, and each physical machine hard disk has multiple shard nodes, and The mapping relationship table is adjusted in real time as the number of hard disks changes.

As a preference of the above-mentioned first aspect, each database stores two primary and secondary copies at the same time based on a multi-copy consistency protocol, so as to realize read-write separation.

In a second aspect, the present invention provides a distributed storage device for massive remote sensing data, which includes a memory and a processor;

the memory for storing computer programs;

The processor is configured to, when executing the computer program, implement the distributed storage method for huge amounts of remote sensing data according to any one of the solutions in the first aspect.

In a third aspect, the present invention provides a computer-readable storage medium, where a computer program is stored on the storage medium, and when the computer program is executed by a processor, the solution described in any one of the above-mentioned first aspect can be implemented. A distributed storage method for massive remote sensing data.

Compared with the prior art, the present invention has the following beneficial effects:

The invention optimizes the data division rules of the huge amount of remote sensing data metadata, and uses the data blocks containing multiple spatial elements as the unit of sharding, thereby realizing the relative balance of the required calculation amount of remote sensing data in different sharding nodes. The invention takes into account the distributed storage of massive remote sensing data and the massive computing requirements of subsequent remote sensing data, not only satisfies the unified storage of heterogeneous remote sensing data, but also solves the hot data and load caused by concurrent access to the same system by multiple services under massive users imbalance problem. The invention can improve the stability and load balance of the distributed engine, and has great significance for the application of the global comprehensive observation results.

Description of drawings

Fig. 1 is the step flow chart of the distributed storage method for huge amount of remote sensing data;

Fig. 2 is the schematic diagram that hash algorithm performs fragmentation storage;

Figure 3 is a schematic diagram of sub-library storage.

Detailed ways

In order to make the above objects, features and advantages of the present invention more clearly understood, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined correspondingly on the premise that there is no conflict with each other.

In a preferred embodiment of the present invention, a distributed storage method for massive remote sensing data is provided, which is used for distributed storage of massive remote sensing data. Specifically, the method can be used for global comprehensive observation results. Distributed storage. The global comprehensive observation results have typical multi-source heterogeneous characteristics, including remote sensing data in various forms such as images and vectors. For example, in terms of images, the data sources include satellites such as GF1, GF2, HJ, ZY, etc. In terms of storage structure, the metadata modes adopted by different acquisition devices are also different, but their structure descriptions often include certain spatial locations. information. Therefore, the basic design idea of the distributed storage model of the global comprehensive observation results is based on the distributed database storage architecture, considering the multi-source heterogeneity, spatial relationship, time series and other characteristics of the observation data, and the design supports multiple structures, high expansion, and easy access. The realized distributed storage method provides basic distributed storage and efficient parallel query and computing services for remote sensing data of massive global comprehensive observation results.

It should be noted that the so-called "giant amount" in the present invention is only used to describe the large amount of data of remote sensing data, which can reach the level of hundreds of petabytes, but does not limit its specific data size.

In the distributed database cluster of the present invention, a routing node and a fragmentation node are included, and the routing node is responsible for distributing data to the remaining fragmented nodes according to the routing mechanism. The routing mechanism is different from traditional storage. There is no centralized metadata management, and the metadata service is no longer the performance bottleneck of the system. In this way, the problem of storage scalability can be effectively solved.

As shown in Figure 1, the steps of the distributed storage method for huge amounts of remote sensing data are described in detail below:

S1. After receiving the remote sensing data writing request submitted by the client, the routing node in the distributed database cluster reads all the spatial elements involved in the writing request to form a spatial element set.

In this embodiment, each remote sensing data writing request specifies the remote sensing data to be stored, and these remote sensing data may be composed of different types of spatial elements. Among them, the types of spatial elements include image data and vector data. Image data can be either an entire image in the form of a raster, or an image tile, depending on the type of image data to be written; in addition, vector data There are three types of point features, line features and polygon features. All spatial elements specified in the write request form a spatial element set, and subsequent routing nodes will store this spatial element set in shards according to the steps of S2 to S5.

S2. Count the total size T of the ground object area of all spatial elements in the spatial element set, and calculate the ratio V=T/M of the total size T of the ground object area to the total number of data blocks M according to the preset total number M of data blocks, as each The allocation area threshold for the data block.

It should be noted that each spatial element in the remote sensing data corresponds to a certain type of ground object during earth observation, and the field of the original data will also store the size of the ground object corresponding to each spatial element. Therefore, in this embodiment, the field value of the corresponding object area size can be directly read from the attribute field of the spatial element, and then the ground object area of all the spatial elements in the spatial element set can be summed to obtain the value of all the spatial elements. The total size of the terrain area T. However, it is particularly important to note that the size of the ground object area of the space element here can be the actual area value or a relative value converted according to the scaling ratio. When the relative value is used, the scaling of the ground object area of all spatial elements is guaranteed. The proportions are the same.

S3. For each space element in the aforementioned space element set, determine the coordinates of its geometric center point respectively, and then convert the geometric center point coordinates of all the space elements in the space element set into a space code value through a space filling curve. The elements are sorted to form a spatial element sequence O.

In this embodiment, since the spatial element has two types of remote sensing image and vector data, its geometric center point needs to be determined according to the type. For image data containing four-to-position information, the center point of the four-to-range can be used as the geometric center point; for point elements, such as global GPS data, each point can be used as its geometric center point; for line elements or area elements, Then determine the center point of the element as the geometric center point. After the geometric center point of the spatial element is determined, the spatial coding aspect can be realized by using a space filling curve such as Peano and Hilbert, preferably using a Hilbert curve.

S4. The aforementioned M data blocks are initialized and emptied in advance, all the empty data blocks are given unique identifiers and arranged in order, and the spatial elements in the aforementioned spatial element sequence O are taken out one by one in order and filled in the currently unfilled ones. In the first data block, when the total size of the ground object area of the spatial elements in a data block exceeds the allocated area threshold V, the data block is considered to be full, and the next empty data block continues to be filled with data until the spatial elements are filled. All spatial elements in sequence O are taken out and filled into M data blocks.

In order to facilitate balanced distribution, the preset total number of data blocks M is recommended to be a positive integer multiple of the total number of shard nodes in the distributed database cluster. For example, if there are 10 shard nodes in total, M can be set to three times the number of shard nodes. times, that is, M=30.

It should be noted that the aforementioned allocation area threshold V is not the maximum area of elements that can be stored in a data block, but an estimated average value. During the execution of this step, the M data blocks can be initialized to set the total area S _i of the spatial elements to be 0. After the spatial elements are filled into the data blocks one by one _, Si increases accordingly. When _i exceeds the value of V, the filling of this data block is stopped. Therefore, the final sum of the surface area S _i in a data block is slightly larger than V.

In addition, it should be noted that the spatial elements in the spatial element sequence O are taken out one by one in order and filled with M data blocks. In another form, that is, in the spatial element sequence O, if the sum of the ground object areas of the first n ₁ spatial elements is greater than V but the sum of the ground object areas of the first n ₁ -1 spatial elements is not greater than V, then The first n space elements are taken out from the space element sequence O and filled in the first data block; in the remaining space element sequence O after the first n ₁ space elements are taken out, if the sum of the ground object areas of the first n ₂ space elements If it is greater than V but the sum of the ground object areas of the first n ₂ -1 spatial elements is not greater than V, then the first n ₂ spatial elements are taken out from the spatial element sequence O and filled into the second data block; and so on. ..., until all spatial elements are taken out, the sum of the area of the objects in the last data block may not exceed V.

In the present invention, the ground object area is used as the basis for dividing the data block, mainly to take into account the calculation requirements of the remote sensing data. Because the remote sensing data actually corresponds to the ground objects in the earth observation, when the remote sensing data is calculated, the calculation amount is often related to the corresponding ground object area. The larger the ground object area, the greater the computing resources required. . Through the division rules of S4 in the present invention, all the spatial elements to be written are divided into data blocks, and the sum of the ground object areas of the spatial elements in each data block is basically uniform, which can ensure that it is distributed in the distributed After storage, the computing resource consumption of each subsequent node is relatively balanced.

S5. After completing the division of the data blocks, the routing node converts the unique identifier of each data block into a hash value through a hash algorithm, and establishes the corresponding relationship between each data block and the sharding node according to the hash value, The write operation is performed according to the corresponding relationship, and each data block is stored on the hard disk of the physical machine corresponding to the sharding node.

In the distributed storage of the present invention, the hash algorithm may be a hard hash algorithm or a consistent hash algorithm. In this embodiment, in order to facilitate dynamic adjustment of distributed storage nodes, it is recommended to use a consistent hash algorithm. The consistent hash algorithm can realize the balanced distribution of data blocks on the sharding nodes, and the specific process thereof belongs to the prior art. For ease of understanding, the process of the consistent hash algorithm is briefly described as follows: First, find the hash value of each shard node and configure it to the hash ring; then use the same method to find each data The hash value of the primary key of the block is mapped to the same hash ring; finally, the data block is searched clockwise from the location where the data block is mapped, and the data block is stored on the first shard node found, thereby establishing each The correspondence between a data block and the sharding node to be stored is subsequently stored through the mapping between the sharding node and the physical hard disk, so that the data Value i will be written to the hard disk according to the primary key Key i, as shown in Figure 2 .

In addition, the storage of huge amounts of remote sensing data also needs to solve the scalability problem of the storage layer. The storage layer can use the Distribute Hash Table (DHT) routing algorithm. Each storage node is responsible for storing a small part of the data, which is implemented based on SDHT. Addressing and storage of data throughout the system. In specific implementation, the storage layer may divide the hash space into N equal parts, each equal part is a partition node (Partition), and the N equal parts are equally divided according to the number of hard disks. For example, the default value of system N is 3600. Assuming that the current system has 50 hard disks, each hard disk carries 72 partitions. When the system is initialized, the above-mentioned mapping relationship table between the sharding node and the physical machine hard disk will be established, and the mapping relationship will be adjusted in real time as the number of hard disks in the system changes subsequently. The space required for the mapping table is very small, and each shard node in the distributed database cluster stores a mapping relationship table between the shard node and the hard disk of the physical machine for fast routing. The routing mechanism is different from traditional storage. There is no centralized metadata management, and the metadata service is no longer the performance bottleneck of the system. In this way, the problem of storage scalability can be effectively solved.

In addition, remote sensing data has the typical characteristics of multi-source heterogeneity. Therefore, in order to meet the unified storage of heterogeneous metadata, it is necessary to design a flexible data storage and organization structure. Considering that XML, JSON and other extensible data description modes are currently used for the unified exchange of metadata, this kind of data model with flexible structure is also used in the distributed metadata database to describe the metadata of the global comprehensive observation results. A binary stream is physically stored.

In addition, in order to ensure the reliability of the distributed storage of remote sensing data, the organizational structure of the distributed database needs to be designed. As a preferred implementation form of the present invention, each database needs to store two primary and backup copies at the same time based on the multi-copy consistency protocol, which are stored in different sub-databases. As shown in Figure 3, the master database (M) is used to provide metadata writing services, while the slave database (S) is mainly used to provide metadata reading services, so as to achieve read-write separation during data access and improve The concurrent service capability of the system.

Therefore, the distributed storage method for massive remote sensing data described in the above S1 to S5 can perform balanced and reliable distributed storage of multi-source and heterogeneous remote sensing data, and it can also take into account the massive computing requirements of subsequent remote sensing data. , balancing the computing resources required for data stored on different shard nodes. When the client needs to access data, it can calculate the shard node where the query result is located according to the query conditions and routing rules.

It should be noted that, according to the embodiments disclosed in the present invention, the functions of the above-mentioned memory for realizing each step can be realized by a written computer software program, and the computer program includes program codes for executing the corresponding method. Therefore, corresponding to the aforementioned distributed storage method, the present invention can further provide a corresponding distributed storage device and a medium for storing the aforementioned program code.

In another embodiment of the present invention, a distributed storage device for massive remote sensing data is also provided, which includes a memory and a processor;

the memory for storing computer programs;

The processor is configured to, when executing the computer program, implement the distributed storage method for huge amounts of remote sensing data as described in S1 to S5 above.

In another embodiment of the present invention, a computer-readable storage medium is also provided, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the above-mentioned S1 to S5 can be implemented. A distributed storage method for massive remote sensing data.

It should be noted that the above-mentioned memory may include random access memory (Random Access Memory, RAM), and may also include non-volatile memory (Non-Volatile Memory, NVM), such as at least one disk memory. The above-mentioned processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; may also be a digital signal processor (Digital Signal Processing, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. Of course, the device should also have necessary components to realize program running, such as power supply, communication bus and so on.

The above-mentioned embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. Various changes and modifications can also be made by those of ordinary skill in the relevant technical field without departing from the spirit and scope of the present invention. Therefore, all technical solutions obtained by means of equivalent replacement or equivalent transformation fall within the protection scope of the present invention.

Claims (10)

1. a distributed storage method for huge amount of remote sensing data, is characterized in that, comprises: S1. After receiving the remote sensing data writing request submitted by the client, the routing node in the distributed database cluster reads all the spatial elements involved in the writing request to form a spatial element set; S2. Count the total size of the ground object area of all the spatial elements in the spatial element set, and calculate the ratio of the total size of the ground object area to the total number of the data blocks according to the preset total number of data blocks, as the value of each data block Allocating area threshold; S3, converting the geometric center point coordinates of all spatial elements in the spatial element set into spatial coding values through a spatial filling curve, and sorting all spatial elements according to the spatial coding values to form a spatial element sequence; S4. All the empty data blocks are given unique identifiers and arranged in order, and the spatial elements in the spatial element sequence are taken out one by one in sequence and filled into the first data block that is not currently filled. When the total size of the ground object area of the spatial elements exceeds the allocated area threshold, the data block is considered to be full, and the next empty data block continues to be filled with data until all the spatial elements in the spatial element sequence are filled. Take out and fill in the data block; S5. The routing node converts the unique identifier of each data block into a hash value through a hash algorithm, establishes a corresponding relationship between each data block and the sharding node according to the hash value, and executes writing according to the corresponding relationship operation to store each data block on the hard disk of the physical machine corresponding to the sharding node. 2 . The distributed storage method for massive remote sensing data according to claim 1 , wherein the space filling curve is a Hilbert curve. 3 . 3 . The distributed storage method for massive remote sensing data according to claim 1 , wherein the spatial element types in the spatial element set include image data and vector data, and the image data includes entire images and images. 4 . Tiles, the vector data includes point features, line features, and polygon features. 4 . The distributed storage method for massive remote sensing data according to claim 3 , wherein the geometric center point of the point element is the point element itself. 5 . 5 . The distributed storage method for massive remote sensing data according to claim 1 , wherein the preset total number of data blocks is a positive integer multiple of the total number of all fragmented nodes in the distributed database cluster. 6 . 6 . The distributed storage method for massive remote sensing data according to claim 1 , wherein the hash algorithm is a consistent hash algorithm. 7 . 7. The distributed storage method for huge amount of remote sensing data as claimed in claim 1, characterized in that, each fragment node in the distributed database cluster stores the mapping relationship between the fragment node and the physical machine hard disk Each physical machine hard disk has multiple shard nodes, and the mapping relationship table is adjusted in real time as the number of hard disks changes. 8 . The distributed storage method for huge amounts of remote sensing data according to claim 1 , wherein each database stores two copies of master and backup at the same time based on a multi-copy consistency protocol to realize read-write separation. 9 . 9. A distributed storage device for huge amounts of remote sensing data, comprising a memory and a processor; the memory for storing computer programs; The processor is configured to, when executing the computer program, implement the distributed storage method for massive remote sensing data according to any one of claims 1 to 8. 10. A computer-readable storage medium, characterized in that, a computer program is stored on the storage medium, and when the computer program is executed by a processor, the computer program as described in any one of claims 1 to 8 can be implemented. A distributed storage method for remote sensing data.

Images