CN118037533A - Remote sensing image data storage method - Google Patents
Remote sensing image data storage method Download PDFInfo
- Publication number
- CN118037533A CN118037533A CN202410208454.1A CN202410208454A CN118037533A CN 118037533 A CN118037533 A CN 118037533A CN 202410208454 A CN202410208454 A CN 202410208454A CN 118037533 A CN118037533 A CN 118037533A
- Authority
- CN
- China
- Prior art keywords
- image
- remote sensing
- image data
- images
- sub
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 32
- 238000013500 data storage Methods 0.000 title claims abstract description 16
- 238000012952 Resampling Methods 0.000 claims description 12
- 230000000903 blocking effect Effects 0.000 claims description 10
- 238000007781 pre-processing Methods 0.000 claims description 8
- 238000000638 solvent extraction Methods 0.000 claims description 6
- 238000012545 processing Methods 0.000 claims description 5
- 239000013589 supplement Substances 0.000 claims description 3
- 238000010276 construction Methods 0.000 abstract description 17
- 238000005516 engineering process Methods 0.000 description 7
- 238000012937 correction Methods 0.000 description 6
- 230000000875 corresponding effect Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 6
- 238000011161 development Methods 0.000 description 4
- 230000018109 developmental process Effects 0.000 description 4
- 230000011218 segmentation Effects 0.000 description 4
- 238000007726 management method Methods 0.000 description 3
- 230000005855 radiation Effects 0.000 description 3
- 230000002596 correlated effect Effects 0.000 description 2
- 238000000354 decomposition reaction Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Landscapes
- Processing Or Creating Images (AREA)
Abstract
The invention discloses a remote sensing image data storage method, which is characterized in that from a remote sensing image pyramid construction model, in consideration of the data storage redundancy factor, a proper block size is selected according to the rule condition of image data, then a block-first and layer-last method is provided, and the construction process of the pyramid after the block is repeated recursion and mutually independent, so that sub-pyramids can be constructed in parallel, and finally the complete pyramid image is integrated.
Description
Technical Field
The invention belongs to the field of geographic information data processing, and particularly relates to a remote sensing image data storage method.
Background
In the wave of the digital age, data has become one of the important factors that drive social progress and development. The rapid development of remote sensing and computer technology makes remote sensing data trend toward diversification and complexity, presents the development trend of GB level, TB level and PB level, simultaneously provides challenges for data storage, and the application field of the remote sensing data trend toward rapid growth. In addition to the ever-increasing volume and data types of data, data storage in the space-time large data background is more in consideration of the requirements of diversified applications for the display and concurrency of data.
At present, the storage modes of remote sensing images are mainly divided into three types: (1) Expanding the relational database to enable the relational database to have the capacity of storing space data; (2) The space data engine is realized in the relational database, so that the management and processing capacity of mass image data can be realized; however, with the increasing of data volume, the traditional spatial database based on the relational database has great problems in the aspects of remote sensing image data storage, read-write performance and the like, and the relational database is stored in a regular table form, so that the storage management capability of unstructured data is weak. And in the later period, the maintenance, migration and capacity expansion costs are increased, and the technical difficulty is increased gradually. (3) a distributed storage system.
The image pyramid technology is initially applied to the fields of machine vision and image compression, and gradually becomes a common method for people to process remote sensing image storage and indexing along with the development of multi-resolution remote sensing image technology, and mainly comprises two parts of image pyramid construction and indexing. The pyramid generation technology mainly comprises a mean pyramid, a Laplacian pyramid, a Gaussian pyramid and the like, the basic ideas of the pyramid generation technology are that image data of all layers with different scales are sequentially generated through resampling based on original image data, and all the layers are stored in a plurality of slice files with the same size. The pyramid construction aims to generate images with various resolutions in advance, and a certain storage space is sacrificed to replace the reduction of display time so as to realize the rapid access of data with different specifications. The technology is mature at present and has wide application in google maps, hundred-degree maps and Goldmap. In recent years, students introduce a distributed concept into pyramid construction, and many researches are carried out on parallel pyramid construction, so that the pyramid construction efficiency is greatly improved.
The blocking is to cut the image with uniform resolution, essentially divide a remote sensing image into a plurality of small blocks for storage, and load corresponding block data according to the information such as the display area range and resolution when the image is displayed. The image slices are generally equal in width (xSize) and height (ySize), that is, square slices are typically used. The width and height dimensions of the slice should preferably be 128 pixels or more, and generally:
the image pyramid has a plurality of coding modes, and the conventional methods such as a conventional quadtree, a linear quadtree, a spatial index curve and the like are used for indexing.
The current research on pyramid layering and blocking technology mainly focuses on how to build pyramids concurrently, and then blocking is carried out. Generally, the block size is a square shape such as (256×256 or 128×128) according to requirements. The segmentation method has less consideration on unevenly distributed images, a large amount of data redundancy exists, the storage space is wasted, and the pyramid construction efficiency is affected because complete layering is performed before the segmentation.
Disclosure of Invention
In order to solve the defects in the prior art, a remote sensing image data storage method is provided, wherein the remote sensing image data storage method is firstly divided into blocks and then layered on the whole, the divided images are not affected each other when sub-pyramids are constructed, so that resampling operation among the image blocks is processed in parallel, and then a plurality of pyramids are integrated into a pyramid layered and blocked model of a complete image pyramid data set.
In order to achieve the above purpose, the technical scheme adopted by the invention is as follows:
a remote sensing image data storage method is characterized in that: the method comprises the following steps:
S1: acquiring remote sensing image data, and preprocessing the image data by using ENVI, wherein the preprocessing specifically comprises radiation correction and atmospheric correction;
S2: and (3) blocking: selecting proper block sizes according to different preprocessed image data, avoiding areas with few or no objects from entering a circulation sequence, and blocking the preprocessed same image data, namely cutting an image with uniform resolution to obtain a plurality of block data;
s3: layering: when layering is carried out on each image block, the image blocks are processed in parallel through resampling operation, a plurality of sub-pyramids are constructed, unique image coding ID is determined in a quadtree coding mode, and then the sub-pyramids are integrated into a complete image pyramid data set;
S4: taking the specific stored position of the remote sensing image as a supplementary item of metadata;
s5: creating a root directory folder for storing remote sensing images, and respectively creating root directories for storing different specific images under the folder; the pyramid layer level is used as the number of subfolders, and a plurality of folders L1, L2 and L3 are respectively created and used for respectively storing image tile data of different levels; and then the image blocks of each layer of images are respectively stored into corresponding folders according to the division of the directions of the image blocks to form a remote sensing image data file directory tree, and finally remote sensing image data storage is realized.
Furthermore, the distributed computation MapReduce is utilized for processing in each image block layering stage, so that resampling operation among the image blocks is processed in parallel.
Further, the specific method of quadtree coding is that firstly, the original image is used as a root node, and the code is 0; secondly, dividing the original remote sensing image into four sub-images with the same size, wherein the codes are 00, 01, 10 and 11 in sequence; then, each sub-image is further subdivided into four smaller sub-images with the same size, taking the sub-image coded as 00 as an example, and is subdivided into 4 sub-images, so as to obtain sub-images with the codes of 0000, 0001, 0010 and 0011 respectively; and so on, the recursive partitioning is repeated until the quadtree reaches a specified depth.
Further, metadata supplement refers to metadata addition items, and a metadata base is associated with the image data; metadata is descriptive data of image data, and custom metadata is added on the basis of metadata provided by an original image.
Compared with the prior art, the technical scheme has the following beneficial effects:
The pyramid model constructed by the method of firstly partitioning and then layering is adopted, so that the storage of unevenly distributed images is more comprehensively considered, and the storage redundancy of data is reduced; and the sub-pyramids are constructed after the segmentation, and finally the pyramid parallel construction idea of integrating the sub-pyramids into a complete image pyramid data set improves the construction efficiency.
According to the method, the differences of the original images are fully considered during blocking, the proper blocking size is selected, and the areas with few or no objects are prevented from entering a circulation sequence as much as possible, so that the redundancy of data is reduced, and the pyramid construction efficiency is improved.
The invention is used for a distributed file system, backup is automatically adopted, the safety of data can be ensured, and the whole system is displayed externally.
Drawings
Fig. 1 is a flow chart of the basic idea of the decomposition.
Fig. 2 (a) and (b) are original and modified segmentation diagrams, respectively.
FIG. 3 is a flow chart of an image storage model.
Fig. 4 is a diagram showing correspondence between coding of an image pyramid and a quadtree.
Fig. 5 is a network topology of a distributed environment.
Fig. 6 shows the time required for reading and writing of different amounts of data, wherein (a) the time required for writing (b) the time required for reading.
Fig. 7 is a diagram of image duty ratios of different block sizes.
FIG. 8 is a timing diagram of image pyramid construction for different data volumes.
Detailed Description
The invention is further described below with reference to fig. 1-8:
As shown in fig. 1-8, a method for storing remote sensing image data is characterized in that: the method comprises the following steps:
S1: acquiring remote sensing image data, and preprocessing the image data by using ENVI, wherein the preprocessing specifically comprises radiation correction and atmospheric correction;
S2: and (3) blocking: selecting proper block sizes according to different preprocessed image data, avoiding areas with few or no objects from entering a circulation sequence, and blocking the preprocessed same image data, namely cutting an image with uniform resolution to obtain a plurality of block data;
s3: layering: when layering is carried out on each image block, the image blocks are processed in parallel through resampling operation, a plurality of sub-pyramids are constructed, unique image coding ID is determined in a quadtree coding mode, and then the sub-pyramids are integrated into a complete image pyramid data set;
S4: taking the specific stored position of the remote sensing image as a supplementary item of metadata;
s5: creating a root directory folder for storing remote sensing images, and respectively creating root directories for storing different specific images under the folder; the pyramid layer level is used as the number of subfolders, and a plurality of folders L1, L2 and L3 are respectively created and used for respectively storing image tile data of different levels; and then the image blocks of each layer of images are respectively stored into corresponding folders according to the division of the directions of the image blocks to form a remote sensing image data file directory tree, and finally remote sensing image data storage is realized.
And processing each image block in the layering stage by using distributed computation MapReduce, so that resampling operation among the image blocks is processed in parallel.
The specific method of quadtree coding is that firstly, the original image is used as a root node, and is coded as 0; secondly, dividing the original remote sensing image into four sub-images with the same size, wherein the codes are 00, 01, 10 and 11 in sequence; then, each sub-image is further subdivided into four smaller sub-images with the same size, taking the sub-image coded as 00 as an example, and is subdivided into 4 sub-images, so as to obtain sub-images with the codes of 0000, 0001, 0010 and 0011 respectively; and so on, the recursive partitioning is repeated until the quadtree reaches a specified depth.
The supplement of metadata refers to metadata adding items, and a metadata base is associated with image data; metadata is descriptive data of image data, and custom metadata is added on the basis of metadata provided by an original image.
The invention provides a pyramid layering block model which is characterized in that the blocks are firstly segmented and then layered on the whole, the images of the blocks are not affected when the sub-pyramids are constructed, so that resampling operation among the image blocks is processed in parallel, and then a plurality of pyramids are generalized into a complete image pyramid data set. It should be noted that, when the block is divided, the difference of the original image is fully considered, and the appropriate size of the block is selected. And then obtaining the unique identification code of the image by using a linear quadtree in a specific mode, namely obtaining the code of the target image by overlapping the unique identification code and the layer number for multiple times according to the azimuth and the layer number of the sub-image, wherein the layer number and the code length are positively correlated.
The embodiments are described below with reference to the accompanying drawings:
(1) Acquiring current remote sensing image data;
(2) Preprocessing the image by using ENVI, wherein the preprocessing specifically comprises radiation correction and atmospheric correction;
(3) The remote sensing image pyramid model is constructed by adopting a method of firstly partitioning and then layering, and the remote sensing image pyramid model can be processed by using distributed computation MapReduce in the layering stage of each group of image blocks, so that resampling operation among the image blocks is processed in parallel, the image pyramid can be constructed by constructing a plurality of pyramids, and then the pyramids are integrated into a complete image pyramid data set to replace the construction flow of the traditional pyramid. The basic idea of decomposition is shown in fig. 1. In the process of constructing the image pyramid of 'block-first-layer-later', the sub-image block resampling stage is very suitable for MapReduce to decompose, the grouped image block group is executed on each node as Map task, map function reads the image block and the unique code of the corresponding image block, the key is the unique code of the image block, the value is the image block data, and then the resampling process of the image block is merged and generalized in the reduce stage. It should be noted that, when the blocks are segmented, the difference of the original images is fully considered, and a proper block size is selected, so that the region with few or no objects is prevented from entering the cyclic sequence as much as possible, thereby reducing the redundancy of data and improving the pyramid construction efficiency. The split map of the unevenly distributed images is shown in fig. 2; the image ratio of different block sizes is shown in fig. 7; the image pyramid construction time conditions of different data volumes are shown in fig. 8;
(4) The index mode of the image pyramid selects a linear quadtree. The four-way tree index is constructed in such a way that an original image is firstly taken as a root node and is coded as 0; secondly, dividing the original remote sensing image into four sub-images with the same size, wherein the codes are 00, 01, 10 and 11 in sequence; each sub-image is then further subdivided into four smaller sub-images of equal size. Taking a sub-image coded as 00 as an example, the sub-image is divided into 4 sub-images, so as to obtain sub-images respectively coded as 0000, 0001, 0010 and 0011; and so on, the recursive partitioning is repeated until the quadtree reaches a specified depth. The corresponding relation diagram of the coding of the image pyramid and the quadtree is shown in fig. 4. The number of layers of the pyramid of the image and the length of the code are positively correlated, and the code of the target image can be obtained by overlapping the pyramid of the image and the number of layers for multiple times according to the azimuth of the sub-image, and although the expansion capability of the quadtree index is weak, after the pyramid image dataset is constructed, too many inserting and deleting operations are hardly needed, and good expandability is also not needed.
(5) Metadata adding items, which correlate the metadata database with the image data; metadata is descriptive data of image data, such as: sensor type, time, longitude and latitude, cloud cover and other information. Custom metadata is added on the basis of metadata provided by the original image, and each item is shown in fig. 1. The metadata base is established as a core and a foundation for realizing image data management so as to support the overall management, query and search and other works of the subsequent image data. The metadata base can accurately inquire the required image data information according to the attribute, and provides mixed inquiry and single inquiry based on the image data information.
(6) In the process, a distributed file system is used, backup is automatically adopted, the safety of data can be ensured, and the whole system is externally displayed, only services are provided, and users do not need to care about an internal logic structure. The image storage model flow chart is shown in fig. 3, and the topological diagram is shown in fig. 5. After the Hadoop is initialized and smoothly started, mkdir is executed to create a file data directory tree, a root directory folder for storing remote sensing images is created, and root directories for storing different specific images are respectively created under the folder; the pyramid layer level is used as the number of subfolders, and a plurality of folders L1, L2 and L3 are respectively created and used for respectively storing image tile data of different levels; then the image blocks of each layer of images are respectively stored into the corresponding folders according to the division of the directions of the image blocks to form a remote sensing image data file directory tree, and the time required for reading and writing the image data with different data volumes is shown in fig. 6.
The method mainly starts from a construction model of the remote sensing image pyramid, considers the factor of data storage redundancy, and selects proper block sizes according to the rule condition of image data. The method of dividing into blocks and layering is put forward later, the construction process of the pyramid after dividing is repeated recursion and independent from each other, so that the sub-pyramids can be constructed in parallel, and finally the complete pyramid images are integrated.
The foregoing is a preferred embodiment of the present application, and modifications, obvious to those skilled in the art, of the various equivalent forms of the present application can be made without departing from the principles of the present application, are intended to be within the scope of the appended claims.
Claims (4)
1. A remote sensing image data storage method is characterized in that: the method comprises the following steps:
S1: acquiring remote sensing image data, and preprocessing the image data by using ENVI;
S2: and (3) blocking: and selecting proper block sizes according to different preprocessing image data, and avoiding the areas with few or no objects from entering a circulation sequence. The same preprocessed image data are segmented, namely the images with uniform resolution are cut, and a plurality of segmented data are obtained;
S3: layering: when layering is carried out on each image block, each image block is processed in parallel through resampling operation, a plurality of sub-pyramids are constructed, unique image coding ID is determined in a quadtree coding mode, and then the sub-pyramids are integrated into a complete image pyramid data set;
S4: taking the specific stored position of the remote sensing image as a supplementary item of metadata;
S5: creating a root directory folder for storing remote sensing images, and respectively creating root directories for storing different specific images under the folder; the pyramid layer level is used as the number of subfolders, and a plurality of folders L1, L2 and L3 are respectively created and used for respectively storing image tile data of different levels; then the image blocks of each layer of images respectively determine the image codes according to the division of the azimuth of each layer of images, and store the images into corresponding folders, so as to form a remote sensing image data file directory tree, and finally realize remote sensing image data storage.
2. The method of claim 1, wherein the remote sensing image data is stored by: and processing each image block in the layering stage by using distributed computation MapReduce, so that resampling operation among the image blocks is processed in parallel.
3. The method of claim 1, wherein the remote sensing image data is stored by: the specific method of quadtree coding is that firstly, the original image is used as a root node, and is coded as 0; secondly, dividing the original remote sensing image into four sub-images with the same size, wherein the codes are 00, 01, 10 and 11 in sequence; then, each sub-image is further subdivided into four smaller sub-images with the same size, taking the sub-image coded as 00 as an example, and is subdivided into 4 sub-images, so as to obtain sub-images with the codes of 0000, 0001, 0010 and 0011 respectively; and so on, the recursive partitioning is repeated until the quadtree reaches a specified depth.
4. The method of claim 1, wherein the remote sensing image data is stored by: the supplement of metadata refers to metadata adding items, and a metadata base is associated with image data; metadata is descriptive data of image data, and custom metadata is added on the basis of metadata provided by an original image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410208454.1A CN118037533A (en) | 2024-02-26 | 2024-02-26 | Remote sensing image data storage method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410208454.1A CN118037533A (en) | 2024-02-26 | 2024-02-26 | Remote sensing image data storage method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN118037533A true CN118037533A (en) | 2024-05-14 |
Family
ID=90996498
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410208454.1A Pending CN118037533A (en) | 2024-02-26 | 2024-02-26 | Remote sensing image data storage method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118037533A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103455624A (en) * | 2013-09-16 | 2013-12-18 | 湖北文理学院 | Implement method of lightweight-class global multi-dimensional remote-sensing image network map service |
| CN103491185A (en) * | 2013-09-25 | 2014-01-01 | 浙江大学 | Remote sensing data cloud storage method based on image block organization |
| CN103809969A (en) * | 2014-01-15 | 2014-05-21 | 中国公路工程咨询集团有限公司 | Remote-sensing image data parallel resampling method based on pre-fragmentation in cloud environment |
| US20220121688A1 (en) * | 2019-06-18 | 2022-04-21 | Computer Network Information Center, Chinese Academy Of Sciences | Parallel data access method and system for massive remote-sensing images |
-
2024
- 2024-02-26 CN CN202410208454.1A patent/CN118037533A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103455624A (en) * | 2013-09-16 | 2013-12-18 | 湖北文理学院 | Implement method of lightweight-class global multi-dimensional remote-sensing image network map service |
| CN103491185A (en) * | 2013-09-25 | 2014-01-01 | 浙江大学 | Remote sensing data cloud storage method based on image block organization |
| CN103809969A (en) * | 2014-01-15 | 2014-05-21 | 中国公路工程咨询集团有限公司 | Remote-sensing image data parallel resampling method based on pre-fragmentation in cloud environment |
| US20220121688A1 (en) * | 2019-06-18 | 2022-04-21 | Computer Network Information Center, Chinese Academy Of Sciences | Parallel data access method and system for massive remote-sensing images |
Non-Patent Citations (2)
| Title |
|---|
| 彭望琭，周冠华，江澄编著: "中国遥感卫星应用技术 上", 31 October 2021, 北京：中国宇航出版社, pages: 76 - 78 * |
| 陈时远: "基于HDFS的分布式海量遥感影像数据存储技术研究", 中国优秀硕士学位论文全文数据库, 15 August 2014 (2014-08-15), pages 25 - 42 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109992636B (en) | Space-time coding method, space-time index and query method and device | |
| CN109284338B (en) | Satellite remote sensing big data optimization query method based on mixed index | |
| Arge et al. | Scalable sweeping-based spatial join | |
| Pfoser et al. | Novel approaches to the indexing of moving object trajectories. | |
| CN110781325A (en) | High-resolution remote sensing data grid refined management model and construction method thereof | |
| CN106933833B (en) | Method for quickly querying position information based on spatial index technology | |
| CN109492060A (en) | A kind of map tile storage method based on MBTiles | |
| CN108804602A (en) | A kind of distributed spatial data storage computational methods based on SPARK | |
| CN105786942A (en) | Geographic information storage system based on cloud platform | |
| CN105608222A (en) | Rapid building method of tile pyramid for large-scale raster data set | |
| CN102376160A (en) | Method and system for updating real-time traffic information | |
| CN111639075B (en) | Non-relational database vector data management method based on flattened R tree | |
| van Oosterom | Spatial access methods | |
| CN101299213A (en) | N-dimension clustering order recording tree space index method | |
| CN116860905B (en) | Space unit coding generation method of city information model | |
| CN112395288B (en) | R-tree index merging and updating method, device and medium based on Hilbert curve | |
| CN112131333A (en) | Tile map storage method based on oracle data file | |
| Baumann et al. | Putting pixels in place: A storage layout language for scientific data | |
| CN114820975A (en) | Three-dimensional scene simulation reconstruction system and method based on all-element parameter symbolization | |
| Faloutsos et al. | Analysis of n-dimensional quadtrees using the Hausdorff fractal dimension | |
| Angelo | A brief introduction to quadtrees and their applications | |
| CN118037533A (en) | Remote sensing image data storage method | |
| Ogayar-Anguita et al. | Nested spatial data structures for optimal indexing of LiDAR data | |
| CN113344943B (en) | Mosaic encoding method for tile fragments of remote sensing image | |
| Yang et al. | Managing spatial objects with the VMO-Tree |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |