CN112632303A - Distributed storage method, analysis method and device for image data - Google Patents
Distributed storage method, analysis method and device for image data Download PDFInfo
- Publication number
- CN112632303A CN112632303A CN202011604380.1A CN202011604380A CN112632303A CN 112632303 A CN112632303 A CN 112632303A CN 202011604380 A CN202011604380 A CN 202011604380A CN 112632303 A CN112632303 A CN 112632303A
- Authority
- CN
- China
- Prior art keywords
- image data
- distributed
- analysis
- stored
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003860 storage Methods 0.000 title claims abstract description 110
- 238000004458 analytical method Methods 0.000 title claims abstract description 81
- 238000000034 method Methods 0.000 title claims abstract description 48
- 230000000903 blocking effect Effects 0.000 claims abstract description 16
- 238000005520 cutting process Methods 0.000 claims description 29
- 238000012545 processing Methods 0.000 claims description 23
- 238000007405 data analysis Methods 0.000 claims description 16
- 238000013500 data storage Methods 0.000 claims description 14
- 238000010191 image analysis Methods 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 description 4
- 238000012423 maintenance Methods 0.000 description 4
- 238000007726 management method Methods 0.000 description 4
- 238000012952 Resampling Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/51—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20112—Image segmentation details
- G06T2207/20132—Image cropping
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Image Processing (AREA)
Abstract
The invention provides a distributed storage method, an analysis method and a device of image data, which are used for blocking the image data to be stored, and carrying out distributed storage on a plurality of image data blocks obtained after blocking in a distributed computing cluster, and supporting effective management on the image data stored in a distributed manner through an embedded data set. Meanwhile, by utilizing the distributed computing capability of the distributed computing cluster, the distributed analysis of the image data in the distributed computing cluster in the form of an elastic distributed data set can be realized, and the analysis performance of the image data is improved.
Description
Technical Field
The invention relates to the technical field of image data storage, in particular to a distributed storage method, an analysis method and a device for large-scale image data.
Background
With the large-scale application of telemetry technologies such as satellite remote sensing and aerial survey, the data volume of multi-source remote sensing image data from various industries is increased in a geometric mode. For image data, the data volume of several hundreds of GB or even TB level is very common, which puts a great pressure on the software system for image data management, analysis and application. In the face of massive remote sensing image data and rich application scenes, how to efficiently realize management and analysis of image data has become a key point of research in the field of remote sensing data application.
At present, in the aspect of image data storage, the image data managed by the mosaic data set is mainly in a traditional single-machine storage mode, that is, the image data managed by the mosaic data set is stored locally, and the storage requirement of massive image data cannot be met due to limited local storage idle. In addition, the calculation of the mosaic data set in the single-machine calculation mode has the problems of insufficient calculation resources and low calculation efficiency when the amount of image data is gradually increased.
Disclosure of Invention
In view of this, the present invention provides a distributed storage method, an analysis method and an apparatus for image data, so as to achieve effective storage and analysis of massive image data.
In order to achieve the above purpose, the invention provides the following specific technical scheme:
a distributed storage method of image data comprises the following steps:
under the condition of receiving an image data storage request, analyzing the image data storage request to obtain image data to be stored;
carrying out blocking processing on the image data to be stored, and carrying out distributed storage on a plurality of image data blocks obtained after the blocking processing in a distributed computing cluster;
adding storage paths of a plurality of image data blocks in the distributed computing cluster to metadata of the image data to be stored;
and creating a mosaic data set according to the metadata of the image data to be stored.
Optionally, the method further includes:
creating an image pyramid corresponding to the image data to be stored;
adding a storage path of the image pyramid to the mosaic dataset.
Optionally, the method further includes:
under the condition that the image data to be stored is a single-waveband image, setting an invalid value of the image data to be stored in the mosaic data set, and removing the invalid value in each image data block in batches in an invalid value transparent mode according to a storage path of each image data block in the mosaic data set;
and under the condition that the image data to be stored is a multiband image, setting RGB invalid values of the image data to be stored in the mosaic data set, determining a cutting area of each image data block to obtain a cutting sub data set, and reading and cutting the cutting area in each image data block in batch by adopting a cutting display mode.
Optionally, the method further includes:
setting properties of the overview, the properties of the overview including: maximum width, maximum height, resolution ratio of two adjacent layers of overview views, storage path and whether invalid values are removed or not;
constructing a summary view of the image data to be stored according to the attribute of the summary view;
adding attributes of the overview view into the mosaic dataset.
Optionally, the method further includes:
calculating a boundary subdata set in the mosaic data set according to the display range of the image data to be stored;
and under the condition that image data to be stored corresponding to the mosaic data set is newly added, deleted and updated, updating a boundary subdata set, a cutting subdata set and an outline subdata set in the mosaic data set, wherein the outline subdata set stores a storage path of the image data block.
A method of distributed analysis of image data, the method comprising:
under the condition of receiving an image data analysis request, analyzing the image data analysis request to obtain configuration parameters and an analysis mode of image data to be analyzed, wherein the image data to be analyzed is stored in a distributed computing cluster in advance according to the distributed storage method of the image data disclosed by the embodiment;
reading the mosaic data set according to the storage path of the mosaic data set of the image data to be analyzed in the configuration parameters, and acquiring the storage path of each image data block in the metadata in the mosaic data set in the distributed computing cluster;
reading each image data block from the distributed computing cluster in a form of an elastic distributed data set according to a storage path of each image data block in the distributed computing cluster;
and according to the configuration parameters, performing distributed analysis corresponding to the analysis mode on each image data block in the distributed computing cluster in the form of an elastic distributed data set to obtain an analysis result of the image data to be analyzed.
Optionally, the analysis method includes: the method comprises the steps of grid algebra operation, terrain analysis, space search, grid data processing, image analysis and grid statistics.
Optionally, the method further includes:
and carrying out local storage or distributed storage on the analysis result.
A distributed storage apparatus for image data, comprising:
the storage request analysis unit is used for analyzing the image data storage request to obtain image data to be stored under the condition of receiving the image data storage request;
the distributed storage unit is used for carrying out blocking processing on the image data to be stored and carrying out distributed storage on a plurality of image data blocks obtained after the blocking processing in a distributed computing cluster;
a storage path adding unit, configured to add a storage path of the plurality of image data blocks in the distributed computing cluster to metadata of the image data to be stored;
and the mosaic dataset creating unit is used for creating a mosaic dataset according to the metadata of the image data to be stored.
Optionally, the storage device further includes:
the image pyramid creating unit is used for creating an image pyramid corresponding to the image data to be stored; adding a storage path of the image pyramid to the mosaic dataset.
Optionally, the storage device further includes:
the single-waveband invalid value removing unit is used for setting an invalid value of the image data to be stored in the mosaic data set under the condition that the image data to be stored is a single-waveband image, and removing the invalid value in each image data block in batches in an invalid value transparent mode according to the storage path of each image data block in the mosaic data set;
and the multiband invalid value removing unit is used for setting the RGB invalid values of the image data to be stored in the mosaic data set under the condition that the image data to be stored is a multiband image, determining the cutting area of each image data block to obtain a cutting sub data set, and reading and cutting the cutting area in each image data block in batch in a cutting display mode.
Optionally, the apparatus further comprises:
the mosaic data set maintenance unit is used for calculating a boundary subdata set in the mosaic data set according to the display range of the image data to be stored; and under the condition that image data to be stored corresponding to the mosaic data set is newly added, deleted and updated, updating a boundary subdata set, a cutting subdata set and an outline subdata set in the mosaic data set, wherein the outline subdata set stores a storage path of the image data block.
An apparatus for distributed analysis of image data, the apparatus comprising:
the analysis request analysis unit is used for analyzing the image data analysis request to obtain configuration parameters and an analysis mode of the image data to be analyzed under the condition that the image data analysis request is received, wherein the image data to be analyzed is stored in the distributed computing cluster in advance according to the distributed storage method of the image data disclosed by the embodiment;
a storage path reading unit, configured to read the mosaic data set according to a storage path of the mosaic data set of the image data to be analyzed in the configuration parameters, and obtain a storage path of each image data block in metadata in the mosaic data set in the distributed computing cluster;
the image data block reading unit is used for reading each image data block from the distributed computing cluster in the form of an elastic distributed data set according to the storage path of each image data block in the distributed computing cluster;
and the image data analysis unit is used for performing distributed analysis corresponding to the analysis mode on each image data block in the distributed computing cluster in an elastic distributed data set mode according to the configuration parameters to obtain an analysis result of the image data to be analyzed.
Optionally, the analysis method includes: the method comprises the steps of grid algebra operation, terrain analysis, space search, grid data processing, image analysis and grid statistics.
Optionally, the analysis apparatus further comprises:
and the analysis result storage unit is used for locally storing or distributively storing the analysis result.
Compared with the prior art, the invention has the following beneficial effects:
the invention discloses a distributed storage method and an analysis method of image data, which are characterized in that the image data to be stored is processed in a blocking way, a plurality of image data blocks obtained after the block processing are stored in a distributed computing cluster in a distributed way, metadata of an embedded data set realizes effective management of the image data stored in the distributed way by recording the storage path of the image data blocks in the distributed computing cluster, and the storage space can be flexibly expanded due to the distributed storage of the image data in the distributed computing cluster, thereby meeting the storage requirement of mass image data. Meanwhile, by utilizing the distributed computing capability of the distributed computing cluster, the distributed analysis of the image data in the distributed computing cluster in the form of an elastic distributed data set can be realized, and the analysis performance of the image data is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a schematic flowchart illustrating a distributed storage method for image data according to an embodiment of the present invention;
fig. 2 is a schematic flowchart illustrating a distributed analysis method of image data according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a distributed storage apparatus for image data according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a distributed analysis apparatus for image data according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The embodiment of the present invention discloses a distributed storage method of image data, which can be applied to a processor in a distributed computing cluster, please refer to fig. 1, and the distributed storage method specifically includes the following steps:
s101: under the condition of receiving an image data storage request, analyzing the image data storage request to obtain image data to be stored;
s102: the method comprises the steps of carrying out blocking processing on image data to be stored, and carrying out distributed storage on a plurality of image data blocks obtained after the blocking processing in a distributed computing cluster;
the image data to be stored is processed in a blocking mode, and format conversion is carried out, namely the original image format is converted into a preset format, so that the display efficiency of the image is improved.
S103: adding storage paths of a plurality of image data blocks in a distributed computing cluster into metadata of image data to be stored;
s104: and creating a mosaic dataset according to the metadata of the image data to be stored.
When the mosaic data set is created, a coordinate system of the mosaic data set needs to be appointed for a coordinate system used during image splicing display, metadata of image data to be stored needs to be added to the mosaic data set, when the metadata of an image is added, the metadata of a plurality of image data blocks can be simultaneously selected to be added in batches, or a directory is appointed, all metadata of tif and img images under the directory are automatically added, and when the metadata is added, an shp file which is used for reading an image cutting area can be selected to limit the effective range of the image.
Further, after the image data to be stored are stored in a distributed mode, cloud optimization can be performed, and preparation is made for improving the display efficiency of the image, wherein the image cloud optimization includes creation of an image pyramid corresponding to the image data to be stored.
The image pyramid is created, the image pyramid can be created directly for the images, and the image pyramid can also be created for the managed images in batches by embedding the data sets.
When the image pyramid is created, a resampling method of the image pyramid needs to be determined, and the resampling method can be a nearest neighbor method, an average value method, a gaussian kernel method and an average joint data method.
Further, the mosaic data set can be optimized in a distributed mode, such as removing image invalid values, constructing a summary view and the like.
The invalid value in the image presents a black edge or a white edge, which seriously affects the display effect of the image. For removing the image invalid value, different methods for removing the image invalid value can be adopted according to the type of the image data to be stored, the image invalid value is removed in an invalid value transparent mode under the condition that the image data to be stored is single-waveband image data, and the image invalid value is removed in a cutting display mode under the condition that the image data to be stored is multiband image data.
Specifically, for a single-waveband image, setting an invalid value of image data to be stored in an embedded data set, and removing the invalid value in each image data block in a batch manner in an invalid value transparent manner according to a storage path of each image data block in the embedded data set.
For a multiband image, the value of an invalid value is an RGB value after multiband synthesis, when a pixel value equal to the invalid value exists in an image effective value area, the invalid value needs to be removed by using a cutting display mode, and a cutting area of each image data block is determined by setting the RGB invalid value of the image data to be stored in the mosaic data set. And in default, the clipping subdata set uses the image outline as a clipping area, and the clipping area can also be automatically constructed according to the effective area of the image data and updated to the clipping subdata set. When the single image clipping area has a vector clipping data file in a single shape format, the file clipping can be performed by reading the image clipping area and updating the image clipping area to a clipping subdata set.
After the image invalid value is removed, a overview can be constructed, the display of the spliced image under a small scale is realized, and the purpose of the global overview image is achieved.
When the overview is constructed, the attributes of the overview are required to be preset, and the attributes of the overview comprise: maximum width, maximum height, resolution ratio of two adjacent layers of overview views, storage path and whether to remove invalid values. And then constructing a summary view of the image data to be stored according to the attribute of the summary view. After the overview is constructed, a video file will be generated under the overview output path, and overview-related records, including the overview's attributes, file name, storage path, video file resolution, etc., are added to the attribute table of the outline sub-data set in the mosaic data set.
Further, after the distributed storage of the image data to be stored is completed, flexible maintenance can be performed on the mosaic data set in the distributed computing cluster, and the maintenance of the mosaic data set includes: reconstruction scope, repair mosaic dataset path, and adding and removing image files managed by the mosaic dataset.
Wherein, the reconstruction range refers to: when the image of the local range needs to be displayed, the range reconstruction can be quickly realized by using the boundary sub data set of the mosaic data set, the boundary sub data set in the mosaic data set is obtained by pre-calculating according to the display range of the image data, the display range of the mosaic data set is controlled by the boundary sub data set, and the boundary sub data set is constructed based on the outline by default.
Repairing the mosaic dataset path refers to: when the storage location of the image data changes, the user is required to re-designate the changed path, and the image file path in the contour data set can be updated.
Adding and removing the image file managed by the mosaic data set refers to: and adding a new image managed by the mosaic data set in a mode of selecting an image data file or selecting a folder, automatically updating data such as the outline, the path, the resolution and the like of the newly added image into an outline sub-data set of the mosaic data set after the image is added, and automatically updating the boundary and the cutting sub-data set. The operation of deleting part of the images is realized by editing the profile subdata set of the mosaic data set, a specific image is removed in a mode of supporting a specified profile, or a query expression is constructed, the screened image profile meeting the conditions is deleted, and the specified image is removed.
The embodiment further supports a data export function, obtains a target image meeting query conditions by screening attribute fields of the mosaic dataset attribute table, such as image acquisition time, image hierarchy, image number and the like, and obtains a target image intersecting with a query range by inputting specific numerical values to specify a spatial range, drawing a rectangular determination range or selecting an existing geometric object determination range. Then appointing a target image output path, and storing the target image file in the appointed range under the output path to realize data export.
Based on the distributed storage method of image data disclosed in the foregoing embodiment, this embodiment correspondingly discloses a distributed analysis method of image data, which is used for analyzing an embedded data set stored by using the distributed storage method of image data disclosed in the foregoing embodiment, please refer to fig. 2, and the method specifically includes the following steps:
s201: and under the condition of receiving the image data analysis request, analyzing the image data analysis request to obtain the configuration parameters and the analysis mode of the image data to be analyzed.
The configuration parameters of the image data to be analyzed at least include the storage path of the mosaic data set.
The analysis mode comprises grid algebra operation, terrain analysis, space search, grid data processing, image analysis, grid statistics and the like.
S202: and reading the mosaic data set according to the storage path of the mosaic data set of the image data to be analyzed in the configuration parameters, and acquiring the storage path of each image data block in the metadata in the mosaic data set in the distributed computing cluster.
S203: and reading each image data block from the distributed computing cluster in the form of an elastic distributed data set according to the storage path of each image data block in the distributed computing cluster.
Specifically, each image data block is read into the memory to become an elastic distributed data set.
S204: and according to the configuration parameters, performing distributed analysis corresponding to the analysis mode on each image data block in the distributed computing cluster in the form of an elastic distributed data set to obtain an analysis result of the image data to be analyzed.
In other words, distributed analysis corresponding to the analysis mode, such as grid algebra operation, terrain analysis, space search, grid data processing, image analysis, grid statistics, etc., is performed on each image data block in the form of an elastic distributed data set in the memory.
The analysis result can be stored locally or in a distributed mode, and can be written in a COG format or an avro format.
The distributed storage method and the analysis method for image data disclosed in this embodiment perform block processing on image data to be stored, perform distributed storage on a plurality of image data blocks obtained after the block processing in a distributed computing cluster, and implement effective management on the image data stored in the distributed computing cluster by storing metadata including storage paths of the image data blocks in the distributed computing cluster in an embedded data set. Meanwhile, by utilizing the distributed computing capability of the distributed computing cluster, the distributed analysis of the image data in the distributed computing cluster in the form of an elastic distributed data set can be realized, and the analysis performance of the image data is improved.
Based on the distributed storage method of image data disclosed in the foregoing embodiment, this embodiment correspondingly discloses a distributed storage device of image data, please refer to fig. 3, and the device includes:
a storage request analysis unit 101, configured to, in a case that an image data storage request is received, analyze the image data storage request to obtain image data to be stored;
the distributed storage unit 102 is configured to perform blocking processing on the image data to be stored, and perform distributed storage on a plurality of image data blocks obtained after the blocking processing in a distributed computing cluster;
a storage path adding unit 103, configured to add a storage path of the plurality of image data blocks in the distributed computing cluster to metadata of the image data to be stored;
a mosaic dataset creating unit 104, configured to create a mosaic dataset according to the metadata of the image data to be stored.
Optionally, the storage device further includes:
the image pyramid creating unit is used for creating an image pyramid corresponding to the image data to be stored; adding a storage path of the image pyramid to the mosaic dataset.
Optionally, the storage device further includes:
the single-waveband invalid value removing unit is used for setting an invalid value of the image data to be stored in the mosaic data set under the condition that the image data to be stored is a single-waveband image, and removing the invalid value in each image data block in batches in an invalid value transparent mode according to the storage path of each image data block in the mosaic data set;
and the multiband invalid value removing unit is used for setting the RGB invalid values of the image data to be stored in the mosaic data set under the condition that the image data to be stored is a multiband image, determining the cutting area of each image data block to obtain a cutting sub data set, and reading and cutting the cutting area in each image data block in batch in a cutting display mode.
Optionally, the apparatus further comprises:
the mosaic data set maintenance unit is used for calculating a boundary subdata set in the mosaic data set according to the display range of the image data to be stored; and under the condition that image data to be stored corresponding to the mosaic data set is newly added, deleted and updated, updating a boundary subdata set, a cutting subdata set and an outline subdata set in the mosaic data set, wherein the outline subdata set stores a storage path of the image data block.
Based on the distributed analysis method of image data disclosed in the foregoing embodiment, this embodiment correspondingly discloses a distributed analysis apparatus of image data, please refer to fig. 4, and the apparatus includes:
an analysis request analysis unit 201, configured to, in a case that an image data analysis request is received, analyze the image data analysis request to obtain configuration parameters and an analysis mode of image data to be analyzed, where the image data to be analyzed is stored in a distributed computing cluster in advance according to the distributed storage method for image data disclosed in the foregoing embodiment;
a storage path reading unit 202, configured to read the mosaic data set according to a storage path of the mosaic data set of the image data to be analyzed in the configuration parameters, and obtain a storage path of each image data block in metadata in the mosaic data set in the distributed computing cluster;
an image data block reading unit 203, configured to read each image data block from the distributed computing cluster in the form of an elastic distributed data set according to a storage path of each image data block in the distributed computing cluster;
and the image data analysis unit 204 is configured to perform distributed analysis corresponding to the analysis manner on each image data block in the distributed computing cluster in the form of an elastic distributed data set according to the configuration parameters, so as to obtain an analysis result of the image data to be analyzed.
Optionally, the analysis method includes: the method comprises the steps of grid algebra operation, terrain analysis, space search, grid data processing, image analysis and grid statistics.
Optionally, the analysis apparatus further comprises:
and the analysis result storage unit is used for locally storing or distributively storing the analysis result.
The distributed storage device and the analysis device for image data disclosed in this embodiment perform block processing on image data to be stored, perform distributed storage on a plurality of image data blocks obtained after the block processing in a distributed computing cluster, and implement effective management on the image data stored in the distributed computing cluster by storing metadata including storage paths of the image data blocks in the distributed computing cluster in an embedded data set. Meanwhile, by utilizing the distributed computing capability of the distributed computing cluster, the distributed analysis of the image data in the distributed computing cluster in the form of an elastic distributed data set can be realized, and the analysis performance of the image data is improved.
The above embodiments can be combined arbitrarily, and the features described in the embodiments in the present specification can be replaced or combined with each other in the above description of the disclosed embodiments, so that those skilled in the art can implement or use the present application.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A distributed storage method of image data is characterized by comprising the following steps:
under the condition of receiving an image data storage request, analyzing the image data storage request to obtain image data to be stored;
carrying out blocking processing on the image data to be stored, and carrying out distributed storage on a plurality of image data blocks obtained after the blocking processing in a distributed computing cluster;
adding storage paths of a plurality of image data blocks in the distributed computing cluster to metadata of the image data to be stored;
and creating a mosaic data set according to the metadata of the image data to be stored.
2. The method of claim 1, further comprising:
creating an image pyramid corresponding to the image data to be stored;
adding a storage path of the image pyramid to the mosaic dataset.
3. The method of claim 1, further comprising:
under the condition that the image data to be stored is a single-waveband image, setting an invalid value of the image data to be stored in the mosaic data set, and removing the invalid value in each image data block in batches in an invalid value transparent mode according to a storage path of each image data block in the mosaic data set;
and under the condition that the image data to be stored is a multiband image, setting RGB invalid values of the image data to be stored in the mosaic data set, determining a cutting area of each image data block to obtain a cutting sub data set, and reading and cutting the cutting area in each image data block in batch by adopting a cutting display mode.
4. The method of claim 3, further comprising:
setting properties of the overview, the properties of the overview including: maximum width, maximum height, resolution ratio of two adjacent layers of overview views, storage path and whether invalid values are removed or not;
constructing a summary view of the image data to be stored according to the attribute of the summary view;
adding attributes of the overview view into the mosaic dataset.
5. The method of claim 1, further comprising:
calculating a boundary subdata set in the mosaic data set according to the display range of the image data to be stored;
and under the condition that image data to be stored corresponding to the mosaic data set is newly added, deleted and updated, updating a boundary subdata set, a cutting subdata set and an outline subdata set in the mosaic data set, wherein the outline subdata set stores a storage path of the image data block.
6. A method for distributed analysis of image data, the method comprising:
under the condition of receiving an image data analysis request, analyzing the image data analysis request to obtain configuration parameters and an analysis mode of image data to be analyzed, wherein the image data to be analyzed is stored in a distributed computing cluster according to the distributed storage method of the image data in claim 1 in advance;
reading the mosaic data set according to the storage path of the mosaic data set of the image data to be analyzed in the configuration parameters, and acquiring the storage path of each image data block in the metadata in the mosaic data set in the distributed computing cluster;
reading each image data block from the distributed computing cluster in a form of an elastic distributed data set according to a storage path of each image data block in the distributed computing cluster;
and according to the configuration parameters, performing distributed analysis corresponding to the analysis mode on each image data block in the distributed computing cluster in the form of an elastic distributed data set to obtain an analysis result of the image data to be analyzed.
7. The method of claim 6, wherein the analysis mode comprises: the method comprises the steps of grid algebra operation, terrain analysis, space search, grid data processing, image analysis and grid statistics.
8. The method of claim 6, further comprising:
and carrying out local storage or distributed storage on the analysis result.
9. A distributed storage apparatus for image data, comprising:
the storage request analysis unit is used for analyzing the image data storage request to obtain image data to be stored under the condition of receiving the image data storage request;
the distributed storage unit is used for carrying out blocking processing on the image data to be stored and carrying out distributed storage on a plurality of image data blocks obtained after the blocking processing in a distributed computing cluster;
a storage path adding unit, configured to add a storage path of the plurality of image data blocks in the distributed computing cluster to metadata of the image data to be stored;
and the mosaic dataset creating unit is used for creating a mosaic dataset according to the metadata of the image data to be stored.
10. An apparatus for analyzing image data, applied to a map display engine, the apparatus comprising:
an analysis request analysis unit, configured to, in a case that an image data analysis request is received, analyze the image data analysis request to obtain configuration parameters and an analysis mode of image data to be analyzed, where the image data to be analyzed is stored in a distributed computing cluster according to the distributed storage method of image data according to claim 1 in advance;
a storage path reading unit, configured to read the mosaic data set according to a storage path of the mosaic data set of the image data to be analyzed in the configuration parameters, and obtain a storage path of each image data block in metadata in the mosaic data set in the distributed computing cluster;
the image data block reading unit is used for reading each image data block from the distributed computing cluster in the form of an elastic distributed data set according to the storage path of each image data block in the distributed computing cluster;
and the image data analysis unit is used for performing distributed analysis corresponding to the analysis mode on each image data block in the distributed computing cluster in an elastic distributed data set mode according to the configuration parameters to obtain an analysis result of the image data to be analyzed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011604380.1A CN112632303A (en) | 2020-12-30 | 2020-12-30 | Distributed storage method, analysis method and device for image data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011604380.1A CN112632303A (en) | 2020-12-30 | 2020-12-30 | Distributed storage method, analysis method and device for image data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112632303A true CN112632303A (en) | 2021-04-09 |
Family
ID=75287537
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011604380.1A Pending CN112632303A (en) | 2020-12-30 | 2020-12-30 | Distributed storage method, analysis method and device for image data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112632303A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113157214A (en) * | 2021-05-11 | 2021-07-23 | 中煤航测遥感集团有限公司 | Remote sensing image display method, device, equipment and storage medium |
| CN113852823A (en) * | 2021-11-30 | 2021-12-28 | 深圳市通恒伟创科技有限公司 | Image data uploading method, system and device based on Internet of things |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103475905A (en) * | 2013-09-25 | 2013-12-25 | 浙江大学 | Image data block efficient transmission method based on network connection multiplexing under cloud environment |
| CN105630919A (en) * | 2015-12-22 | 2016-06-01 | 曙光信息产业(北京)有限公司 | Storage method and system |
| CN108491445A (en) * | 2018-02-13 | 2018-09-04 | 北京天元创新科技有限公司 | The methods of exhibiting and system of achievement data in region |
| CN109190450A (en) * | 2018-07-09 | 2019-01-11 | 中科遥感科技集团有限公司 | Artificial intelligence remote sensing image data extraction method based on distributed computing platform |
| CN109299060A (en) * | 2018-11-22 | 2019-02-01 | 华东计算技术研究所（中国电子科技集团公司第三十二研究所） | Distributed file method and system based on image rectangular blocks |
| CN109389140A (en) * | 2017-08-14 | 2019-02-26 | 中国科学院计算技术研究所 | The method and system of quick searching cluster centre based on Spark |
| CN109657080A (en) * | 2018-09-29 | 2019-04-19 | 中国科学院上海高等研究院 | Distributed approach/the system and medium of high score satellite remote sensing date |
| CN110442444A (en) * | 2019-06-18 | 2019-11-12 | 中国科学院计算机网络信息中心 | A kind of parallel data access method and system towards mass remote sensing image |
| CN110968714A (en) * | 2019-12-23 | 2020-04-07 | 自然资源部国土卫星遥感应用中心 | Satellite remote sensing image instant service method and instant service platform |
| CN111949817A (en) * | 2020-09-09 | 2020-11-17 | 中国农业科学院农业信息研究所 | Crop information display system, method, equipment and medium based on remote sensing image |
-
2020
- 2020-12-30 CN CN202011604380.1A patent/CN112632303A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103475905A (en) * | 2013-09-25 | 2013-12-25 | 浙江大学 | Image data block efficient transmission method based on network connection multiplexing under cloud environment |
| CN105630919A (en) * | 2015-12-22 | 2016-06-01 | 曙光信息产业(北京)有限公司 | Storage method and system |
| CN109389140A (en) * | 2017-08-14 | 2019-02-26 | 中国科学院计算技术研究所 | The method and system of quick searching cluster centre based on Spark |
| CN108491445A (en) * | 2018-02-13 | 2018-09-04 | 北京天元创新科技有限公司 | The methods of exhibiting and system of achievement data in region |
| CN109190450A (en) * | 2018-07-09 | 2019-01-11 | 中科遥感科技集团有限公司 | Artificial intelligence remote sensing image data extraction method based on distributed computing platform |
| CN109657080A (en) * | 2018-09-29 | 2019-04-19 | 中国科学院上海高等研究院 | Distributed approach/the system and medium of high score satellite remote sensing date |
| CN109299060A (en) * | 2018-11-22 | 2019-02-01 | 华东计算技术研究所（中国电子科技集团公司第三十二研究所） | Distributed file method and system based on image rectangular blocks |
| CN110442444A (en) * | 2019-06-18 | 2019-11-12 | 中国科学院计算机网络信息中心 | A kind of parallel data access method and system towards mass remote sensing image |
| CN110968714A (en) * | 2019-12-23 | 2020-04-07 | 自然资源部国土卫星遥感应用中心 | Satellite remote sensing image instant service method and instant service platform |
| CN111949817A (en) * | 2020-09-09 | 2020-11-17 | 中国农业科学院农业信息研究所 | Crop information display system, method, equipment and medium based on remote sensing image |
Non-Patent Citations (2)
| Title |
|---|
| KIKITA: "为影像数据去除无效值", 《HTTPS://BLOG.CSDN.NET/KIKITAMOON/ARTICLE/DETAILS/50625033》, pages 1 - 4 * |
| 韩建平: "地理国情监测数据库管理系统设计与实现", 《地理空间信息》, pages 39 - 42 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113157214A (en) * | 2021-05-11 | 2021-07-23 | 中煤航测遥感集团有限公司 | Remote sensing image display method, device, equipment and storage medium |
| CN113157214B (en) * | 2021-05-11 | 2024-01-26 | 中煤航测遥感集团有限公司 | Remote sensing image display method, device, equipment and storage medium |
| CN113852823A (en) * | 2021-11-30 | 2021-12-28 | 深圳市通恒伟创科技有限公司 | Image data uploading method, system and device based on Internet of things |
| CN113852823B (en) * | 2021-11-30 | 2022-03-01 | 深圳市通恒伟创科技有限公司 | Image data uploading method, system and device based on Internet of things |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111723221B (en) | Mass remote sensing data processing method and system based on distributed architecture | |
| CN110442444B (en) | Massive remote sensing image-oriented parallel data access method and system | |
| CN112115198B (en) | Urban remote sensing intelligent service platform | |
| CN111862292B (en) | Data rendering method and device for transmission line corridor and computer equipment | |
| CN110647596B (en) | Map data processing method and device | |
| CN106960029B (en) | Method for extracting framing raster data across map geographical range | |
| CN111949817B (en) | Crop information display system, method, equipment and medium based on remote sensing image | |
| CN115114356B (en) | Real-time decryption method based on vector data front-end display | |
| Psomadaki | Using a space filling curve for the management of dynamic point cloud data in a relational DBMS | |
| CN112579677B (en) | Automatic processing method for satellite remote sensing image | |
| CN112632303A (en) | Distributed storage method, analysis method and device for image data | |
| US10818083B1 (en) | Pyramid generation via depth-first walk | |
| CN114820975B (en) | Three-dimensional scene simulation reconstruction system and method based on all-element parameter symbolization | |
| CN116502317B (en) | Water conservancy and hydropower engineering multisource data fusion method and terminal equipment | |
| CN103678682A (en) | Mass grid data processing and management method based on abstract templates | |
| Yan et al. | An improved coverage-oriented retrieval algorithm for large-area remote sensing data | |
| Angelo | A brief introduction to quadtrees and their applications | |
| Liu et al. | A remote sensing data management system for sea area usage management in China | |
| CN113722518A (en) | Storage method, retrieval method, equipment and medium based on remote sensing image metadata | |
| CN114677468B (en) | Model correction method, device, equipment and storage medium based on reverse modeling | |
| CN114020825A (en) | WebGIS-based coal mine underground mine map visualization method | |
| CN112966041A (en) | Data processing method, device, equipment and storage medium | |
| KR101028767B1 (en) | A Method For Updating Geographic Image Information | |
| CN110688186A (en) | Mass remote sensing image data management and rapid display method | |
| CN111179378A (en) | Global electric power meteorological atlas drawing method applicable to webGIS |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |