CN115934794B - Elastic management method for massive multi-source heterogeneous remote sensing space data query - Google Patents
Elastic management method for massive multi-source heterogeneous remote sensing space data query Download PDFInfo
- Publication number
- CN115934794B CN115934794B CN202211520789.4A CN202211520789A CN115934794B CN 115934794 B CN115934794 B CN 115934794B CN 202211520789 A CN202211520789 A CN 202211520789A CN 115934794 B CN115934794 B CN 115934794B
- Authority
- CN
- China
- Prior art keywords
- index
- data
- elastic
- weight
- hard disk
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000007726 management method Methods 0.000 title claims abstract description 63
- 238000000034 method Methods 0.000 claims abstract description 48
- 238000010276 construction Methods 0.000 claims description 9
- 230000005012 migration Effects 0.000 claims description 9
- 238000005457 optimization Methods 0.000 claims description 4
- 230000004044 response Effects 0.000 claims description 4
- 230000005540 biological transmission Effects 0.000 claims description 3
- 238000004458 analytical method Methods 0.000 abstract description 6
- 238000004364 calculation method Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000013523 data management Methods 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 238000013500 data storage Methods 0.000 description 2
- 230000007423 decrease Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000013508 migration Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000000926 separation method Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000010387 memory retrieval Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 235000019633 pungent taste Nutrition 0.000 description 1
- 239000000344 soap Substances 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides an elastic management method for massive multi-source heterogeneous remote sensing space data query, which is applied to the technical field of computer big data, and the method is based on a distributed architecture of an elastic search and constructs a data running environment; then constructing an elastic management model according to an elastic environment building method, an elastic index building method, an elastic weight building method and a hierarchical elastic storage method; and finally, realizing the elastic management of remote sensing space data query in the data operation environment according to the elastic management model. The invention realizes the elastic management of hundred million-level or more space data based on the elastic search distributed search and analysis engine and the elastic management model, improves the statistical efficiency of data query, and can lead a user to quickly and accurately return the query result in milliseconds when carrying out data query service.
Description
Technical Field
The invention relates to the technical field of big data of computers, in particular to an elastic management method for inquiring massive multi-source heterogeneous remote sensing space data.
Background
In recent years, the observation capability of remote sensing satellites at home and abroad is continuously improved, satellite remote sensing and space information service step into a rapid development period, but as the types of remote sensing satellite data are more and more, the resolution is higher and the data volume is larger, the method provides serious challenges for the isomerism of massive multi-source isomerism satellite data forms, the management flexibility and the like. In order to obtain effective application service from massive multi-source heterogeneous satellite data, a satellite data cataloging system is often required to be constructed, unified cataloging data management is achieved by adopting a space query technology of a geographic information system, satellite data of various domestic optical loads, radar loads, electromagnetic loads, atmospheric loads and the like are accessed in a normalized mode, and finally data cross-platform satellite data management, online data processing service, remote sensing interpretation functions and even entity data distribution are achieved.
The prior art mainly has the following problems: firstly, in order to facilitate management and improve efficiency, only unified cataloging is carried out on data, and differentiated storage and personalized display cannot be carried out; secondly, based on the distributed cluster management technology, the query efficiency and the hard service are improved simply through hardware expansion, and the computing acceleration capacity and the soft service capacity of the platform system kernel are improved without establishing an elastic management strategy.
Disclosure of Invention
The invention provides an elastic management method for massive multi-source heterogeneous remote sensing space data query, which is based on E L AST I CSEARCH distributed search and analysis engines, comprehensively considers the integration fusion of hardware-software-algorithm, builds an elastic read-write separation environment, builds an elastic management model, forms an elastic data management algorithm, automatically and effectively selects the shortest path when a user performs data query service, and rapidly and accurately returns the result in the shortest time.
In a first aspect, an embodiment of the present invention provides a method for elastically managing a massive multi-source heterogeneous remote sensing spatial data query, where the method includes:
Constructing a data running environment based on E L AST I CSEARCH distributed architecture;
Constructing an elastic management model according to an elastic environment building method, an elastic index building method, an elastic weight building method and a hierarchical elastic storage method;
And realizing the elastic management of remote sensing space data query in a data operation environment according to the elastic management model.
According to the technical scheme, based on E L AST I CSEARCH distributed search and analysis engines and an elastic management model, elastic management of hundred million-level or more space data is achieved, data query statistical efficiency is improved, and a user can quickly and accurately return a query result in milliseconds when performing data query service.
Optionally, the elastic environment establishment method includes:
When the concurrent access quantity of the service node is greater than or equal to the access threshold value of the service node, automatically expanding the service node;
Wherein the serving node access threshold = number of concurrency/average response time;
and when the stored data quantity of the service node is greater than or equal to the data threshold value, automatically expanding the service node.
Optionally, the elastic index establishing method includes:
establishing a three-level index mode according to the business rule and the index rule;
Establishing a sequential inclusion relationship by taking a service business name as an index alias and a satellite data source and a time phase as index names;
the service provider is a first-level index, the satellite data source index is a second-level index, and the time phase index is a third-level index.
According to the technical scheme, three-level data systemization management of 'service provider + satellite data source (resolution) +time phase' is set according to the business rule and the index rule, so that the data query efficiency can be improved better.
Optionally, the elastic weight construction method includes:
determining the weight value of the index by adopting a weight adjustment algorithm according to the freshness, resolution and access quantity of different satellites;
and establishing and storing the relation between the index and the weight value according to the weight value.
Optionally, the elastic weight construction method further includes:
dividing a plurality of weight levels according to a ladder algorithm, wherein each weight level is divided into a plurality of levels, and each level is provided with a corresponding access number increment value and a highest access number critical value;
and updating the weight value of the index according to the access amount and the stepwise algorithm.
According to the technical scheme, the weight adjustment algorithm and the dynamic optimization method are applied, so that the data query efficiency can be improved better.
Optionally, the hierarchical elastic storage method includes:
Dividing a storage area into a hot area, a warm area and a cold area;
the hot area adopts a high-speed memory, the warm area adopts an SSD high-speed hard disk, and the cold area adopts a common SATA hard disk.
Optionally, the hierarchical elastic storage method further comprises:
storing the data to an SSD high-speed hard disk and a common SATA hard disk according to the weight level;
The data transmission between the SSD high-speed hard disk and the common SATA hard disk is realized through a snapshot migration program.
Through the technical scheme, the shortest path can be effectively and automatically selected and queried, and the retrieval efficiency is accelerated by preferably high-speed memory retrieval. The memory mechanism of the hot area, the warm area and the cold area can fix the use amount of the memory of the hot area and the use amount of the high-speed hard disk, and the data which are not commonly used are migrated to the low-speed hard disk through the dynamic migration mechanism, so that the hardware resources are saved.
Optionally, the elasticity management method further includes:
Establishing a filter, and recording the index access times of a user on the same day;
The index times are used for expansion optimization of the elastic weight construction method, the hierarchical elastic storage method and the service nodes.
Optionally, the elasticity management method further includes:
and determining a query result according to satellite data sources, space and time conditions queried by the user and combining an elastic management model.
The invention provides an elastic management method for massive multi-source heterogeneous remote sensing space data query, which is based on E L AST I CSEARCH distributed architecture and constructs a data running environment; then constructing an elastic management model according to an elastic environment building method, an elastic index building method, an elastic weight building method and a hierarchical elastic storage method; and finally, realizing the elastic management of remote sensing space data query in the data operation environment according to the elastic management model. The invention realizes the elastic management of hundred million-level even more space data based on E L AST I CSEARCH distributed search and analysis engines and an elastic management model, improves the statistical efficiency of data query, and can lead a user to quickly and accurately return a query result in milliseconds when carrying out data query service.
It should be understood that the description in this summary is not intended to limit the critical or essential features of the embodiments of the invention, nor is it intended to limit the scope of the invention. Other features of the present invention will become apparent from the description that follows.
Drawings
The above and other features, advantages and aspects of embodiments of the present invention will become more apparent by reference to the following detailed description when taken in conjunction with the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements.
FIG. 1 is a flow chart of an elastic management method for querying massive multi-source heterogeneous remote sensing space data according to an embodiment of the invention;
FIG. 2 is a schematic diagram of an elastic index according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of an elastic storage device according to an embodiment of the present invention.
Detailed Description
In order to enable a person skilled in the art to better understand the technical solutions in one or more embodiments of the present specification, the technical solutions in one or more embodiments of the present specification will be clearly and completely described below with reference to the drawings in one or more embodiments of the present specification, and it is obvious that the described embodiments are only some embodiments of the present specification, not all embodiments. All other embodiments, which can be made by one or more embodiments of the present disclosure without inventive faculty, are intended to be within the scope of the present disclosure.
It should be noted that, the description of the embodiment of the present invention is only for the purpose of more clearly describing the technical solution of the embodiment of the present invention, and does not constitute a limitation on the technical solution provided by the embodiment of the present invention.
Fig. 1 is a flowchart of an elastic management method for querying massive multi-source heterogeneous remote sensing space data according to an embodiment of the invention. As shown in fig. 1, includes:
s101, constructing a data running environment based on E L AST I CSEARCH distributed architecture.
Optionally, constructing an operation environment, and constructing E L AST I CSEARCH containerized read-write separation cluster environments on different machines; the hardware adopts a 1+1+N architecture, and at least two machines are provided, one machine is used for data storage and index generation of a write library, the other machine is used for data retrieval and query service, and the other machines can be dynamically expanded according to service requirements and performance requirements to form a write-read-many cluster.
Optionally, according to the cataloging acquisition mode provided by the remote sensing satellite service provider, constructing various data acquisition strategies such as interfaces, files, SOAP services and the like, forming a satellite cataloging management standard, and realizing unified analysis, unified processing, differentiated management and differentiated storage of the multi-source heterogeneous catalogs.
Optionally, a file sharing catalog for the system is set, a snapshot warehouse is built, snapshot generation and snapshot synchronization are realized, full-quantity and increment index snapshot generation and synchronization are supported, full-quantity indexes are exported from a write library at one time and synchronized to a read library cluster, and the elasticity of an operation environment is expandable and highly reliable.
S102, an elasticity management model is built according to an elasticity environment building method, an elasticity index building method, an elasticity weight building method and a hierarchical elasticity storage method.
Optionally, the elastic environment establishment method includes:
When the concurrent access quantity of the service node is greater than or equal to the access threshold value of the service node, automatically expanding the service node;
Wherein the serving node access threshold = number of concurrency/average response time;
Illustratively, access performance metrics are monitored, triggering service node extensions in real-time. Based on the existing facility environment, a wrk testing method is applied to measure the concurrent access quantity of each node, the concurrent access quantity is recorded as QPS_A, QPS (TPS) is used as a service node access threshold, and in real-time monitoring, if QPS_A > = QPS (TPS), the service node is automatically expanded.
And when the stored data quantity of the service node is greater than or equal to the data threshold value, automatically expanding the service node.
Illustratively, based on the existing facility environment, confirming the disk space limit value X of the server and the disk space occupied value of the data volume, setting 30G based on the limit condition of E L AST I CSEARCH official guide index of 20GB to 40GB for each slice, and guiding the memory setting rule of E L AST I CSEARCH JVM, setting the normal memory to be not more than 32G, suggesting the number of slices of the index to be sum (i ndex)/32+1 slices, etc., determining the size of the data volume stored in each service node, and automatically expanding the node when (sum (i ndex)/32+1) ×30g > =x.
Optionally, the elastic index establishing method includes:
establishing a three-level index mode according to the business rule and the index rule;
Establishing a sequential inclusion relationship by taking a service business name as an index alias and a satellite data source and a time phase as index names;
the service provider is a first-level index, the satellite data source index is a second-level index, and the time phase index is a third-level index.
Illustratively, as shown in FIG. 2:
According to the business rule and the index rule, a three-level data query index mode of 'service provider + satellite data source (resolution) +time phase' is set. The satellite service provider is taken as an index alias, satellites (resolution) and time phases are taken as index names, and a sequential containing relation is established, so that the index alias contains all data, the satellite data source (resolution) index contains single satellite data source (resolution) data, and the time phase index contains specific year data under a certain satellite data source. And (5) quickly positioning the index and inquiring the data according to the index alias and the regular index characteristic during inquiring. The data is stored widely to finely and widely to small.
Optionally, the elastic weight construction method includes:
determining the weight value of the index by adopting a weight adjustment algorithm according to the freshness, resolution and access quantity of different satellites;
and establishing and storing the relation between the index and the weight value according to the weight value.
Dividing a plurality of weight levels according to a ladder algorithm, wherein each weight level is divided into a plurality of levels, and each level is provided with a corresponding access number increment value and a highest access number critical value;
and updating the weight value of the index according to the access amount and the stepwise algorithm.
Illustratively, the relevant data for the ladder algorithm application are shown in table 1 below:
TABLE 1
Illustratively, the stepwise algorithm includes a slow rise: the weight class is divided into 10 levels, n levels are subdivided in each level, the maximum access number of each level is 10 n < -1 >, the initial all index weight value is 0, when the index is accessed and hit, the index access number is increased by 1, when the access number reaches above the highest weight critical value, the weight level is increased by one level, the base numbers of the second level to the n < th > level are increased by 10, and the like. The weight hierarchy weights of the index are defined, and the hierarchy weight threshold formula is as follows:
In the present embodiment, n is 10, but is not limited to 10, and may be another positive integer.
Illustratively, the ladder algorithm further includes a fast ramp down: setting up the calculation once a day, and automatically subtracting n from all index levels. For example: from level 6 to level 5, the access to the index is reduced to the corresponding level and level threshold as well. The number of accesses that need to be added per upgrade increases with the level, and the higher the level, the more index energy decreases per level decrease.
Illustratively, the ladder algorithm further comprises: and (3) calculating once per day, comparing the access quantity with the weight critical value, and setting the weight grade as the grade corresponding to the current critical value if the access quantity < = the weight critical value.
Optionally, dividing the storage area into a hot zone, a warm zone and a cold zone; the hot area adopts a high-speed memory, the warm area adopts an SSD high-speed hard disk, and the cold area adopts a common SATA hard disk.
Storing the data to an SSD high-speed hard disk and a common SATA hard disk according to the weight level;
the data transmission between the SSD high-speed hard disk and the common SATA hard disk is realized through a snapshot migration program.
Illustratively, as shown in fig. 3: a storage mechanism of a hot area, a warm area and a cold area is established, the hot area adopts a high-speed memory, the warm area adopts an SSD high-speed hard disk, and the cold area adopts a common SATA hard disk. And then, according to the calculation result of the elastic weight value, establishing an index hotness access recording rule, storing hot spot cataloging data in a memory or an SSD hard disk according to different use amounts, and storing unusual data in a low-configuration hard disk, so that the storage cost can be reduced, and the storage performance can be optimized.
The hot zone data is judged according to the access amount of the index, the satellite data source index with the largest weight is initially set as the hot zone data according to the index weight ranking, the index data is pre-indexed at regular time per hour, the index data is queried, the data is preheated to a high-speed memory, and the data access is accelerated.
And judging the data in the warm area and the cold area according to the set disk space, setting 80% of the space of the SSD hard disk as a hot spot storage area, executing snapshot migration every day at regular time, when the index data size does not exceed the hot spot storage area of the SSD hard disk, completely dropping the SSD hard disk, and when the index data size exceeds the space limit, migrating the data exceeding 80% of the space of the SSD hard disk to the SATA hard disk through a snapshot migration program according to the weight level. The formula is as follows: index data size > SSD hard disk size 0.8.
When the elastic management model is used, one or more of an elastic environment establishment method, an elastic index establishment method, an elastic weight establishment method and a hierarchical elastic storage method are called to perform elastic management of data query.
S103, realizing the elastic management of remote sensing space data query in a data operation environment according to the elastic management model.
Optionally, the elasticity management method further includes:
Establishing a filter, and recording the index access times of a user on the same day;
The index times are used for expansion optimization of the elastic weight construction method, the hierarchical elastic storage method and the service nodes.
Optionally, the elasticity management method further includes:
and determining a query result according to satellite data sources, space and time conditions queried by the user and combining an elastic management model.
By way of example, according to satellite data sources (resolution), space and time conditions of a user, and combining index weights and a three-level storage strategy, a corresponding index range is selected, a high-order weight index is preferentially searched, high-speed memory or high-speed hard disk data is automatically selected, and a data query result can be quickly obtained.
The advantageous effects of the present invention will be described in the following with a preferred embodiment;
Illustratively, a clustered environment is built.
Environmental cluster: three virtual servers are applied, and all the virtual servers are configured into 8 cores, 16G memory and 200G hard disk. And (3) containerizing three server bases, constructing a cluster environment, wherein one server is used as a database writing service for data importing and indexing to generate a database writing system, and the other two servers are used as database reading clusters for retrieval service.
Shared directory: in order to prevent the side-reading writing from affecting the data acquisition efficiency, a file directory which can be shared with each other is set for three servers and is used for storing the snapshot of the shared write library, the write library and the read library data are synchronized through the snapshot technology, and the AP I of E L AST I CSEARCH is called in the shared directory to generate a corresponding snapshot warehouse.
And (3) data storage: and normalizing Beijing series data and sentinel series data, and storing the normalized Beijing series data and the sentinel series data into a writing library.
Snapshot program: and constructing a snapshot synchronization program, realizing snapshot generation, backup and snapshot synchronization programs, realizing the time delay synchronization of a specific index to a retrieval special system, and realizing the real-time synchronization of a specified index to a special system function. And providing a full-quantity and increment index snapshot synchronization function, leading out the full-quantity index from the write library at one time, generating a snapshot, and synchronizing to the read library cluster.
And (3) incremental indexing, namely naming the snapshot by an index name and a time stamp, synchronizing incremental data, and merging indexes after synchronization is completed.
And (3) synchronizing data, namely cataloging Beijing series satellite data sources of companies, slicing catalogs and catalogs of public welfare sentry series satellites, and synchronizing the catalogs to a reading cluster node through a data synchronization snapshot program.
Illustratively, an elasticity management model is constructed.
And (3) constructing an elastic expansion program, realizing cluster elastic expansion, obtaining a single node QPS as 3000 by an access amount calculation method QPS (TPS) =concurrency number/average response time based on the current facility environment and service requirements, obtaining a single machine storage amount as 180G by applying a data amount calculation method (sum (i ndex)/32+1) 30G, and automatically expanding service nodes when the data amount X is more than 180G when the concurrency access amount QPS_A is more than 3000.
An elastic index strategy is constructed, index elastic construction is realized, a multi-source heterogeneous spatial cataloging index template is constructed to be ds_product_image.json, the index is used for storing indexes containing all spatial cataloging data, the indexes at least contain spatial range (geo_shp), data I D (key word), cloud cover (f l oat), time phase (date), track number (text), satellite data source (text), side pendulum (f l oat), sensor (text), resolution (f l oat), adding time (date) and other spatial cataloging attributes, and the application indexes are stored as Beijing series data and sentinel series data, and the index creation rules are as follows:
ds_product_ imagery: all index data is contained;
ds_product_ imagery _bj: all data including Beijing series;
ds_product_ imagery _send_i ne l: all data comprising the sentinel series;
ds_product_ imagery _bj_bj2: all data including Beijing No. 2;
ds_product_ imagery _bjj2_2021: all data from year 2021 of beijing No. 2 are included.
The method comprises the steps of constructing elastic weight, setting index 10 grade weight according to freshness and called times of index data based on a weight adjustment program, defining weight value grades according to index access numbers of different spatial data sources, adjusting a query queue when service is applied, preferentially searching indexes with larger weight according to index weight values of the spatial data when the data volume is overlarge, and achieving the effect of accelerating positioning index.
Elastic storage: based on a dynamic optimization program, the index with the best pre-index weight is calculated once a day to a cache, the index is calculated once a week, an index migration program is constructed according to weight grade division, three grades of 8,9 and 10 are migrated to an SSD hard disk through a snapshot migration program index, unusual data are placed into a low-configuration hard disk for storage, dynamic pre-caching is needed to be accessed, and accelerated access of hot zone data is guaranteed.
Illustratively, application of data queries.
Providing a service interface, supporting a user to acquire a data result based on conditions such as a pre-generated space or a custom space-time, providing a data query interface, and supporting the query and calculation of combined conditions such as the space-time. The query application selects a corresponding index range according to satellite data sources (resolution), space and time conditions of a user and combining index weights and a three-level storage strategy, preferentially retrieves the high-order weight index and automatically selects high-speed memory or high-speed hard disk data. And quickly acquiring a data query result.
The embodiment of the invention provides an elastic management method for inquiring massive multi-source heterogeneous remote sensing space data, which is based on E L AST I CSEARCH distributed architecture and constructs a data running environment; then constructing an elastic management model according to an elastic environment building method, an elastic index building method, an elastic weight building method and a hierarchical elastic storage method; and finally, realizing the elastic management of remote sensing space data query in the data operation environment according to the elastic management model. The invention realizes the elastic management of hundred million-level even more space data based on E L AST I CSEARCH distributed search and analysis engines and an elastic management model, improves the statistical efficiency of data query, and can lead a user to quickly and accurately return a query result in milliseconds when carrying out data query service.
The above description is only illustrative of the preferred embodiments of the present invention and of the principles of the technology employed. It will be appreciated by persons skilled in the art that the scope of the invention referred to in the present invention is not limited to the specific combinations of the technical features described above, but also covers other technical features formed by any combination of the technical features described above or their equivalents without departing from the inventive concept. Such as the above-mentioned features and the technical features disclosed in the present invention (but not limited to) having similar functions are replaced with each other.
Claims (6)
1. The elastic management method for querying massive multi-source heterogeneous remote sensing space data is characterized by comprising the following steps of:
constructing a data running environment based on the distributed architecture of the elastiscearch;
Constructing an elastic management model according to an elastic environment building method, an elastic index building method, an elastic weight building method and a hierarchical elastic storage method;
According to the elastic management model, realizing the elastic management of remote sensing space data inquiry in the data operation environment;
the elastic index establishing method comprises the following steps:
establishing a three-level index mode according to the business rule and the index rule;
Establishing a sequential inclusion relationship by taking a service business name as an index alias and a satellite data source and a time phase as index names;
The service provider name index is a primary index, the satellite data source index is a secondary index, and the time phase index is a tertiary index;
the elastic weight construction method comprises the following steps:
determining the weight value of the index by adopting a weight adjustment algorithm according to the freshness, resolution and access quantity of different satellites;
Establishing and storing a relation between an index and the weight value according to the weight value;
the elastic weight construction method further comprises the following steps:
dividing a plurality of weight levels according to a ladder algorithm, wherein each weight level is divided into a plurality of levels, and each level is provided with a corresponding access number increment value and a highest access number critical value;
Updating the weight value of the index according to the access quantity and the ladder algorithm;
the updating the weight value of the index according to the access quantity and the ladder algorithm comprises the following steps:
When the index is accessed and hit, the access amount of the index is increased by 1, and if the access amount reaches above the highest access amount critical value corresponding to the level where the index is located, the weight level of the index is increased by one level;
after each preset time interval threshold value, reducing the levels of all indexes by preset level values at regular time, and reducing the corresponding access amount to the access amount highest critical value of the corresponding level after the reduction;
The implementation of the elastic management of satellite data query in the data running environment according to the elastic management model includes:
According to satellite data sources, space and time conditions of a user, an index weight and a three-level storage strategy are combined, a corresponding index range is selected, a high-order weight index is preferentially searched, and high-speed memory or high-speed hard disk data is automatically selected so as to quickly acquire a data query result.
2. The elasticity management method according to claim 1, wherein the elasticity environment establishment method includes:
When the concurrent access quantity of the service node is greater than or equal to the access threshold value of the service node, automatically expanding the service node; the serving node access threshold = number of concurrency/average response time;
When the stored data quantity of the service node is more than or equal to a data threshold value, automatically expanding the service node;
and when the concurrent access quantity of the service node is greater than or equal to the access threshold value of the service node, automatically expanding the service node, including:
Based on the constructed data running environment, applying wrk method to measure the concurrent access quantity of each service node, and marking as QPS_A, if QPS_A > =QPS (TPS), automatically expanding the service node; wherein QPS (TPS) is the serving node access threshold.
3. The elasticity management method of claim 1, wherein the hierarchical elasticity storage method includes:
Dividing a storage area into a hot area, a warm area and a cold area; the hot area adopts a high-speed memory, the warm area adopts an SSD high-speed hard disk, and the cold area adopts a common SATA hard disk.
4. The elasticity management method of claim 3, wherein the hierarchical elasticity storage method further comprises:
storing data to the SSD high-speed hard disk and the common SATA hard disk according to the weight level;
The data transmission between the SSD high-speed hard disk and the common SATA hard disk is realized through a snapshot migration program;
The storing data to the SSD high speed hard disk and the normal SATA hard disk according to the weight level includes:
Taking the satellite data source index with the largest index weight as hot zone data, pre-indexing at regular time per hour, inquiring index data, and preheating the data into an SSD high-speed hard disk;
taking 80% space of the SSD hard disk as a hot spot storage area, when the index data size does not exceed the hot spot storage area of the SSD hard disk, completely dropping the SSD hard disk, and when the index data size exceeds the space limit, migrating the data exceeding 80% space of the SSD hard disk to the SATA hard disk through a snapshot migration program according to the weight level.
5. The elasticity management method of claim 1, wherein the elasticity management method further comprises:
Establishing a filter, and recording the index access times of a user on the same day; the index times are used for expansion optimization of the elastic weight construction method, the hierarchical elastic storage method and the service nodes.
6. The elasticity management method of claim 1, wherein the elasticity management method further comprises:
and determining a query result according to satellite data sources, space and time conditions queried by the user and combining the elastic management model.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211520789.4A CN115934794B (en) | 2022-11-30 | 2022-11-30 | Elastic management method for massive multi-source heterogeneous remote sensing space data query |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211520789.4A CN115934794B (en) | 2022-11-30 | 2022-11-30 | Elastic management method for massive multi-source heterogeneous remote sensing space data query |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115934794A CN115934794A (en) | 2023-04-07 |
| CN115934794B true CN115934794B (en) | 2024-05-24 |
Family
ID=86648439
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211520789.4A Active CN115934794B (en) | 2022-11-30 | 2022-11-30 | Elastic management method for massive multi-source heterogeneous remote sensing space data query |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115934794B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116303249B (en) * | 2023-04-13 | 2023-08-04 | 中国科学院空天信息创新研究院 | Lake-bin integrated multi-source remote sensing space-time big data processing method and device |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108446367A (en) * | 2018-03-15 | 2018-08-24 | 湖南工业大学 | A kind of the packaging industry data search method and equipment of knowledge based collection of illustrative plates |
| CN108647266A (en) * | 2018-04-28 | 2018-10-12 | 重庆沐信润喆网络科技有限公司 | A kind of isomeric data is quickly distributed storage, exchange method |
| CN109284338A (en) * | 2018-10-25 | 2019-01-29 | 南京航空航天大学 | A kind of satellite remote sensing big data Optimizing Queries method based on hybrid index |
| CN109885642A (en) * | 2019-02-18 | 2019-06-14 | 国家计算机网络与信息安全管理中心 | Classification storage method and device towards full-text search |
| CN109902072A (en) * | 2019-02-21 | 2019-06-18 | 云南电网有限责任公司红河供电局 | A kind of log processing system |
| CN112015771A (en) * | 2020-10-15 | 2020-12-01 | 北京新唐思创教育科技有限公司 | Data retrieval method and device, electronic equipment and computer storage medium |
-
2022
- 2022-11-30 CN CN202211520789.4A patent/CN115934794B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108446367A (en) * | 2018-03-15 | 2018-08-24 | 湖南工业大学 | A kind of the packaging industry data search method and equipment of knowledge based collection of illustrative plates |
| CN108647266A (en) * | 2018-04-28 | 2018-10-12 | 重庆沐信润喆网络科技有限公司 | A kind of isomeric data is quickly distributed storage, exchange method |
| CN109284338A (en) * | 2018-10-25 | 2019-01-29 | 南京航空航天大学 | A kind of satellite remote sensing big data Optimizing Queries method based on hybrid index |
| CN109885642A (en) * | 2019-02-18 | 2019-06-14 | 国家计算机网络与信息安全管理中心 | Classification storage method and device towards full-text search |
| CN109902072A (en) * | 2019-02-21 | 2019-06-18 | 云南电网有限责任公司红河供电局 | A kind of log processing system |
| CN112015771A (en) * | 2020-10-15 | 2020-12-01 | 北京新唐思创教育科技有限公司 | Data retrieval method and device, electronic equipment and computer storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115934794A (en) | 2023-04-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108304444B (en) | Information query method and device | |
| US7890541B2 (en) | Partition by growth table space | |
| CN107423422B (en) | Spatial data distributed storage and search method and system based on grid | |
| US8356050B1 (en) | Method or system for spilling in query environments | |
| CN103020078B (en) | Distributing real-time data bank data hierarchy indexing means | |
| CN105160039A (en) | Query method based on big data | |
| CN105117502A (en) | Search method based on big data | |
| US11080207B2 (en) | Caching framework for big-data engines in the cloud | |
| Elmeleegy et al. | Spongefiles: Mitigating data skew in mapreduce using distributed memory | |
| CN115934794B (en) | Elastic management method for massive multi-source heterogeneous remote sensing space data query | |
| WO2023179787A1 (en) | Metadata management method and apparatus for distributed file system | |
| US20080250017A1 (en) | System and method for aiding file searching and file serving by indexing historical filenames and locations | |
| US20110153677A1 (en) | Apparatus and method for managing index information of high-dimensional data | |
| US10146833B1 (en) | Write-back techniques at datastore accelerators | |
| Qian et al. | An evaluation of Lucene for keywords search in large-scale short text storage | |
| Qi | Digital forensics and NoSQL databases | |
| CN107273443B (en) | Mixed indexing method based on metadata of big data model | |
| CN115981848B (en) | Memory database fragment adjustment method and equipment | |
| US20230169079A1 (en) | Scaling query processing resources for efficient utilization and performance | |
| EP4016312B1 (en) | Data operations using a cache table in a file system | |
| EP3995972A1 (en) | Metadata processing method and apparatus, and computer-readable storage medium | |
| Patgiri | MDS: In-depth insight | |
| CN106649462A (en) | Implementation method for mass data full-text retrieval scene | |
| CN111949439B (en) | Database-based data file updating method and device | |
| WO2024125799A1 (en) | Method for managing a database table storage tiering and a database management system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |