CN116662349A - Resampling system and method for raster data under high concurrency scene - Google Patents

Abstract

The invention relates to the field of raster data application, in particular to a method and a system for resampling raster data under a high concurrence scene, wherein the system comprises the following steps: the system comprises an expandable sampling module, an expandable cache calculation module and a storage area, wherein raster data, pyramid data, coordinate system data and index table data are stored in the storage module; the extensible cache computing module carries out pyramid processing on newly uploaded raster data to obtain pyramid data; and a plurality of expandable cache calculation modules work in parallel; the scalable sampling module comprises a plurality of independently operated modules for resampling pyramid data in response to a resampling requirement and feeding back to a user. In a high concurrency scene, a distributed cache storage technology and a distributed resampling technology are adopted, so that the complexity of system design is reduced, and meanwhile, the efficiency of data request of a client is optimized.

Description

Resampling system and method for raster data under high concurrency scene

Technical Field

The invention relates to the field of raster data application, in particular to a system and a method for resampling raster data under a high concurrence scene.

Background

Raster data is used as a common data format in a Geographic Information System (GIS), and a specific storage scheme for storing space information is realized by dividing a space into regular grids, each grid representing a unit and having specific attribute information on each unit, wherein the common raster data format comprises: TIFF, GIF, JPEG, etc.

With the development of unmanned aerial vehicle technology, high-definition orthographic image technology rises rapidly, and raster data is as the storage mode of high-definition orthographic image, and the information that needs to be stored is more and more, and the data volume is bigger and more, because the restriction of computer memory, the unable problem of opening the raster data of looking over has appeared. In order to solve the problem, the image pyramid technology is generally adopted to store raster data in a copying mode of gradually reducing resolution, when the raster data is used, the resolution similar to the resolution of a display area is selected, and the scheme can solve the problem of image display by only carrying out a small amount of inquiry and calculation, so that the dependence on memory and the consumption of time when the raster data is applied in a large amount are greatly reduced.

The high-speed development of the internet and high concurrency have become one of factors that must be considered in the design of a system architecture, which generally refers to that one system processes multiple requests simultaneously, and some indexes for checking the high concurrency include: response time, throughput, query rate per second, concurrent users, etc. In recent years, with development of cloud technology, more raster data applications are changed from desktop end single machine applications to cloud end architecture system level applications, and currently mainstream GIS products, such as ArcGis, hypergraph and the like, all adopt to process raster data into slice data, store the slice data in a data warehouse, download and assemble the slice data through front end requirements, so as to form a complete raster data. The above solution is adequate for concurrent lower accesses, but causes computational bottlenecks and retrieval bottlenecks for higher concurrent accesses.

The conventional pyramid processing is generally performed on a large amount of raster data, and has performance limitations which can only be applied to a single computer, and the processed pyramid data is generally stored on a computer hard disk, so that the image of a designated area needs to be resampled every time the image is acquired, and a large amount of calculation and search time is wasted.

The traditional service-oriented raster data slicing processing scheme generally adopts a mode that slicing is carried out on a server, slices are stored on a hard disk, and a client performs real-time calculation each time according to a protocol to acquire correct slice data.

Disclosure of Invention

Aiming at the problems that the high concurrency access can cause a calculation bottleneck and a search bottleneck, the traditional pyramid processing of a large amount of raster data wastes a large amount of calculation and search time, and the traditional service-oriented raster data slicing processing scheme can cause the calculation bottleneck and the search bottleneck and can not adapt to the explosive access requirement of a client, the invention establishes a distributed architecture structure and a flow with fully utilized calculation resources and storage resources, and provides a resampling system and a resampling method of raster data in a high concurrency scene so as to meet the application requirement of the raster data in the high concurrency scene.

In order to achieve the above object, the present invention provides the following technical solutions:

a resampling system for raster data in high concurrency scene is composed of expandable sampling module, expandable buffer calculation module and memory area,

the storage module stores raster data, pyramid data, coordinate system data and index table data;

the extensible cache computing module performs pyramid processing on newly uploaded raster data to obtain pyramid data; and a plurality of expandable cache calculation modules work in parallel;

the extensible sampling module comprises a plurality of independently operated modules and is used for responding to the resampling requirement, resampling the pyramid data and feeding back the pyramid data to a user.

Preferably, in the storage module, there is a correspondence between raster data and pyramid data, and between coordinate system data and raster data, and the index table is a relational data table, and indexes of raster data, pyramid data and coordinate system data are stored.

Preferably, the storage module includes an extensible NAS and a MYSQL storage record table,

the grid data and the pyramid data are stored in an extensible NAS, and the access mode of the extensible NAS is stored in a MYSQL storage record form; the coordinate system data and the index table data are recorded in a MYSQL storage record table.

As a preferred solution, if raster data is absent in the index table data, the scalable cache calculation module automatically generates corresponding pyramid data for the raster data, and stores the corresponding pyramid data in the storage module, which specifically includes the following steps:

uploading the new raster data to the storage module;

performing cache calculation on newly uploaded raster data, and searching idle calculation resources according to a cache calculation result;

the idle computing resource starts the buffer generating logic, on one hand, the image original file required by buffer generation is obtained from the storage module, and on the other hand, the layer-by-layer generation of pyramid data is started through the initialization configuration parameters set by the system;

and storing the generated new pyramid data into a storage module, and generating an index table.

The extensible sampling module is used for calling original raster data, pyramid data and coordinate system data, resampling the original raster data by combining parameters requested by a user, returning a resampling result to the user, storing the resampling result in the buffer module and establishing a corresponding retrieval index.

Preferably, the method further comprises a storage lock, wherein the storage lock is used for creating a piece of lock information for each access of the expandable resampling in the index table data, when the access is performed, the lock is ensured to be closed, and when the access is finished, the lock is released, so that the non-conflict of the data access is ensured.

Based on the same conception, a method for resampling raster data in a high concurrency scene is also provided, a raster data resampling system in the high concurrency scene is constructed, and the following steps are executed:

uploading raster data to a server, and generating an image pyramid;

step two, the client initiates a request to acquire image data of a designated area and a designated level;

step three, whether the data generated by resampling exist in the cache area of the storage module or not is searched, if so, a search result is directly returned, and if not, the step four is executed;

and step four, resampling pyramid data according to the appointed area and the appointed level, generating a sampling result and storing the sampling result in the cache area.

Compared with the prior art, the invention has the beneficial effects that:

the method of the invention realizes the service application of the grid pyramid data, adopts the distributed cache storage technology and the distributed resampling technology under the high concurrency scene, reduces the complexity of system design and optimizes the efficiency of data request of the client.

Description of the drawings:

FIG. 1 is a system architecture diagram of a resampling system for raster data in a high concurrency scenario in accordance with embodiment 1 of the present invention;

FIG. 2 is a block diagram of a memory module in embodiment 1 of the present invention;

FIG. 3 is a flowchart of generating pyramid data after uploading raster data to a storage center in embodiment 1 of the present invention;

FIG. 4 is a diagram of a specific expansion mode for expanding the cache calculation module in embodiment 1 of the present invention;

fig. 5 is a flowchart of a resampling method of raster data in a high concurrency scenario in embodiment 1 of the present invention.

Detailed Description

The present invention will be described in further detail with reference to test examples and specific embodiments. It should not be construed that the scope of the above subject matter of the present invention is limited to the following embodiments, and all techniques realized based on the present invention are within the scope of the present invention.

Example 1

A resampling system of raster data in high concurrency scene is shown in figure 1, which comprises an expandable sampling module, an expandable cache computing module and a storage area, wherein the storage module mainly adopts a hard disk storage or cloud storage mode to store original raster data, coordinate system data, index table data and pyramid data. And the extensible cache computing module is used for pyramid processing aiming at newly uploaded raster data, and the part is provided with a plurality of computing modules which work in parallel, so that pyramid processing work is guaranteed to be completed at the fastest speed. The extensible resampling module is also provided with a plurality of independently operated modules, and the important work of the extensible resampling module is to respond to the resampling requirement of a user, resample the pyramid, obtain user target data and return the user target data to the user.

The storage module generally adopts a hard disk file system for storage or adopts a cloud database for storage, and the storage content comprises structured data and unstructured data. Data classification mainly includes raster data, pyramid data, coordinate system data, and index table data. The structure of the memory module is shown in fig. 2.

Raster data refers to original data, namely a piece of TIFF data, GIF data or JPEG data, which generally has the characteristics of large data volume, fixed data format, low data use frequency and the like, and in order to reasonably store the data in the system, a special raster data warehouse is established, and basic data comparison, search, insertion, deletion and other functions are developed for the data warehouse.

Pyramid data generated based on raster data is stored in the pyramid warehouse, and the data is generally required to be used together with the raster data, so that when the data is stored, the one-to-one binding relationship between the pyramid warehouse data and the raster data is considered, and the data format stored in the data warehouse comprises, but is not limited to, over, rrd.

The coordinate system information of the raster data is stored in the coordinate system warehouse, the relevant information of key positions such as the scaling, the rotation angle and the like of the raster data is identified by the file, and the information of the warehouse and the original raster data have a one-to-one correspondence, so that the one-to-one correspondence is also stored in the system, and the data stored in the coordinate system warehouse comprise file formats such as, but not limited to, jgw, prj and the like.

The index table is used as a key table for original data retrieval, is a starting point for a storage module, an external program uses the storage module, firstly searches the index table, all information and information relations are stored in the index table, and the index table is a relational data table which mainly stores all raster data, pyramid data and coordinate system data indexes in a system and information and relations of the data.

In the design process of the storage area, the main problems are to consider the storage expansion problem and the quick retrieval and calling problem. In order to solve the problem, in the system design process, corresponding storage strategies are formulated aiming at different data types, wherein grid data and pyramid data are mainly unstructured data, the method has the characteristics of large volume of single file data, difficult structuring and difficult excessive copying, the data is inconvenient to store in a database, because the access efficiency of the data is greatly reduced, in addition, the problem of difficult cross-server access exists for the traditional hard disk storage because the traditional hard disk storage cannot be easily expanded to other service servers, and the NAS storage just solves the problem, is suitable for storing unstructured data and can be accessed across servers on a network, and the method expands the NAS storage management. The coordinate system data has a certain diversity, and the PRJ data, the XML data and some unknown expansion data are mainly considered, so that the storage and the management of the coordinate system data adopt the same management scheme as the grid data and the pyramid data in the invention. In order to ensure the stable operation of the system, a special monitoring module is designed in the invention, and the functions of real-time health diagnosis, abnormal warning, automatic repair of health state and the like can be carried out on the storage in the monitoring module.

The extensible cache computing module is mainly used for generating pyramid data, and can collect corresponding data, call a computing process and generate a cache result at the first time for original data submitted by a user. The extensible cache computing module is used for providing a program for automatically generating pyramids aiming at the pyramids without pyramid data, and after a user uploads the pyramids, index tables are searched in the computing module, and when the situation that the pyramids lack pyramid data is found, corresponding pyramid data can be automatically generated aiming at the pyramids. A flowchart of generating pyramid data after uploading raster data to a storage center is shown in fig. 3.

After the raster data is uploaded to the storage center, the uploading program sends a new task message to the task center, the task center is informed to carry out cache calculation on the newly uploaded raster data, after the task center is successful in task, an idle computing resource finder of the computing center is called to find idle computing resources, and once the idle computing resources are determined, the idle computing resources start cache generation logic, on one hand, the newly uploaded raster data required by cache generation is obtained from the storage center, on the other hand, initialization configuration parameters are set through the system, layer-by-layer generation of pyramid data is started, finally, the generated data is stored in the storage center again, and an index table is generated for subsequent service call.

In the process of generating the cache, basic operations such as interrupt and restarting of tasks are supported. The tasks of the task center are directly controlled by simple commands under the condition that the user does not feel. In the computing center, a single computing resource can only process a single image data at the same time, so when the task processing of the task center is too busy, the computing resource of the computing center becomes more intense, in this case, an administrator can automatically expand the computing resource through simple configuration, and a specific expansion mode for expanding the cache computing module is shown in fig. 4. When the resources of the computing center are tense, a new computer is allocated, the new computer can access the computing resource pool and the storage pool, a computing module application program is installed and operated on the new computer, after the program is operated, the computing resource pool is provided with a computing module in an idle state, the computing module can actively access a task to be processed, acquire a task Id and corresponding information, lock the task, acquire image data to be processed from the storage pool, execute the processing work of pyramid data, and store the processing result in the storage pool after the processing is completed.

The expandable resampling module is used for searching the raster data area and the sampling rate required by the user in the buffer module, and if the buffer module does not have the data required by the user, the resampling module is required to regenerate the data. And the resampling module is used for calling the original raster data, the pyramid data and the coordinate system data, resampling the original raster in combination with parameters (such as an image window) requested by a user, returning a resampling result to the user, storing the resampling result in the caching module and establishing a corresponding retrieval index. The request processing module is an index of the calculation module, all client requests enter the request processing module firstly, the request processing module classifies request types, determines request types such as login authentication, cache management or number bin management and the like, and then gives the requests to the appointed module for processing.

The resampling module is a telescopic sampling module, searches for an idle sampling module according to a resampling request of a user, and searches for cache data or resamples the data through the sampling module.

(1) Original image data for existing pyramid data.

And directly accessing pyramid data by a resampling module aiming at original image data of existing pyramid data, and adopting.

(2) For image data for which pyramid data has not been generated.

Firstly, searching whether a task center has an ongoing task, and if so, waiting for the completion of the task and then sampling. If the task of the task center does not have the task of the current image data, creating a new task, generating pyramid data, and then sampling.

Since there are multiple extensible service plugins for the extensible module, there may be a problem with conflicting access to the storage area data for which the present invention creates a storage lock.

In the index data, a piece of lock information is created for each access of the scalable resampling, when the access is performed, the lock is ensured to be closed, and when the access is finished, the lock is released, so that the non-conflict of the data access is ensured.

Example 2

On the basis of the system for resampling raster data in a high concurrency scene in embodiment 1, a method for resampling raster data in a high concurrency scene is also provided, and a flowchart of the method is shown in fig. 5, and includes the following steps:

step one, uploading raster data to a server, and generating an image pyramid.

And step two, the client initiates a request to acquire the image data of the designated area and the designated level.

And step three, whether the data generated by resampling exists in the search buffer area or not is searched, if so, a search result is directly returned, and if not, the step four is executed.

And step four, resampling pyramid data according to the appointed area and the appointed level, generating a sampling result and storing the sampling result in the cache area.

The foregoing description of the preferred embodiments of the invention is not intended to limit the invention to the particular embodiments disclosed, but on the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

Claims (7)

1. A resampling system for raster data in high concurrency scene is characterized by comprising an expandable sampling module, an expandable cache calculating module and a storage area,

the storage module stores raster data, pyramid data, coordinate system data and index table data;

the extensible cache computing module performs pyramid processing on newly uploaded raster data to obtain pyramid data; and a plurality of expandable cache calculation modules work in parallel;

the extensible sampling module comprises a plurality of independently operated modules and is used for responding to the resampling requirement, resampling the pyramid data and feeding back the pyramid data to a user.

2. The system for resampling raster data in a high concurrency scenario of claim 1, wherein in said storage module there is a correspondence between raster data and pyramid data, and coordinate system data and raster data, and said index table is a relational data table storing indexes of raster data, pyramid data and coordinate system data.

3. The system of claim 1, wherein the storage module comprises a scalable NAS and MYSQL storage record table,

the grid data and the pyramid data are stored in an extensible NAS, and the access mode of the extensible NAS is stored in a MYSQL storage record form; the coordinate system data and the index table data are recorded in a MYSQL storage record table.

4. The system for resampling raster data in a high concurrency scenario of claim 1, wherein if raster data is absent from the index table data, the scalable cache computation module automatically generates corresponding pyramid data for raster data and stores the corresponding pyramid data in the storage module, comprising the steps of:

uploading the new raster data to the storage module;

performing cache calculation on newly uploaded raster data, and searching idle calculation resources according to a cache calculation result;

the idle computing resource starts the buffer generating logic, on one hand, the image original file required by buffer generation is obtained from the storage module, and on the other hand, the layer-by-layer generation of pyramid data is started through the initialization configuration parameters set by the system;

and storing the generated new pyramid data into a storage module, and generating an index table.

5. The system for resampling raster data in a high concurrency scenario of claim 1, wherein the scalable sampling module is configured to invoke the original raster data, the pyramid data, and the coordinate system data, resample the original raster data in combination with parameters requested by the user, return the resample result to the user, store the resample result in the buffer module, and build the corresponding search index.

6. The system of claim 1, further comprising a storage lock for creating a lock message for each access of the scalable resampling in the index table data, wherein the lock is guaranteed to be closed when the access is performed, and is released when the access is finished, thereby guaranteeing the non-collision of the data access.

7. A method for resampling raster data in a high concurrency scenario, comprising the steps of:

uploading raster data to a server, and generating an image pyramid;

step two, the client initiates a request to acquire image data of a designated area and a designated level;

step three, whether the data generated by resampling exist in the cache area of the storage module or not is searched, if so, a search result is directly returned, and if not, the step four is executed;

and step four, resampling pyramid data according to the appointed area and the appointed level, generating a sampling result and storing the sampling result in the cache area.