CN109145155A - High-concurrency warehousing processing method for mass remote sensing image metadata - Google Patents
High-concurrency warehousing processing method for mass remote sensing image metadata Download PDFInfo
- Publication number
- CN109145155A CN109145155A CN201810747217.7A CN201810747217A CN109145155A CN 109145155 A CN109145155 A CN 109145155A CN 201810747217 A CN201810747217 A CN 201810747217A CN 109145155 A CN109145155 A CN 109145155A
- Authority
- CN
- China
- Prior art keywords
- remote sensing
- sensing image
- file
- image metadata
- request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/547—Messaging middleware
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a high-concurrency warehousing processing method for mass remote sensing image metadata, which comprises the following specific steps of: the method comprises the following steps that firstly, a requesting party sends a remote sensing image metadata application request file to a resource party according to application record information; step two, the resource side receives the request file, performs file format conversion on the request file, acquires a request parameter value according to a node value of a request file structure, and returns a request result to the request side in the same file format; step three, the request party receives the request result, stores the request result in a file format and puts the storage path into a message queue; and step four, reading a storage path in the message queue through the thread pool, performing database storage processing on the remote sensing image metadata in a multithreading mode, and inquiring and updating a database storage state. The invention improves the file transmission efficiency, enhances the readability of the request result, improves the expansibility and the transportability of the data and improves the data storage amount.
Description
Technical field
The present invention relates to the technical field of remote sensing image data processing method, in particular to a kind of mass remote sensing image member number
Enter library processing method according to high concurrent.
Background technique
With the rapid development of Spatial Information Science, the means and mode of various Spatial data captures are increasingly quickly and various
Change, showing more outstanding is remote sensing image data.Current many national governments are by mass remote sensing image data application in state
The fields such as anti-, territory, environment, resource provide various map navigation services for masses, and gradual remote sensing image data becomes
Important one of the data source of Spatial Information Science.
But often data volume is huge for remote sensing image data, traditional remote sensing image data is usually that image data is placed
Into Large-scale Relational Database, therefore the reading of data, transmission, the time of working process are all very long, this has seriously affected operation
The quick ease for use of system.
Summary of the invention
In order to solve the above-mentioned technical problem the present invention, and provides a kind of mass remote sensing image metadata high concurrent storage processing
Method.
The technical solution used in the present invention is:
Mass remote sensing image metadata high concurrent of the invention enters library processing method, the specific steps are as follows:
Step 1, requesting party send remote sensing image metadata application demand file to resource side according to application record information；
Step 2, resource side receive demand file, demand file are carried out file format conversion, according to demand file structure
Nodal value acquisition request parameter value, and request results are back to requesting party with same file format；
Step 3, requesting party receive request results, and request results are stored as a file format, and storage path is put into and is disappeared
It ceases in queue；
Step 4 reads the store path in message queue by thread pool, using multithreading by remote sensing image metadata
Database storage processing is carried out, and inquires and update database storage state.
In the step one, requesting party sends the application of remote sensing image metadata to resource side according to application record information and asks
Ask, resource side return remote sensing image metadata query signal after, requesting party judge remote sensing image metadata query signal whether at
Function requests to return to remote sensing image metadata query as a result, terminating remote sensing image metadata Shen if failure if success to resource side
It please operate.
In the step one, after requesting party receives remote sensing image metadata query result, remote sensing image metadata is judged
Whether query result is empty, if it is not, the image position picture of remote sensing image metadata query result is saved to temporary folder, if
It is to terminate remote sensing image metadata application operation.
In the step one, requesting party is further according to application record information by the image of remote sensing image metadata query result
Position picture is converted into the demand file of Base64 character string, and the demand file of Base64 character string is sent to resource side,
Then it updates database storage state and sends the demand file time.
In the step two, received Base64 character string is converted to XML format file and thumbnail pressure by resource side
Contracting file, and XML format file and thumbnail compressed file are back to requesting party.
In the step three, requesting party stores the XML format file received and thumbnail compressed file to interim
File, then temporary folder path is stored into Message Queuing.
In the step four, requesting party first passes through thread pool and obtains temporary folder path from Message Queuing, then solves
XML format file content is read in analysis, and thumbnail compressed file is de-compressed into image thumbnail, by XML format file content
Save to database, and upload the GT-DATA file of image thumbnail to database, then judge image thumbnail whether on
It passes successfully, if so, image thumbnail storage path in more new database, and delete the compression text of the thumbnail in temporary folder
Part, if it is not, repeating step 3.
In the step four, requesting party's timing inquires current time and before the transmission demand file time between 5 minutes
The image thumbnail of all remote sensing image metadata in database, carries out storage processing for image thumbnail, and judge database
In image thumbnail whether be put in storage success, if so, deleting the image thumbnail in GT-DATA file, and more new database
Binning state is to be put in storage successfully, then terminates remote sensing image metadata application operation, if it is not, updating database storage state is not
Storage waits and retransmits remote sensing image metadata application request next time.
The advantages and positive effects of the present invention are:
The present invention can store using xml formatted file by encrypted picture file, and file transmission efficiency is improved, and enhancing is asked
It is readable to seek result, while also improving the expansion and portability of data, also solves the same time by original single thread
The problem of processing individual data becomes same time-triggered protocol multiple data, improves the amount of data loading.
Detailed description of the invention
Fig. 1 is flow diagram of the invention.
Specific embodiment
Referring to the drawings and embodiment the present invention will be described in detail.
As shown in Figure 1, mass remote sensing image metadata high concurrent of the invention enters library processing method, the specific steps are as follows:
Step 1, requesting party send remote sensing image metadata application demand file to resource side according to application record information；
Step 2, resource side receive demand file, demand file are carried out file format conversion, according to demand file structure
Nodal value acquisition request parameter value, and request results are back to requesting party with same file format；
Step 3, requesting party receive request results, and request results are stored as a file format, and storage path is put into and is disappeared
It ceases in queue；
Step 4 reads the store path in message queue by thread pool, using multithreading by remote sensing image metadata
Database storage processing is carried out, and inquires and update database storage state.
In the step one, requesting party sends the application of remote sensing image metadata to resource side according to application record information and asks
Ask, resource side return remote sensing image metadata query signal after, requesting party judge remote sensing image metadata query signal whether at
Function requests to return to remote sensing image metadata query as a result, terminating remote sensing image metadata Shen if failure if success to resource side
It please operate.
In the step one, after requesting party receives remote sensing image metadata query result, remote sensing image metadata is judged
Whether query result is empty, if it is not, the image position picture of remote sensing image metadata query result is saved to temporary folder, if
It is to terminate remote sensing image metadata application operation.
In the step one, requesting party is further according to application record information by the image of remote sensing image metadata query result
Position picture is converted into the demand file of Base64 character string, and the demand file of Base64 character string is sent to resource side,
Then it updates database storage state and sends the demand file time.
In the step two, received Base64 character string is converted to XML format file and thumbnail pressure by resource side
Contracting file, and XML format file and thumbnail compressed file are back to requesting party.
In the step three, requesting party stores the XML format file received and thumbnail compressed file to interim
File, then temporary folder path is stored into Message Queuing.
In the step four, requesting party first passes through thread pool and obtains temporary folder path from Message Queuing, then solves
XML format file content is read in analysis, and thumbnail compressed file is de-compressed into image thumbnail, by XML format file content
Save to database, and upload the GT-DATA file of image thumbnail to database, then judge image thumbnail whether on
It passes successfully, if so, image thumbnail storage path in more new database, and delete the compression text of the thumbnail in temporary folder
Part, if it is not, repeating step 3.
In the step four, requesting party's timing inquires current time and before the transmission demand file time between 5 minutes
The image thumbnail of all remote sensing image metadata in database, carries out storage processing for image thumbnail, and judge database
In image thumbnail whether be put in storage success, if so, deleting the image thumbnail in GT-DATA file, and more new database
Binning state is to be put in storage successfully, then terminates remote sensing image metadata application operation, if it is not, updating database storage state is not
Storage waits and retransmits remote sensing image metadata application request next time.
Remote sensing image metadata directly can be converted to the self-defined structure file that operating system can manage by the present invention,
And extract corresponding metadata information and be stored in operating system relational database, each type data all correspond one certainly
The file structure of definition.By transmitting file, file content is read according still further to file node structure, it by such method can be with
The processing of mass remote sensing image metadata high concurrent is greatly improved, is conveniently put in storage, checks corresponding metadata information.
The use that the present invention is integrated using message queue component and thread pool, in conjunction with the performance of server, by database
Maximum number of concurrent is debugged in conjunction with thread pool into optimum state.Message Queuing Middleware can solve the decoupling between component, both
It can solve the business processing of part high concurrent, and message queue Transmission system log can be passed through.The present invention utilizes thread pool
Can the multiple requests of concurrent processing simultaneously, reduce thread and take time in creation and destruction and the expense of system, improve and be
The performance of system.
Claims (8)
1. a kind of mass remote sensing image metadata high concurrent enters library processing method, it is characterised in that: specific step is as follows:
Step 1, requesting party send remote sensing image metadata application demand file to resource side according to application record information；
Step 2, resource side receive demand file, demand file are carried out file format conversion, according to demand file structure
Nodal value acquisition request parameter value, and request results are back to requesting party with same file format；
Step 3, requesting party receive request results, and request results are stored as a file format, and storage path is put into and is disappeared
It ceases in queue；
Step 4 reads the store path in message queue by thread pool, using multithreading by remote sensing image metadata
Database storage processing is carried out, and inquires and update database storage state.
2. mass remote sensing image metadata high concurrent according to claim 1 enters library processing method, it is characterised in that: described
The step of one in, requesting party according to application record information to resource side send the application of remote sensing image metadata request, resource side returns
After returning remote sensing image metadata query signal, requesting party judges whether remote sensing image metadata query signal succeeds, if success, to
The request of resource side returns to remote sensing image metadata query as a result, if failure, terminates remote sensing image metadata application operation.
3. mass remote sensing image metadata high concurrent according to claim 2 enters library processing method, it is characterised in that: described
The step of one in, after requesting party receives remote sensing image metadata query result, whether judge remote sensing image metadata query result
For sky, if it is not, saving the image position picture of remote sensing image metadata query result to temporary folder, if so, terminating remote sensing
Image metadata application operation.
4. mass remote sensing image metadata high concurrent according to claim 3 enters library processing method, it is characterised in that: described
The step of one in, requesting party converts the image position picture of remote sensing image metadata query result further according to application record information
It is sent to resource side at the demand file of Base64 character string, and by the demand file of Base64 character string, then more new data
Library binning state and transmission demand file time.
5. mass remote sensing image metadata high concurrent according to claim 1 enters library processing method, it is characterised in that: described
The step of two in, received Base64 character string is converted to XML format file and thumbnail compressed file by resource side, and will
XML format file and thumbnail compressed file are back to requesting party.
6. mass remote sensing image metadata high concurrent according to claim 1 enters library processing method, it is characterised in that: described
The step of three in, requesting party stores the XML format file received and thumbnail compressed file to temporary folder, then will face
When folder path store into Message Queuing.
7. mass remote sensing image metadata high concurrent according to claim 1 enters library processing method, it is characterised in that: described
The step of four in, requesting party first passes through thread pool and obtains temporary folder path from Message Queuing, then parses reading XML format
File content, and thumbnail compressed file is de-compressed into image thumbnail, XML format file content is saved to database,
And the GT-DATA file of image thumbnail to database is uploaded, then judge whether image thumbnail uploads success, if so,
Image thumbnail storage path in more new database, and the thumbnail compressed file in temporary folder is deleted, if it is not, repeating to walk
Rapid three.
8. mass remote sensing image metadata high concurrent according to claim 1 enters library processing method, it is characterised in that: described
The step of four in, requesting party's timing inquiry current time and sending before the demand file time owns in database between 5 minutes
Image thumbnail is carried out storage processing by the image thumbnail of remote sensing image metadata, and judges the image breviary in database
Figure whether be put in storage success, if so, delete GT-DATA file in image thumbnail, and update database storage state be into
Then Kucheng's function terminates remote sensing image metadata application operation, if it is not, updating database storage state is not to be put in storage, under waiting
Secondary retransmission remote sensing image metadata application request.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810747217.7A CN109145155A (en) | 2018-07-09 | 2018-07-09 | High-concurrency warehousing processing method for mass remote sensing image metadata |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810747217.7A CN109145155A (en) | 2018-07-09 | 2018-07-09 | High-concurrency warehousing processing method for mass remote sensing image metadata |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109145155A true CN109145155A (en) | 2019-01-04 |
Family
ID=64800129
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810747217.7A Pending CN109145155A (en) | 2018-07-09 | 2018-07-09 | High-concurrency warehousing processing method for mass remote sensing image metadata |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109145155A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110515797A (en) * | 2019-07-31 | 2019-11-29 | 广东浪潮大数据研究有限公司 | A kind of data processing method, device, equipment and computer readable storage medium |
| CN111565220A (en) * | 2020-04-24 | 2020-08-21 | 中国科学院空天信息创新研究院 | Remote sensing image data quick access method and system |
| CN112214461A (en) * | 2020-10-12 | 2021-01-12 | 河南大学 | Fuzzy XML (extensive markup language) compression method for remote sensing metadata |
| CN112631801A (en) * | 2020-12-22 | 2021-04-09 | 无锡江南计算技术研究所 | Distributed parallel method for intelligent remote sensing image model |
-
2018
- 2018-07-09 CN CN201810747217.7A patent/CN109145155A/en active Pending
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110515797A (en) * | 2019-07-31 | 2019-11-29 | 广东浪潮大数据研究有限公司 | A kind of data processing method, device, equipment and computer readable storage medium |
| CN111565220A (en) * | 2020-04-24 | 2020-08-21 | 中国科学院空天信息创新研究院 | Remote sensing image data quick access method and system |
| CN112214461A (en) * | 2020-10-12 | 2021-01-12 | 河南大学 | Fuzzy XML (extensive markup language) compression method for remote sensing metadata |
| CN112214461B (en) * | 2020-10-12 | 2022-09-30 | 河南大学 | Fuzzy XML compression method for remote sensing metadata |
| CN112631801A (en) * | 2020-12-22 | 2021-04-09 | 无锡江南计算技术研究所 | Distributed parallel method for intelligent remote sensing image model |
| CN112631801B (en) * | 2020-12-22 | 2022-10-04 | 无锡江南计算技术研究所 | Distributed parallel method for intelligent remote sensing image model |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109145155A (en) | High-concurrency warehousing processing method for mass remote sensing image metadata | |
| CN107038162B (en) | Real-time data query method and system based on database log | |
| JP7018516B2 (en) | Data query | |
| US7487168B2 (en) | System and method for loading hierarchical data into relational database systems | |
| US9037555B2 (en) | Asynchronous collection and correlation of trace and communications event data | |
| US8407363B2 (en) | Generating low resolution user interfaces for transmission to mobile devices | |
| US10043220B2 (en) | Method, device and storage medium for data processing | |
| CN111046100B (en) | Method and system for synchronizing relational database to non-relational database | |
| CN107423436A (en) | A kind of method migrated for online data between distinct type data-base | |
| CN110505495B (en) | Multimedia resource frame extraction method, device, server and storage medium | |
| WO2006004717A2 (en) | System and method for extending business systems to a mobile workforce | |
| CN107818120A (en) | Data processing method and device based on big data | |
| CN112988916B (en) | Full and incremental synchronization method, apparatus and storage medium for Clickhouse | |
| CN104765840A (en) | Big data distributed storage method and device | |
| CN104794190A (en) | Method and device for effectively storing big data | |
| CN103944744A (en) | Method and system for log acquisition | |
| CN103607476A (en) | System and method for distributed multi-stage persistence of huge number of messages | |
| CN112181393B (en) | Front-end and back-end code generation method and device, computer equipment and storage medium | |
| CN102594874B (en) | Synchronization processing method and device | |
| CN115033569A (en) | Self-defined remote sensing image metadata warehousing method | |
| CN112948410A (en) | Data processing method, device, equipment and medium | |
| CN116186082A (en) | Data summarizing method based on distribution, first server and electronic equipment | |
| JP5546909B2 (en) | Data processing system, method and program | |
| CN110569456B (en) | WEB end data offline caching method and device and electronic equipment | |
| CN113542822A (en) | Image file transmission method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20190104 |
|
| WD01 | Invention patent application deemed withdrawn after publication |