CN113220807A - Automatic remote sensing data publishing and online process monitoring method - Google Patents
Automatic remote sensing data publishing and online process monitoring method Download PDFInfo
- Publication number
- CN113220807A CN113220807A CN202110354911.4A CN202110354911A CN113220807A CN 113220807 A CN113220807 A CN 113220807A CN 202110354911 A CN202110354911 A CN 202110354911A CN 113220807 A CN113220807 A CN 113220807A
- Authority
- CN
- China
- Prior art keywords
- data
- remote sensing
- sensing data
- file
- monitoring
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 72
- 238000012544 monitoring process Methods 0.000 title claims abstract description 24
- 230000009193 crawling Effects 0.000 claims abstract description 10
- 238000005516 engineering process Methods 0.000 claims abstract description 10
- 238000000605 extraction Methods 0.000 claims abstract description 10
- 230000000694 effects Effects 0.000 claims abstract description 5
- 238000013461 design Methods 0.000 claims abstract description 4
- 238000012545 processing Methods 0.000 claims description 23
- 238000013500 data storage Methods 0.000 claims description 6
- 238000001914 filtration Methods 0.000 claims description 2
- 230000014509 gene expression Effects 0.000 claims description 2
- 238000012216 screening Methods 0.000 claims description 2
- 230000001105 regulatory effect Effects 0.000 claims 1
- 238000004519 manufacturing process Methods 0.000 abstract description 6
- 238000007726 management method Methods 0.000 abstract description 3
- 238000011160 research Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 240000004282 Grewia occidentalis Species 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000009440 infrastructure construction Methods 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/29—Geographical information databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2246—Trees, e.g. B+trees
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
- G06F16/24573—Query processing with adaptation to user needs using data annotations, e.g. user-defined metadata
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/547—Remote procedure calls [RPC]; Web services
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Library & Information Science (AREA)
- Computational Linguistics (AREA)
- Remote Sensing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method for automatically releasing remote sensing data and monitoring an online process, which designs and completes data crawling, data cataloging, metadata extraction, image releasing and data warehousing to process the remote sensing data. The method comprises the technology of using remote sensing data in a data crawling link, naming rules of Modis data and Landsat8OLI image data related to data catalogs, a method for extracting Modis metadata and Landsat8 metadata, and related technology and steps for realizing automatic release of the remote sensing image data based on GeoServer RESTAPI. And adopting an Activiti workflow technology to carry out flow design on the links of acquisition, cataloging, release and metadata extraction of the remote sensing data to a warehouse, constructing a workflow scheduling model, and finishing automatic release and online flow monitoring of the remote sensing data. The invention can realize the automatic release of the remote sensing data, realize the on-line monitoring of the automatic release process, accelerate the production efficiency of the remote sensing data release and assist the management personnel to carry out the on-line real-time monitoring of the release process.
Description
Technical Field
The invention relates to the technical field of data release and process monitoring, in particular to a method for automatically releasing remote sensing data and monitoring an online process.
Background
Because the existing remote sensing data processing system is not perfect enough, in the traditional remote sensing infrastructure construction, each link of remote sensing processing is regarded as an individual which independently exists, data cannot be mutually communicated, even the data which are mutually associated often need to be distributed and distributed manually, the whole processing flow is very lagged, the efficiency is low, a streamlined and automatic resource processing system is not really established, meanwhile, a manager cannot supervise and control the working condition of the manager in time when facing a plurality of data processing flows, the problem is an important problem which hinders the remote sensing production and application to be solved urgently, and how to face and solve the higher requirement which is provided for the modern remote sensing product processing system.
The remote sensing information is used as an information resource, the sharing and application of the remote sensing information are directly related to the service level in the remote sensing field of China, although the remote sensing data volume is rapidly increased in recent years, a remote sensing information sharing mechanism, an application service and the like are not really established, the acquisition, production, processing and application of the remote sensing data are disconnected, and the data are in a 'more or less' state. Aiming at the situation, a Remote Sensing Information Model (RSIM) integrating data query, service subscription and information publishing services is provided for students of all the categories based on a service chain working mode, and social services and values are realized through the subscription of an intelligent remote sensing service model and the automatic processing of remote sensing data (refer to RenOm, Liqing, Shanyang, and the like. The students of Lizhou Qing et al propose a demo WPS platform realizing an embedding process based on OGC, which is used for solving the problem of interoperation of spatial information processing functions (refer to Sun rain, Lizhou Qing, Huang Zhen Chun. the research on realizing processing services based on OGC WPS standard [ J ] computer science, 2009,36(8): 86-88). Zhang Chang Rong, Shu le, Deng super propose a remote sensing image distributed processing system architecture; a complex processing process is realized by linking simple functional services (refer to Zhang Cheng Rong, Shu le, Dun super, and the like, and the research on Web environment remote sensing image processing technology based on OGC WPS [ J ]. university of Zhejiang university newspaper (engineering edition), 2008,42(7): 1184-1188). Doctor Jing Chang Feng proposes a distributed application integration reference model of a GIS service chain based on workflow in his thesis, and realizes interaction of data and system construction and tracking and monitoring of execution of the GIS service chain according to the model (refer to Jing Chang Feng. GIS service chain model research and realization based on workflow technology [ D ]. Zhejiang university, 2008). The method proposed by the scholars starts to adopt a method for making a remote sensing information model of a service chain and a workflow mode and tracking and monitoring data in the technical field of remote sensing data processing to solve the problem in the aspect of remote sensing data flow management, but the method can achieve automation, relatively few semi-automatic production processing methods are adopted, and meanwhile standardization and industrialization facing to the application field of remote sensing satellites are in the primary stage, and the method is difficult to provide support for the business operation of remote sensing monitoring. Therefore, a remote sensing data automatic publishing and online process monitoring method is provided.
Disclosure of Invention
In view of the above technical problems, an object of the present invention is to provide a method for automatically publishing remote sensing data and monitoring an online process, which comprises the steps of processing the remote sensing data through data crawling, data cataloging, metadata extraction, image publishing and data warehousing. And adopting an Activiti workflow technology to carry out flow design on the links of acquisition, cataloging, release and metadata extraction of the remote sensing data to a warehouse, and constructing a workflow scheduling model. The achievement can accelerate the production efficiency of remote sensing parameter products, realize monitoring the automatic processing flow of remote sensing data, and assist management personnel to better supervise and control the processing flow.
In order to achieve the aim, the invention provides a method for automatically releasing remote sensing data and monitoring an online process, which comprises the following steps:
s1: data crawling: regularly crawling landsdat 8 data of the USGS (American geological exploration institute) official website by using a crawler technology;
s2: cataloging data: and presetting corresponding regular matching expressions for files with different naming rules, and identifying massive remote sensing data files. Dividing the remote sensing data into directory tree structures of different categories and sequences according to a certain rule, so as to be convenient for subsequent calling and using;
s3: data release: issuing the remote sensing image data in batches through a GeoServer REST API based on the result obtained by S2;
s4: and (3) metadata extraction: extracting Modis metadata from hdf and Landsat8 metadata from txt file based on the result obtained at S2;
s5: and (4) data storage: storing remote sensing images issued by the GeoServer into a database in an OGC (open log record) resource form, so that geographic services can be conveniently used in subsequent work, and geographic information resources can be quickly and conveniently accessed and used anywhere;
s6: scheduling a remote sensing data processing flow: the remote sensing task scheduling process model is generated through an Activiti plug-in supported by Eclipse, a bpmn file and a corresponding png file are generated through a process designer, and the bpmn file is used for defining a process rule and is used for a computer; the png file is a picture for showing the progress of the process and is used by the user.
Optionally, the step 3 specifically implements data release as follows:
(1) the method comprises the steps of parameter input, wherein a user needs to input necessary data as parameters, and a system can find an appointed folder according to read parameter information;
(2) the multi-source data identification comprises two data files, namely Landsat8 and MODIS data files. The MODIS file is in an HDF format, needs to be converted into a GeoTIFF file, and is released in a unified mode. And then, performing file identification and filtering, and screening out files conforming to the release format. And judging the code in a mode of intercepting the suffix of the file, issuing the image according with the format, not issuing the image according with the format, and continuously judging the next file. Repeating the steps until all the files under the path traverse;
(3) and image issuing, namely acquiring the permission of the GeoServer REST API method through a user name and a password, issuing the remote sensing image according to user-defined parameter information, generating corresponding OGC service according to the ID of the image successfully issued, and outputting the OGC service to a specified txt file.
Optionally, the specific implementation method for extracting Modis metadata in step 4 is as follows: the hdf file path is imported in the c + + environment through the gdal library. And calling a GDNLOpen () function to open the HDF data set and judging whether the data set is empty or not. If the data reading is empty, the data reading fails. And if not, calling a GetMetadata () function to acquire the metadata character string.
Optionally, the specific implementation method for metadata extraction in step 4Landsat8 is as follows: and directly reading the compressed packet through java IO stream to obtain the metadata, txt file.
Optionally, the remote sensing data processing tool process scheduling in step 6 specifically includes: firstly, related data is crawled regularly through a data crawling link, the acquired data is sent to a data catalog, catalog classification is carried out after the data is structured according to rules, and after execution is finished, a parallel gateway is entered to divide a business flow into two sequential flows. One sequence flow enters a remote sensing data publishing link, and the other sequence flow enters a remote sensing metadata extraction link. And simultaneously carrying out two sequential flows until the two sequential flows run through a second parallel gateway, and then converging to finish data storage.
Further, a data module and a cache slice module are arranged in the data release:
(1) the data module is provided with six sub-modules of layer preview, a working area, data storage, layers, a layer group, a pattern and the like;
(2) the cache slicing module is provided with four sub-modules of layer slicing, cache setting, grid setting and disk quota.
Therefore, the method for automatically releasing the remote sensing data and monitoring the on-line process can accelerate the production efficiency of remote sensing parameter products, monitor the automatic processing process of the remote sensing data and assist managers to better supervise, control and control the processing process. The method provides a key technology related to remote sensing data crawling, cataloging and metadata extraction. The remote sensing image data can be automatically released based on the GeoServer REST API. The remote sensing data is acquired, catalogued, released and metadata extracted to a warehousing link through an Activiti workflow technology, a workflow scheduling model is constructed, and the remote sensing data is automatically arranged to produce remote sensing parameter products.
Drawings
FIG. 1 is a flow chart of a remote sensing data processing tool of the present invention;
FIG. 2 is a flow chart of remote sensing parametric product inspection of the present invention;
FIG. 3 is a diagram of a four-corner latitude and longitude frame selection query result of the present invention;
fig. 4 is a diagram of the Landsat8 data management interface of the present invention.
Claims (8)
1. A remote sensing data automatic release and online process monitoring method is characterized by comprising the following steps:
s1: data crawling: regularly crawling landsdat 8 data of the USGS (American geological exploration institute) official website by using a crawler technology;
s2: cataloging data: and presetting corresponding regular matching expressions for files with different naming rules, and identifying massive remote sensing data files. Dividing the remote sensing data into directory tree structures of different categories and sequences according to a certain rule, so as to be convenient for subsequent calling and using;
s3: data release: issuing the remote sensing image data in batches through a GeoServer REST API based on the result obtained by S2;
s4: and (3) metadata extraction: extracting Modis metadata from hdf and Landsat8 metadata from txt file based on the result obtained at S2;
s5: and (4) data storage: storing remote sensing images issued by the GeoServer into a database in an OGC (open log record) resource form, so that geographic services can be conveniently used in subsequent work, and geographic information resources can be quickly and conveniently accessed and used anywhere;
s6: scheduling a remote sensing data processing flow: the remote sensing task scheduling process model is generated through an Activiti plug-in supported by Eclipse, a bpmn file and a corresponding png file are generated through a process designer, and the bpmn file is used for defining a process rule and is used for a computer; the png file is a picture for showing the progress of the process and is used by the user.
2. The method for automatically releasing remote sensing data and monitoring the online process according to claim 1, wherein the method comprises the following steps: the step S3 is implemented by the following steps:
(1) the method comprises the steps of parameter input, wherein a user needs to input necessary data as parameters, and a system can find an appointed folder according to read parameter information;
(2) multi-source data identification, including two data files, Landsat8 and MODIS data files; the MODIS file is in HDF format and needs to be converted into a GeoTIFF file, and the format is released uniformly; then, performing file identification and filtering, and screening out files conforming to the release format; judging the code in a mode of intercepting suffix of the file, issuing the image according with the format, not issuing the image according with the format, and continuously judging the next file; repeating the steps until all the files under the path traverse;
(3) and image issuing, namely acquiring the permission of the GeoServer REST API method through a user name and a password, issuing the remote sensing image according to user-defined parameter information, generating corresponding OGC service according to the ID of the image successfully issued, and outputting the OGC service to a specified txt file.
3. The method for automatically releasing remote sensing data and monitoring the online process according to claim 1, wherein the method comprises the following steps: the specific implementation method for extracting the Modis metadata in the step S4 is as follows: importing an hdf file path in a c + + environment through a gdal library; calling a GDNLOpen () function to open an HDF data set and judging whether the data set is empty or not; if the data reading is null, the data reading fails; and if not, calling a GetMetadata () function to acquire the metadata character string.
4. The method for automatically releasing remote sensing data and monitoring the online process according to claim 1, wherein the method comprises the following steps: the specific implementation method for extracting metadata in step S4 Landsat8 is as follows: and directly reading the compressed packet through the java IO stream to obtain the metadata, txt file.
5. The method for automatically releasing remote sensing data and monitoring the online process according to claim 1, wherein the method comprises the following steps: the remote sensing data processing tool process scheduling design of the step S6 specifically comprises the following steps: firstly, related data are crawled regularly through a data crawling link, the acquired data are sent to a data catalog, catalog classification is carried out after the data are regulated according to rules, and after execution is finished, a parallel gateway is entered to divide a business flow into two sequential flows; one sequence flow enters a remote sensing data publishing link, and the other sequence flow enters a remote sensing metadata extraction link; and simultaneously carrying out two sequential flows until the two sequential flows run through a second parallel gateway, and then converging to finish data storage.
6. The method for automatically releasing remote sensing data and monitoring the online process according to claim 1, wherein the method comprises the following steps: in the step S2, two modules, namely a data module and a cache slice module, are set in the data distribution.
7. The method for automatically releasing remote sensing data and monitoring the online process according to claim 1, wherein the method comprises the following steps: the data module is provided with six sub-modules of layer preview, a working area, data storage, layers, a layer group, a pattern and the like.
8. The method for automatically releasing remote sensing data and monitoring the online process according to claim 1, wherein the method comprises the following steps: the cache slicing module is provided with four sub-modules of layer slicing, cache setting, grid setting and disk quota.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110354911.4A CN113220807A (en) | 2021-04-01 | 2021-04-01 | Automatic remote sensing data publishing and online process monitoring method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110354911.4A CN113220807A (en) | 2021-04-01 | 2021-04-01 | Automatic remote sensing data publishing and online process monitoring method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113220807A true CN113220807A (en) | 2021-08-06 |
Family
ID=77086247
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110354911.4A Pending CN113220807A (en) | 2021-04-01 | 2021-04-01 | Automatic remote sensing data publishing and online process monitoring method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113220807A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106210064A (en) * | 2016-07-15 | 2016-12-07 | 桂林电子科技大学 | A kind of based on the controllable P 2 P extensive remotely-sensed data distribution method authorized |
| WO2019200750A1 (en) * | 2018-04-20 | 2019-10-24 | 平安科技（深圳）有限公司 | Report monitoring method, apparatus, computing device and storage medium |
| CN111274294A (en) * | 2020-01-09 | 2020-06-12 | 中国科学院计算机网络信息中心 | Universal distributed heterogeneous data integrated logic convergence organization, release and service method and system |
| US20200210421A1 (en) * | 2018-12-29 | 2020-07-02 | Wuhan University | Method of storing remote sensing big data in hbase database |
| CN111723221A (en) * | 2020-06-19 | 2020-09-29 | 珠江水利委员会珠江水利科学研究院 | Mass remote sensing data processing method and system based on distributed architecture |
-
2021
- 2021-04-01 CN CN202110354911.4A patent/CN113220807A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106210064A (en) * | 2016-07-15 | 2016-12-07 | 桂林电子科技大学 | A kind of based on the controllable P 2 P extensive remotely-sensed data distribution method authorized |
| WO2019200750A1 (en) * | 2018-04-20 | 2019-10-24 | 平安科技（深圳）有限公司 | Report monitoring method, apparatus, computing device and storage medium |
| US20200210421A1 (en) * | 2018-12-29 | 2020-07-02 | Wuhan University | Method of storing remote sensing big data in hbase database |
| CN111274294A (en) * | 2020-01-09 | 2020-06-12 | 中国科学院计算机网络信息中心 | Universal distributed heterogeneous data integrated logic convergence organization, release and service method and system |
| CN111723221A (en) * | 2020-06-19 | 2020-09-29 | 珠江水利委员会珠江水利科学研究院 | Mass remote sensing data processing method and system based on distributed architecture |
Non-Patent Citations (1)
| Title |
|---|
| 焦汉科;马林兵;邓孺孺;梁业恒;魏慧丽: "多源遥感数据管理与远程处理集成实现", 测绘科学, vol. 40, no. 11, 31 December 2015 (2015-12-31) * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Santos et al. | A big data system supporting bosch braga industry 4.0 strategy | |
| CN105554070B (en) | A method of based on police service large data center Service and Construction | |
| CN112115198B (en) | Urban remote sensing intelligent service platform | |
| CN101743542B (en) | Collecting and presenting temporal-based action information | |
| CN102508706B (en) | Multi-source data integrating platform and establishing method thereof | |
| CN112199433A (en) | Data management system for city-level data middling station | |
| CN111176867B (en) | Data sharing exchange and open application platform | |
| CN101908165A (en) | Geographic information system (GIS)-based industrial cluster information integration service system and method | |
| CN109271382A (en) | A kind of data lake system towards full data shape opening and shares | |
| CN106202346A (en) | A kind of data load and clean engine, dispatch and storage system | |
| CN104679827A (en) | Big data-based public information association method and mining engine | |
| LU505379B1 (en) | Method for managing information data based on big data | |
| CN102508919A (en) | Data processing method and system | |
| CN113741883B (en) | RPA lightweight data middling station system | |
| CN112883001A (en) | Data processing method, device and medium based on marketing and distribution through data visualization platform | |
| CN111352982A (en) | Manpower extraction analysis system based on big data | |
| CN108287889A (en) | A kind of multi-source heterogeneous date storage method and system based on elastic table model | |
| CN101441645B (en) | The System and method for of technical data analysis | |
| CN113220807A (en) | Automatic remote sensing data publishing and online process monitoring method | |
| CN115664785A (en) | Big data platform data desensitization system | |
| CN116136842A (en) | Information resource integration method based on big data mode | |
| CN111797084B (en) | Data coding through mark inspection method and system based on weapon equipment test flow | |
| CN114331387A (en) | Cultural relic intelligent integrated service management system | |
| CN113934796A (en) | Database subsystem for underground water application service system and data query method | |
| CN113886465A (en) | Big data analysis platform for automobile logistics |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |