CN113076361A - Method for realizing massive meteorological data unified service interface based on big data - Google Patents
Method for realizing massive meteorological data unified service interface based on big data Download PDFInfo
- Publication number
- CN113076361A CN113076361A CN202110285941.4A CN202110285941A CN113076361A CN 113076361 A CN113076361 A CN 113076361A CN 202110285941 A CN202110285941 A CN 202110285941A CN 113076361 A CN113076361 A CN 113076361A
- Authority
- CN
- China
- Prior art keywords
- data
- interface
- service interface
- unified service
- meteorological
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/547—Remote procedure calls [RPC]; Web services
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Stored Programmes (AREA)
Abstract
The invention discloses a method for realizing a massive meteorological data unified service interface based on a big data technology, which comprises the following steps: firstly, establishing interfaces adapting to different data formats based on various data resources in a meteorological data environment, and acquiring and processing various types of data; secondly, the interface is subjected to standardized packaging according to standards such as input parameters, output formats and the like of the interface, and a standardized calling interface is provided; thirdly, calling a standardized calling interface to realize the meteorological data unified service according to different service types, wherein the meteorological data unified service interface is divided into a general interface and a customized interface; and finally, calling a meteorological data unified service interface by the user to acquire different services for providing unified data support services for each application system.
Description
Technical Field
The invention belongs to the technical field of meteorological data management and application, and particularly relates to a method for realizing a massive meteorological data unified service interface based on big data.
Background
With the development of the meteorological observation system, more and more meteorological data are obtained, the types are more and more complicated, the formats of the data are also various, and the technology of the back-end data organization management mode is continuously upgraded, so that unnecessary troubles are brought to the front-end user for calling the data.
The traditional data usage mode of the current weather application system is to directly access a database, such as: MICAPS, CIPAS, SWAN, provincial applications, and the like. When the database is directly accessed, the application is tightly coupled with the data storage, and the content, format and storage mode of the data resource are changed, the application is required to be changed accordingly. The data format is changed and new data is continuously increased due to the reform and development of observation services; new storage technologies are continuously developed, and storage systems are continuously upgraded, which all bring impact to the adaptation of data calls by applications.
With the development of internet technology in recent years, data resources and supporting capacity can be modified into services to construct a flexible and extensible system architecture. In the meteorological industry, in recent years, national meteorological information centers and individual province and city information centers such as Guangdong, Sichuan and Chongqing, respective service interfaces are developed based on local databases to support local application, and a good effect is achieved. In 2015, the central meteorological office establishes a unified data environment of the national province, and how to provide unified data support service for application systems of all levels of the national province, the city and the county based on massive meteorological data resources in the meteorological data environment is a problem to be solved urgently at present.
Disclosure of Invention
Aiming at the existing problems, the invention provides a method for realizing a massive meteorological data unified service interface based on a big data technology.
The technical solution for realizing the purpose of the invention is as follows:
a realization method of a massive meteorological data unified service interface based on big data is characterized by comprising the following steps:
step 1: establishing interfaces adapting to different data formats based on various data resources in the meteorological data environment, and acquiring and processing various types of data;
step 2: the interface is subjected to standardized packaging according to standards such as input parameters, output formats and the like of the interface, and a standardized calling interface is provided;
and step 3: calling a standardized calling interface to realize meteorological data unified service according to different service types, wherein the meteorological data unified service interface is divided into a general interface and a customized interface;
and 4, step 4: and the user calls the meteorological data unified service interface to obtain different services for providing unified data support services for each application system.
Compared with the prior art, the method has the following beneficial effects:
firstly, the meteorological data Unified Service Interface music (meteorological Unified Service Interface Community) provided by the invention is based on a distributed architecture, follows a Unified Service Interface standard, adopts technologies such as multithreading pooling, data caching, Web Service and communication middleware, and the like, shields a bottom layer complex data storage technology and various data formats, and realizes Unified transparent access of multi-source and heterogeneous data.
Second, to meet the call requirements of different users, MUSIC provides cross-platform SDK, Web Service, restful api, and scripting services. Through MUSIC, the meteorological data in the meteorological data environment can be searched, counted, cut, extracted and the like, and the application can quickly access the meteorological data.
Thirdly, in order to meet the popularization and deployment requirements of the application, the MUSIC provides a synchronous flow of each sub-node customized interface, ensures the consistency of each node customized interface, and realizes the rapid popularization and deployment of the application in each sub-node.
Fourthly, in order to meet the requirement of users on understanding of calling interface information, the MUSIC provides online data interface help and a trial website, and the online data interface help and the trial website comprise interface parameter description, assignment styles, development examples, return data styles and the like, so that the development and application difficulty is reduced.
At present, MUSIC is not only deployed in a national-level service center, but also popularized and applied to a provincial service center, supports four-level application in provincial, city and county of China, and has more than 10 hundred million visits per year.
Drawings
FIG. 1 is a schematic flow chart of a method for implementing a massive meteorological data unified service interface based on big data;
FIG. 2 is a schematic diagram of a data flow of an implementation of the present invention;
FIG. 3 is a schematic diagram of a service interface provided in the present invention;
FIG. 4 is a diagram illustrating an interface information distribution website according to an embodiment;
Detailed Description
In order to make those skilled in the art better understand the technical solution of the present invention, the following further describes the technical solution of the present invention with reference to the drawings and the embodiments.
Referring to fig. 1, the implementation method of the massive meteorological data unified service interface based on big data provided by the invention comprises the following steps:
step 1: establishing interfaces adapting to different data formats based on various data resources in the meteorological data environment, and acquiring and processing various types of data;
step 2: the interface is subjected to standardized packaging according to standards such as input parameters, output formats and the like of the interface, and a standardized calling interface is provided;
and step 3: calling a standardized calling interface to realize meteorological data unified service according to different service types, wherein the meteorological data unified service interface is divided into a general interface and a customized interface;
and 4, step 4: and the user calls the meteorological data unified service interface to obtain different services for providing unified data support services for each application system.
Further, as can be seen from fig. 2, the specific steps of establishing interfaces adapted to different data formats, acquiring and processing various types of data in step 1 include:
step 11: establishing different functional interfaces which comprise data acquisition, data statistics, format conversion, calendar year synchronization, lattice point data analysis, data caching and data writing interfaces;
step 12: for various data resources in a meteorological data environment, the data resources comprise storage forms such as a matched distributed database, a distributed file system and the like, and data are read and written in through a data acquisition and data writing interface;
step 13: based on the acquired data, the data is processed by utilizing the established data statistics, format conversion, calendar year synchronization, lattice point data analysis and data cache interface to obtain standardized data.
Further, the standard for establishing the standardized call interface in step 2 is an access format, an interface parameter, a return format and a data dimension.
Further, referring to fig. 3, it can be seen that the meteorological data unified service interface call form established in step 3 includes an SDK, a web service/restful api and a script, which is convenient for a user to use for different system platforms, development languages, development habits and application modes, and additionally provides an interface information publishing website;
as can be appreciated with reference to fig. 4, the interface information distribution website can provide detailed instructions and guidance to the user to use the interface. It describes for each interface its calling method, parameter information, example code and example results for the respective language, etc. The data list which can be called by the issuing interface describes the basic information, the element list, the use instruction and the like of each type of data. And auxiliary information required by interface calling and data use such as the characteristic value definition, the identification code and the return code definition of the meteorological data is also released on the website in a centralized way.
Furthermore, because different application systems have differences in data acquisition and application modes, the frequency of calling interfaces and the amount of data acquired by each calling are different; in addition, the data acquirer has different calling habits, wherein some habits are called in a programming mode, and some habits are called in a script mode; therefore, the meteorological data unified service interface of the invention provides 3 types of interface service modes, and meets different application modes:
the SDK is a multi-language client development kit realized based on an open source data transmission middleware zeroIce, is used for calling data and files with large data volume, has high transmission efficiency, and needs to be developed based on a downloaded SDK by a user. Supports the mainstream system platforms such as AIX, Linux, Windows and the like, and provides multi-language versions such as C/C + +, Fortran, Java, C #, PHP, Python and the like. The method is mainly suitable for background processing systems, such as numerical mode systems; in the SDK, an interface is only responsible for transmitting parameters defined by a standard to a server side and receiving result data in a standard format, and the stability of the SDK can be guaranteed through simple function positioning;
the web service/RESTful API is based on a standard SOAP protocol, is simple to call, does not need to load a client, can ensure the stability of the client by the server based on a standard issuing interface, supports multiple serialized return formats such as XML, JSON, JSONP, HTML and TEXT, and is mainly used for an interactive platform;
the script service interface comprises a calling tool and a script parameter template, and a user sets a retrieval condition and configures a script according to an application scene, and can acquire data by executing the calling tool.
At the granularity of service interface definition, there are generally two options: a generic interface and a custom interface. The former realizes a plurality of interfaces with limited but strong functions, the parameters of the interfaces are indefinite, and a user calls the interfaces by selecting a plurality of parameters in a parameter list for assignment; the latter develops and provides interfaces for each application scene, and the parameters of each interface are limited and fixed; the mode is low in use threshold, and users only need to assign values and call one by one according to fixed parameters.
During actual deployment, the universal interface and the customized interface are comprehensively applied, the universal interface is used, and the customized interface is issued facing a user; each custom interface configures its parameter combination and associates with a general interface through metadata. The user calls the custom interface, and the system forwards the request and its parameters to the corresponding generic interface, which processes and returns the data. Based on the design, firstly, a service interface aiming at an application scene can be provided for a user, the user is convenient to use, and secondly, a new service interface can be issued for a newly added application scene through configuration, so that the user requirement can be quickly responded.
Further, the meteorological data unified service interface is in a distributed deployment mode, and the deployment steps are as follows:
s1: each customized meteorological data unified service interface is regarded as metadata, and a metadata center node and a plurality of metadata sub-nodes are established;
s2: when any sub-node newly adds or updates a service interface, automatically submitting metadata updating information to a central node;
s3: the administrator conducts auditing at the central node, issues a new service interface after the auditing is passed, and sends an updating notice to each sub-node;
s4: and after receiving the notification, each sub-node synchronizes and opens the access of the new service interface.
Taking national province unified data service as an example, a distributed metadata synchronization mechanism is established to realize the national province synchronization of the interfaces. Firstly, establishing a metadata center node, wherein the state level and 31 provinces are subnodes; a service interface is newly added or updated at any sub-node, and the metadata change application is automatically submitted to the central node; the administrator checks the central node and issues the central node through a back interface; each sub-node receives the interface updating notice and performs selection and synchronization according to the local application requirement.
Those not described in detail in this specification are within the skill of the art. Although the present invention has been described in detail with reference to the foregoing embodiments, it will be apparent to those skilled in the art that various changes in the embodiments and modifications of the invention can be made, and equivalents of some features of the invention can be substituted, and any changes, equivalents, improvements and the like, which fall within the spirit and principle of the invention, are intended to be included within the scope of the invention.
Claims (6)
1. A realization method of a massive meteorological data unified service interface based on big data is characterized in that,
the method comprises the following steps:
step 1: establishing interfaces adapting to different data formats based on various data resources in the meteorological data environment, and acquiring and processing various types of data;
step 2: the interface is subjected to standardized packaging according to standards such as input parameters, output formats and the like of the interface, and a standardized calling interface is provided;
and step 3: calling a standardized calling interface to realize meteorological data unified service according to different service types, wherein the meteorological data unified service interface is divided into a general interface and a customized interface;
and 4, step 4: and the user calls the meteorological data unified service interface to obtain different services for providing unified data support services for each application system.
2. The method for implementing massive weather data unified service interface based on big data as claimed in claim 1, wherein the step 1 of establishing the interface adapted to different data formats, the specific steps of obtaining and processing various types of data include:
step 11: establishing different functional interfaces which comprise data acquisition, data statistics, format conversion, calendar year synchronization, lattice point data analysis, data caching and data writing interfaces;
step 12: for various data resources in a meteorological data environment, the data resources comprise storage forms such as a matched distributed database, a distributed file system and the like, and data are read and written in through a data acquisition and data writing interface;
step 13: and based on the acquired data, processing the data by utilizing the established data statistics, format conversion, calendar year synchronization, lattice point data analysis and data cache interface to obtain the data in the standardized format.
3. The method for implementing the massive meteorological data unified service interface based on the big data according to claim 1, wherein the standards for establishing the standardized calling interface in the step 2 are access format, interface parameters, return format and data dimension.
4. The method for implementing the massive weather data unified service interface based on the big data technology as claimed in claim 1, wherein the calling form of the weather data unified service interface established in step 3 includes three types, namely, an SDK, a web service/RESTful API and a script.
5. The implementation method of mass weather data unified service interface based on big data according to claim 4, wherein the SDK is a multi-language client development kit implemented based on open source data transmission middleware zeroIce, and is used for retrieving data and files with large data volume;
the web service/RESTful API is based on a standard SOAP protocol and is used for an interactive platform;
the script form interface comprises a calling tool and a script parameter template, and data can be acquired by executing the calling tool.
6. The method for implementing the massive meteorological data unified service interface based on the big data according to claim 1, wherein the meteorological data unified service interface is a distributed deployment mode, and the deployment steps are as follows:
s1: each customized meteorological data unified service interface is regarded as metadata, and a metadata center node and a plurality of metadata sub-nodes are established;
s2: when any sub-node newly adds or updates a service interface, automatically submitting metadata updating information to a central node;
s3: the administrator conducts auditing at the central node, issues a new service interface after the auditing is passed, and sends an updating notice to each sub-node;
s4: and after receiving the notification, each sub-node synchronizes and opens the access of the new service interface.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110285941.4A CN113076361A (en) | 2021-03-17 | 2021-03-17 | Method for realizing massive meteorological data unified service interface based on big data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110285941.4A CN113076361A (en) | 2021-03-17 | 2021-03-17 | Method for realizing massive meteorological data unified service interface based on big data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113076361A true CN113076361A (en) | 2021-07-06 |
Family
ID=76612953
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110285941.4A Pending CN113076361A (en) | 2021-03-17 | 2021-03-17 | Method for realizing massive meteorological data unified service interface based on big data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113076361A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103413008A (en) * | 2013-08-28 | 2013-11-27 | 北京经纬恒润科技有限公司 | Real-time emulation system based on distributed input/output (I/O) interfaces |
| CN108366109A (en) * | 2018-02-01 | 2018-08-03 | 王晓峰 | A kind of meteorological data numerical forecast cloud shared platform and data sharing method |
| CN108965007A (en) * | 2018-07-19 | 2018-12-07 | 北京车和家信息技术有限公司 | API gateway interface configures update method and device |
| CN110738586A (en) * | 2018-07-20 | 2020-01-31 | 南京洲源信息科技有限公司 | weather integrated service system based on CIMISS and comprehensive database data |
| CN110837492A (en) * | 2019-11-15 | 2020-02-25 | 中科院计算技术研究所大数据研究院 | Method for providing data service by multi-source data unified SQL |
| CN112035438A (en) * | 2020-09-01 | 2020-12-04 | 江苏风云科技服务有限公司 | Government affair big data platform system |
-
2021
- 2021-03-17 CN CN202110285941.4A patent/CN113076361A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103413008A (en) * | 2013-08-28 | 2013-11-27 | 北京经纬恒润科技有限公司 | Real-time emulation system based on distributed input/output (I/O) interfaces |
| CN108366109A (en) * | 2018-02-01 | 2018-08-03 | 王晓峰 | A kind of meteorological data numerical forecast cloud shared platform and data sharing method |
| CN108965007A (en) * | 2018-07-19 | 2018-12-07 | 北京车和家信息技术有限公司 | API gateway interface configures update method and device |
| CN110738586A (en) * | 2018-07-20 | 2020-01-31 | 南京洲源信息科技有限公司 | weather integrated service system based on CIMISS and comprehensive database data |
| CN110837492A (en) * | 2019-11-15 | 2020-02-25 | 中科院计算技术研究所大数据研究院 | Method for providing data service by multi-source data unified SQL |
| CN112035438A (en) * | 2020-09-01 | 2020-12-04 | 江苏风云科技服务有限公司 | Government affair big data platform system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20170329581A1 (en) | Open data protocol services in applications and interfaces across multiple platforms | |
| CN109408043B (en) | Program generation method and device | |
| CN108572963A (en) | Information acquisition method and device | |
| US20140108675A1 (en) | Subscription-Based Dynamic Content Optimization | |
| CN109683998A (en) | Internationalize implementation method, device and system | |
| KR20190039230A (en) | Method and system for server-side rendering of native content for presentations | |
| CN112035510B (en) | Method and system for inquiring complex data of information system | |
| CN103729463A (en) | Method for implementing full-text retrieval based on Lucene and Solr | |
| CN109614088B (en) | Form component generation method and device | |
| CN102999600A (en) | Method and system for automatically generating embedded database | |
| CN109902114A (en) | ES company-data multiplexing method, system, computer installation and storage medium | |
| CN105843867A (en) | Metadata model-based search method and search device based on metadata model | |
| CN112632136A (en) | Data statistical analysis method and device, electronic equipment and storage medium | |
| CN111966692A (en) | Data processing method, medium, device and computing equipment for data warehouse | |
| CN106648917B (en) | A kind of method and system that difference update is data cached | |
| CN109408481A (en) | Update method, device, electronic equipment and the readable medium of log collection rule | |
| CN110109983B (en) | Method and device for operating Redis database | |
| CN113076361A (en) | Method for realizing massive meteorological data unified service interface based on big data | |
| CN113760961A (en) | Data query method and device | |
| CN110764769B (en) | Method and device for processing user request | |
| CN101039320A (en) | Method and apparatus for providing an interface between a client application and a web service | |
| CN113515306B (en) | System transplanting method and device | |
| CN115686497A (en) | Business development data management method, development engine, electronic device, and storage medium | |
| CN113656433B (en) | Entity object expansion method, entity object expansion device, electronic equipment and storage medium | |
| Bigagli et al. | GI-Cat: A web service for dataset cataloguing based on ISO 19115 |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |