CN109344192A - A kind of optimization CIMISS Database Systems and its adaptation method - Google Patents
A kind of optimization CIMISS Database Systems and its adaptation method Download PDFInfo
- Publication number
- CN109344192A CN109344192A CN201811241035.9A CN201811241035A CN109344192A CN 109344192 A CN109344192 A CN 109344192A CN 201811241035 A CN201811241035 A CN 201811241035A CN 109344192 A CN109344192 A CN 109344192A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- dpc
- cimiss
- optimization
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1034—Reaction to server failures by a load balancer
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Multi Processors (AREA)
Abstract
The invention discloses a kind of optimization CIMISS Database Systems, including distributed data base, ORACLE database, DPC, SOD and MUSIC, distributed data base and ORACLE database are double libraries and use, mutually redundant structure；DPC, SOD and MUSIC are connect with distributed data base and ORACLE database communication.Distributed data base includes load-balanced server, engine server, database node and data network switch；Load-balanced server is connect with meteorological data uniform service interface communication；And the adaptation method based on optimization CIMISS Database Systems, including DPC adaptation scheme, SOD adaptation scheme and MUSIC adaptation scheme；The present invention solves the problems, such as that the timeliness of the data query of the existing technology in the increased situation of mass data is deteriorated, search efficiency is not high and the expensive caused extension cost performance of minicomputer is low.
Description
Technical field
The invention belongs to database technical fields, and in particular to a kind of optimization CIMISS Database Systems and its adaptation side
Method.
Background technique
CIMISS (China Integrated Meteorological Information Service System) is
The abbreviation of comprehensive national Meteorological Information Sharing Platform, the system collection data collection and distribution, quality control are generated with product, are stored
Management, shared service, business monitoring are deployed in national and 31 provinces in one, start to put into national each province in 2016
Service operation.Currently, CIMISS has realized the unified storage management of more than 300 basic meteorological datas, and provide real time history one
Body long sequence data online service.A national, provincial level weather service of MICAPS4, CIPAS, MDOS etc. more than 100 is supported to answer
With becoming the basic business system of modern weather business system.
Data storage management system (Service-Oriented Data Storage System, abbreviation in CIMISS
SOD), core is one " weather information database system " of building, this Database Systems is that upstream data is received and dispatched, at processing
Connecting points and turning point between reason and the shared service system in downstream.During service operation, up-stream system is by data source
Source is constantly transmitted to the system, and after the format storage of standardization, the system is by quickly downstream system is issued data and mentioned
For data service.Therefore, the storage capacity of SOD system, operational efficiency and stability will directly influence the entire data of CIMISS
The stability of process flow operation and external data service capabilities.
The prior art has the following problems:
(1) in terms of system operation, with the propulsion that Meteorological Modernization is built, network planning of standing mould gradually expands, and observes frequency
Constantly improve, the detection information type and magnitude for accessing CIMISS increase obviously, cause meteorological data storage timeliness low；
(2) with the increase of CIMISS support meteorological service system quantity, the Share that prior art database uses
Disk framework is limited by disk I/O bottleneck, and system performance difficulty is substantially improved, all kinds of in the user data requests of high concurrent
The timeliness and efficiency of data query are lower, and the especially longer statistics class inquiry of time series span, service effectiveness is difficult to reach
To the demand of user；
(3) node in the Share Disk framework that prior art database uses is all minicomputer, more due to price
Valuableness, the sexual valence being extended to it is relatively low, and when memory interface reaches saturation, can not be extended, further
Cause the efficiency data query of CIMISS not high.
Summary of the invention
For above-mentioned deficiency in the prior art, the present invention provides a kind of data loading timeliness, and high, saving fund is thrown
Enter and optimization CIMISS Database Systems and its adaptation method that expansion is high, improve CIMISS Database Systems performance,
The timeliness for improving data query solves the CIMISS data of the existing technology in the increased situation of mass data and looks into
The timeliness of inquiry is deteriorated, search efficiency is not high and minicomputer it is expensive caused by the low problem of extension cost performance.
In order to achieve the above object of the invention, the technical solution adopted by the present invention are as follows:
A kind of optimization CIMISS Database Systems, including distributed data base, ORACLE database, data mart modeling processing system
System DPC, data storage management system SOD and meteorological data uniform service interface MUSIC, distributed data base and ORACLE number
It is double libraries according to library and uses, mutually redundant structure；
DPC, SOD and MUSIC are connect with distributed data base and ORACLE database communication.
Distributed data base use Share Nothing framework, including successively communicate to connect load-balanced server, draw
Hold up server, database node and data network switch；Load-balanced server and meteorological data uniform service interface communication
Connection.
Further, DPC, for providing the storage process of structural data；
SOD, for providing the storage process of unstructured data；
MUSIC, for providing the data access service and Application Programming Interface of unification, standard, and it is each for state, province and district, county
Grade application system provides unique data access service.
Further, load-balanced server eliminates service for optimizing distribution of the access request between server group
Laod unbalance between device, and the operation conditions of server is monitored, by the access of the database node of operation exception
Request is transferred on the database node of normal work；
Engine server is mounted on multiple database nodes using distribution, for by the performance of each database node
Information and loading condition constantly pass through message mechanism and notify to load-balanced server；
Database node, for storing unstructured data and structural data；
Data network switch, the data information for completing between database node switch through.
Further, database node is expansible, and extend new database node respectively with engine server sum number
It is communicated to connect according to network switch.
A kind of adaptation method based on optimization CIMISS Database Systems, fits DPC, SOD and MUSIC accordingly
It is adjusted with property, realizes and distributed data base is embedded in CIMISS, including DPC adaptation scheme, SOD adaptation scheme and MUSIC
Adaptation scheme.
Further, DPC adaptation scheme includes the following steps:
A-1: extension DPC node01 node configuration simplifies the process of two destinations of data loading request distribution；
A-2: independent in node03, node04 and node05 node difference installation and deployment of DPC according to the DPC after extension
Structural data be put in storage process, and receive DPC host process transmission message；
A-3: the DPC process message received is decoded, the location information of file is obtained；
A-4: according to the location information of file, distributed data base is write data into.
Further, SOD adaptation scheme, to realize unstructured data in ORACLE database by data synchronization means
Asynchronous refresh is to distributed data base.
Further, MUSIC adaptation scheme includes the following steps:
B-1: data source is changed to distributed data base from ORACLE database；
B-2: user access request is parsed, and forms the SQL statement of retrieval；
B-3: according to the JDBC of SQL statement and distributed data base, the access to distributed data base is realized in driving.
This programme has the beneficial effect that
(1) present invention be directed to CIMISS database performance bottleneck point -- database part carry out upgrading, using point
Cloth database substantially enhances the storage and service ability of its data, improves the efficiency of data query；
(2) distributed data base and original ORACLE database form double libraries and use, mutually redundant pattern, improve whole
The storage capacity and service ability of a CIMISS system, improve the timeliness of data loading；
(3) distributed data base uses Share Nothing framework, melts operation and storage one, generallys use cheap
Equipment stacking goes out the group system of high-performance, high scalability and High Availabitity, has preferable extendibility and cost performance, and save
Capital investment.
Detailed description of the invention
Fig. 1 show optimization CIMISS Database Systems block diagram；
Fig. 2 show distributed database structure block diagram；
Fig. 3 show DPC adaptation scheme flow chart；
Fig. 4 show MUSIC adaptation scheme flow chart.
Specific embodiment
Carry out detailed description of the present invention illustrative embodiments with reference to the drawings.It should be appreciated that shown in attached drawing and
The embodiment of description is only exemplary, it is intended that is illustrated the principle and spirit of the invention, and is not limited model of the invention
It encloses.
A kind of optimization CIMISS Database Systems, as shown in Figure 1, including distributed data base, ORACLE database, data
Process and treat system DPC, data storage management system SOD and meteorological data uniform service interface MUSIC, distributed data base
It is double libraries with ORACLE database and uses, mutually redundant structure；
DPC, SOD and MUSIC are connect with distributed data base and ORACLE database communication.
Distributed data base uses Share Nothing framework, as shown in Fig. 2, equal including the load successively communicated to connect
Weigh server, engine server, database node and data network switch；Load-balanced server and the unified clothes of meteorological data
Business interface communication connection.
In the present embodiment, DPC, for providing the storage process of structural data；
SOD, for providing the storage process of unstructured data；
MUSIC, for providing the data access service and Application Programming Interface of unification, standard, and it is each for state, province and district, county
Grade application system provides unique data access service.
In the present embodiment, load-balanced server eliminates clothes for optimizing distribution of the access request between server group
Laod unbalance between business device, and the operation conditions of server is monitored, by the visit of the database node of operation exception
Ask that request is transferred on the database node of normal work；
Engine server is mounted on multiple database nodes using distribution, for by the performance of each database node
Information and loading condition constantly pass through message mechanism and notify to load-balanced server；
Database node, for storing unstructured data and structural data；
Data network switch, the data information for completing between database node switch through.
In the present embodiment, database node is expansible, and extend new database node respectively with engine server and
The connection of data network switch communication.
A kind of adaptation method based on optimization CIMISS Database Systems, fits DPC, SOD and MUSIC accordingly
It is adjusted with property, realizes and distributed data base is embedded in CIMISS, including DPC adaptation scheme, SOD adaptation scheme and MUSIC
Adaptation scheme.
In the present embodiment, DPC adaptation scheme, as shown in figure 3, including the following steps:
A-1: extension DPC node01 node configuration simplifies the process of two destinations of data loading request distribution；
A-2: independent in node03, node04 and node05 node difference installation and deployment of DPC according to the DPC after extension
Structural data be put in storage process, and receive DPC host process transmission message；
A-3: the DPC process message received is decoded, the location information of file is obtained；
A-4: according to the location information of file, distributed data base is write data into.
In the present embodiment, SOD adaptation scheme, to realize unstructured number in ORACLE database by data synchronization means
According to asynchronous refresh to distributed data base.
In the present embodiment, MUSIC adaptation scheme, as shown in figure 4, including the following steps:
B-1: data source is changed to distributed data base from ORACLE database；
B-2: user access request is parsed, and forms the SQL statement of retrieval；
B-3: according to the JDBC of SQL statement and distributed data base, the access to distributed data base is realized in driving.
That the present invention provides a kind of data loading timeliness is high, saves capital investment and optimization CIMISS that expansion is high
Database Systems and its adaptation method improve CIMISS Database Systems performance, improve the timeliness of data query, solve
The timeliness of the CIMISS data query of the existing technology in the increased situation of mass data is deteriorated, search efficiency it is not high with
And minicomputer it is expensive caused by the low problem of extension cost performance.
Claims (8)
1. a kind of optimization CIMISS Database Systems, which is characterized in that including distributed data base, ORACLE database, data
Process and treat system DPC, data storage management system SOD and meteorological data uniform service interface MUSIC, the distribution number
It is double libraries according to library and ORACLE database and uses, mutually redundant structure；
Described DPC, SOD and MUSIC are connect with distributed data base and ORACLE database communication.
The distributed data base use Share Nothing framework, including successively communicate to connect load-balanced server, draw
Hold up server, database node and data network switch；The load-balanced server and meteorological data uniform service interface
Communication connection.
2. optimization CIMISS Database Systems according to claim 1, which is characterized in that the DPC, for providing structure
Change the storage process of data；
The SOD, for providing the storage process of unstructured data；
The MUSIC, for providing the data access service and Application Programming Interface of unification, standard, and it is each for state, province and district, county
Grade application system provides unique data access service.
3. optimization CIMISS Database Systems according to claim 1, which is characterized in that the load-balanced server,
For optimizing distribution of the access request between server group, the laod unbalance between server is eliminated, and to server
Operation conditions is monitored, and the access request of the database node of operation exception is transferred to the database node of normal work
On；
The engine server is mounted on multiple database nodes using distribution, for by the performance of each database node
Information and loading condition constantly pass through message mechanism and notify to load-balanced server；
The database node, for storing unstructured data and structural data；
The data network switch, the data information for completing between database node switch through.
4. optimization CIMISS Database Systems according to claim 1, which is characterized in that the database node can expand
Exhibition, and the new database node extended is communicated to connect with engine server and data network switch respectively.
5. a kind of adaptation method based on any optimization CIMISS Database Systems of claim 1-4, which is characterized in that right
DPC, SOD and MUSIC carry out corresponding suitability adjustment, realize and distributed data base is embedded in CIMISS, including DPC suitable
With scheme, SOD adaptation scheme and MUSIC adaptation scheme.
6. optimization CIMISS Database Systems according to claim 5, which is characterized in that the DPC adaptation scheme, including
Following steps:
A-1: extension DPC node01 node configuration simplifies the process of two destinations of data loading request distribution；
A-2: according to the DPC after extension, the independent knot of installation and deployment is distinguished in node03, node04 and node05 node of DPC
Structure data loading process, and receive the message of DPC host process transmission；
A-3: the DPC process message received is decoded, the location information of file is obtained；
A-4: according to the location information of file, distributed data base is write data into.
7. optimization CIMISS Database Systems according to claim 5, which is characterized in that the SOD adaptation scheme, is logical
It crosses data synchronization means and realizes that unstructured data asynchronous refresh is to distributed data base in ORACLE database.
8. optimization CIMISS Database Systems according to claim 5, which is characterized in that the MUSIC adaptation scheme, packet
Include following steps:
B-1: data source is changed to distributed data base from ORACLE database；
B-2: user access request is parsed, and forms the SQL statement of retrieval；
B-3: according to the JDBC of SQL statement and distributed data base, the access to distributed data base is realized in driving.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811241035.9A CN109344192B (en) | 2018-10-24 | 2018-10-24 | Optimized CIMISS database system and adaptation method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811241035.9A CN109344192B (en) | 2018-10-24 | 2018-10-24 | Optimized CIMISS database system and adaptation method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109344192A true CN109344192A (en) | 2019-02-15 |
| CN109344192B CN109344192B (en) | 2020-03-10 |
Family
ID=65311796
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811241035.9A Active CN109344192B (en) | 2018-10-24 | 2018-10-24 | Optimized CIMISS database system and adaptation method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109344192B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110120972A (en) * | 2019-04-19 | 2019-08-13 | 浙江省气象台 | Distributed storage grid meteorological data read method and reading system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102201010A (en) * | 2011-06-23 | 2011-09-28 | 清华大学 | Distributed database system without sharing structure and realizing method thereof |
| CN102968310A (en) * | 2012-12-05 | 2013-03-13 | 武汉烽火普天信息技术有限公司 | Integrated high-performance application software architecture and construction method thereof |
| CN104376053A (en) * | 2014-11-04 | 2015-02-25 | 南京信息工程大学 | Storage and retrieval method based on massive meteorological data |
| CN104915450A (en) * | 2015-07-01 | 2015-09-16 | 武汉大学 | HBase-based big data storage and retrieval method and system |
| CN105045929A (en) * | 2015-08-31 | 2015-11-11 | 国家电网公司 | MPP architecture based distributed relational database |
-
2018
- 2018-10-24 CN CN201811241035.9A patent/CN109344192B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102201010A (en) * | 2011-06-23 | 2011-09-28 | 清华大学 | Distributed database system without sharing structure and realizing method thereof |
| CN102968310A (en) * | 2012-12-05 | 2013-03-13 | 武汉烽火普天信息技术有限公司 | Integrated high-performance application software architecture and construction method thereof |
| CN104376053A (en) * | 2014-11-04 | 2015-02-25 | 南京信息工程大学 | Storage and retrieval method based on massive meteorological data |
| CN104915450A (en) * | 2015-07-01 | 2015-09-16 | 武汉大学 | HBase-based big data storage and retrieval method and system |
| CN105045929A (en) * | 2015-08-31 | 2015-11-11 | 国家电网公司 | MPP architecture based distributed relational database |
Non-Patent Citations (2)
| Title |
|---|
| 季永华等: "CIMISS中气象观测资料处理入库效率优化方法", 《气象科技》 * |
| 赵芳等: "全国综合气象信息共享平台架构设计技术特征", 《应用气象学报》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110120972A (en) * | 2019-04-19 | 2019-08-13 | 浙江省气象台 | Distributed storage grid meteorological data read method and reading system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109344192B (en) | 2020-03-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109492040B (en) | System suitable for processing mass short message data in data center | |
| CN104965850B (en) | A kind of database high availability implementation method based on open source technology | |
| CN104734915B (en) | A kind of concurrent dynamic emulation method of Multi net voting of compound multi-process multithreading | |
| CN109933631A (en) | Distributed parallel database system and data processing method based on Infiniband network | |
| CN105404701A (en) | Peer-to-peer network-based heterogeneous database synchronization method | |
| CN1783068B (en) | Method for implementing fault diagnosis and monitoring database service | |
| CN104618455B (en) | A kind of general caching system and method | |
| CN104778188A (en) | Distributed device log collection method | |
| CN102567932A (en) | Method, device and system for managing comprehensive resources | |
| CN102937964B (en) | Intelligent data service method based on distributed system | |
| CN102932846A (en) | Data management system for distributed heterogeneous sensing network and data management method for data management system | |
| CN101883094B (en) | Embedded universal plug and play industrial monitoring network database service system | |
| CN103595799A (en) | Method for achieving distributed shared data bank | |
| CN109344192A (en) | A kind of optimization CIMISS Database Systems and its adaptation method | |
| CN118152450A (en) | Data communication method, equipment and medium for unidirectional network isolation environment | |
| CN117076426A (en) | Traffic intelligent engine system construction method and device based on flow batch integration | |
| CN103198041A (en) | Method for establishing IEC61970 GID (general interface definition) interface | |
| CN113360576A (en) | Power grid mass data real-time processing method and device based on Flink Streaming | |
| CN110019085A (en) | A kind of distributed time series database based on HBase | |
| CN111367880A (en) | Universal real-time data storage management system and implementation method thereof | |
| CN115062092B (en) | Database access method, device, system and storage medium | |
| CN115934819A (en) | Universal distributed expansion method for industrial time sequence database | |
| CN116431324A (en) | Edge system based on Kafka high concurrency data acquisition and distribution | |
| Pu et al. | Practical implementation of an OPC UA multi-server aggregation and management architecture for IIoT | |
| CN117131080A (en) | Data processing platform based on stream processing and message queue |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |