CN110083649A - It is a kind of based on cold, warm, dsc data data management system and data managing method - Google Patents
It is a kind of based on cold, warm, dsc data data management system and data managing method Download PDFInfo
- Publication number
- CN110083649A CN110083649A CN201910332128.0A CN201910332128A CN110083649A CN 110083649 A CN110083649 A CN 110083649A CN 201910332128 A CN201910332128 A CN 201910332128A CN 110083649 A CN110083649 A CN 110083649A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- management system
- server
- data management
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/211—Schema design and management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
Abstract
The invention discloses a kind of based on cold, warm, dsc data data management system and data managing method, including computer, at least one first server and at least one second server, wherein the first server includes the first database for storing dsc data and the second database for storing warm data；The second server includes the third database for storing cold data；The computer real-time reception and the monitoring data for showing tested equipment, are respectively stored into the first database, the second database and third database according to the data type of the monitoring data.Embodiment provided by the invention can be improved the efficiency of management, storage efficiency and search efficiency of data, improve software service ability and data service capabilities to the full extent by setting multi-layer, three-dimensional data management mode.
Description
Technical field
The present invention relates to technical field of data administration, more particularly to a kind of based on cold, warm, dsc data data management system
System and data managing method.
Background technique
With the development of electronic technology and information technology, the research and development and use of industrial equipment increasingly tend to enlarged, multiple
Hydridization and automation.The scale of change system is also increased with it with performance, and coupling is more between each subsystem in change system
Closely, inter-net communication is also more intensive, this makes equipment generate a large amount of, type multiplicity data in use.
Traditional equipment data management has very big defect, first is that equipment real time data state labile, data are placed on software
It is managed in memory, software once collapses memory and will discharge；Second is that the historical data analyzed is needed to there is text mostly afterwards
In this document, data management job costs are larger.It is managed by different level third is that equipment data management is not classified according to data characteristics
Reason, this is unable to satisfy present age equipment, particularly the multi-level of high-end equipment, diversification, flexible data management demand.How to have
Effect manages different types of equipment data, and to industrial equipment system research and development, more stringent requirements are proposed for this.
Summary of the invention
At least one to solve the above-mentioned problems, first aspect present invention provides a kind of based on cold, warm, dsc data data
Management system, including computer, at least one first server and at least one second server, wherein
The first server includes the first database for storing dsc data and the second number for storing warm data
According to library；
The second server includes the third database for storing cold data；
The computer real-time reception and the monitoring data for showing tested equipment, according to the data type of the monitoring data
It is respectively stored into the first database, the second database and third database.
Further, the first database is distributed memory database, and second database is relational data
Library, the third database are distributed column database.
Further, the first database includes primary database and at least one standby database, the preliminary data
Library synchronizes the data of the primary database.
Further, the first database is stored in the disk of the first server according to the period is pre-seted.
Further, the first database is Redis database；And/or
Second database is MySQL database, including multiple tables of data；And/or
The third database is the Hbase column database for including structured storage cluster.
Further, the first database and the second database are stored with Key-Value structure.
Further, the computer includes receiving unit, Data Management Unit and visualization, wherein
The receiving unit, for monitoring data described in real-time reception；
The Data Management Unit, for storing, retrieving, editing and deleting respectively according to the type of the monitoring data；
The visualization, for monitoring data described in real-time display.
Further, the computer further includes reading unit, and the reading unit includes reading the first database
First interface, read the second interface of second database and read the third interface of third database.
Second aspect of the present invention provides a kind of data managing method using data management system described in first aspect, packet
It includes:
Computer real-time reception and the monitoring data for showing tested equipment；
The computer is stored according to the type of the monitoring data: the dsc data in the monitoring data is stored
In the first database of first server, the warm data in the monitoring data are stored in the second data of first server
Cold data in the monitoring data is stored in the third database of second server by library.
Further, the method also includes:
The computer further includes receiving unit, Data Management Unit, visualization and reading unit, passes through the reading
Unit and visualization is taken to examine the data of the first database and/or the second database and/or third database
Rope, editor and deletion.
Beneficial effects of the present invention are as follows:
The present invention formulates a kind of based on cold, warm, dsc data data management system and data for problem existing at present
Management method, and by setting multi-layer, three-dimensional three layer data library structures, it is divided into cold number according to data usage frequency
According to, warm data and dsc data three types, it is respectively stored in first database, the second database and third database, can be solved
The certainly efficient management and lateral dilatation of the release loss problem of dsc data, the visualized management deficiency problem of warm data, cold data
Problem guarantees the efficiency of management, storage efficiency and search efficiency of data, utmostly to compensate for problem in the prior art
Upper raising software service ability and data service capabilities.
Detailed description of the invention
Specific embodiments of the present invention will be described in further detail with reference to the accompanying drawing.
Fig. 1 shows the structural block diagram of data management system described in one embodiment of the present of invention；
Fig. 2 shows the structural schematic diagrams of data management system described in one embodiment of the present of invention；
Fig. 3 shows the master/slave database structure schematic diagram of first database described in one embodiment of the present of invention；
Fig. 4 shows the structural schematic diagram of the data storage of third database described in one embodiment of the present of invention；
Fig. 5 shows the flow chart of data managing method described in one embodiment of the present of invention.
Specific embodiment
In order to illustrate more clearly of the present invention, the present invention is done further below with reference to preferred embodiments and drawings
It is bright.Similar component is indicated in attached drawing with identical appended drawing reference.It will be appreciated by those skilled in the art that institute is specific below
The content of description is illustrative and be not restrictive, and should not be limited the scope of the invention with this.
As shown in Figure 1, An embodiment provides a kind of based on cold, warm, dsc data data management system,
Including computer, at least one first server and at least one second server, wherein the first server includes being used for
Store the first database of dsc data and the second database for storing warm data；The second server includes for storing
The third database of cold data；The computer real-time reception and the monitoring data for showing tested equipment, according to the monitoring number
According to data type be respectively stored into the first database, the second database and third database.
In a specific example, it is illustrated by taking equipment Command control system as an example, as shown in Fig. 2, the calculating
Operation has data management software on machine, when computer receives equipment data, will equip according to the frequency of use of equipment data
Data are divided into cold, warm, hot three types, wherein dsc data refers to the higher data of real-time generated in equipment use process, temperature
Data refer to that the data that the intermittence such as configuration parameter uses in the data management system of change system, cold data refer mainly to equipment and use
The data for being used to carry out ex-post analysis that process generates.The characteristic of data, the data management system setting are equipped for three kinds
Two levels server, first server and second server；Three kinds of databases, for storing first database, the storage of dsc data
Second database of warm data and the third database of storage cold data.Wherein first database and the second data lab setting are in institute
It states in first server, third data lab setting is on the second server.
In equipment Command control system, the data management software for running data management system on the computer is flat
Platform receives a large amount of equipment data.
Wherein, it is related to the real-time report of equipment and assigns data to be dsc data, such as the real-time track of target, user are stood in real time
The real time datas such as order, instruction notification information, real-time weather information are turned in location, friendly neighbour user's real time information, sector.To solve heat
The storage problem of data establishes first database in first server, and the first database is distributed memory database,
Real-time dsc data is stored in memory database, realizes the visualized management of data.
To solve the problems, such as equipment real time data state labile, prevent from storing data in memory because software crash discharges
Data loss problem caused by data, in a preferred embodiment, as shown in figure 3, the first database includes main number
According to library and at least one standby database, the standby database synchronizes the data of the primary database.I.e. described first data
Library is designed by the way of primary database and standby database, to realize the High Availabitity of the real time data of first database
Property.Wherein, primary database is Master node, and all from master library, standby database is Slave section for the read-write of dsc data
Point recommends two spare node repositories of design, can also extend multiple, those skilled in the art should carry out according to actual needs
Design.The data of primary database are all replicated to standby database, for the stability for further ensuring that dsc data, described
One database is stored in the disk of the first server according to the period is pre-seted, and is preset between the time in the present embodiment
Every, such as 30 seconds, it is replicated according to the scheduled time interval, so that the consistency of data is kept, if primary database
Because unexpected cause to collapse, standby database is automatically switched by weight order as primary database to solve data loss problem.
In the present embodiment, the first database is Redis database, carries out data using Key-Value structure and deposits
Storage, data are stored in Redis lane database in the form serialized, and wherein Key value is unique index mark, and Value is directly deposited
Data after storing up data structure serializing.Therefore, the retrieval of dsc data can be carried out directly by the Key value of memory database
Inquiry, to realize the quick-searching of dsc data.
For the storage problem for solving warm data, the second database is established in first server, second database is
Relationship type row database, a large amount of configuration parameter data is stored in relational database, realizes the visualized management of database,
Support the structured management and complicated correlation inquiry of mass data.
In the present embodiment, as shown in Fig. 2, second database is Mysql relationship type row database, including multiple numbers
According to table, data storage is carried out using Key-Value structure, structure design is clear, is with good expansibility.Based on Mysql
The characteristic of relationship type row database, the equipment running environment information that the configuration information for being related to data management software and needs are bound
It is stored in the form of tables of data in the database, by realizing storage and reading to configuration information to data table handling.Example
Such as the types of equipment in equipment Command control system, communication protocol, user type, user site, user's messaging parameter, collaboration mark
Draw the data such as information, information processing configuration parameter, display control configuration parameter, equipment resume.
For the storage problem for solving cold data, third database is established on second server, the third database is
Distributed column database includes history in the storage problem of mass historical data, such as equipment Command control system for solving
Receive and dispatch the mass datas such as director data, the history air situation data that protocol data, the notification data of history transmitting-receiving, history are received and dispatched.
In the present embodiment, as shown in Fig. 2, the third database is the Hbase columns for including structured storage cluster
Different from the storage mode of the first database and the second database according to library, the Hbase column database is by a large amount of history numbers
It is stored according to according to column database cluster, by being stored in several tables of data for a large amount of history cold data load balancing, from
And realize the visualized management of data, while HBase database has good scalability, data with increment can expand
Storage is stored, and improves the recall precision of data.
It is worth noting that as shown in figure 4, the Hbase column database is deposited using Key-region structure progress data
Storage, but it is different from the management of dsc data, and the data volume of cold data is more huge, by the database of HBase to magnanimity cold data
Subregion is carried out by different dimensions.In the present embodiment, historical data involved in the equipment Command control system generally has
Certain timing, therefore using timestamp as partition boundaries.Data in tables of data are using timestamp+protocol number as Key
Value, the character string after the decoding of region (region) storing data.
In a preferred embodiment, the computer includes receiving unit, Data Management Unit and visualization,
The wherein receiving unit, for monitoring data described in real-time reception；The Data Management Unit, for according to the monitoring number
According to type store, retrieve, edit and delete respectively；The visualization, for monitoring data described in real-time display.It is based on
The receiving unit, Data Management Unit and visualization, foundation on the computer are based on web data management software
Platform, the platform can support the management service to data in three kinds of databases, can be dynamically to cold data, warm data and heat
Data three types data are visually increased, are deleted, are changed, are looked into.
In a preferred embodiment, the computer further includes reading unit, and the reading unit includes reading institute
The third for stating the first interface of first database, the second interface for reading second database and reading third database connects
Mouthful.It is realized using distinct interface and dsc data, warm data and cold data is managed and maintained, such as passed through in the present embodiment
The redis interface of SpringBoot realizes the access to first database Redis database, is connect by the Jpa of SpringBoot
The existing access to the second database Mysql database of cause for gossip, is realized by HBase JavaAPI to third database HBase number
According to the access in library.
Corresponding with data management system provided by the above embodiment, one embodiment of the application also provides a kind of utilization
The data managing method of above-mentioned data management system, due to data managing method provided by the embodiments of the present application and above-mentioned several realities
The data management system for applying example offer is corresponding, therefore aforementioned embodiments are also applied for data management side provided in this embodiment
Method is not described in detail in the present embodiment.
As shown in figure 5, one embodiment of the application also provides a kind of data management using above-mentioned data management system
Method, comprising: computer real-time reception and the monitoring data for showing tested equipment；The computer is according to the monitoring data
Type is stored: the dsc data in the monitoring data being stored in the first database of first server, by the monitoring
Warm data in data are stored in the second database of first server, and the cold data in the monitoring data is stored in second
The third database of server.
In a preferred embodiment, the computer further includes receiving unit, Data Management Unit, visualization
And reading unit, by the reading unit and visualization to the first database and/or the second database and/or
The data of three databases are retrieved, edited and are deleted.
The present invention formulates a kind of based on cold, warm, dsc data data management system and data for problem existing at present
Management method, and by setting multi-layer, three-dimensional three layer data library structures, it is divided into cold number according to data usage frequency
According to, warm data and dsc data three types, it is respectively stored in first database, the second database and third database, can be solved
The certainly efficient management and lateral dilatation of the release loss problem of dsc data, the visualized management deficiency problem of warm data, cold data
Problem guarantees the efficiency of management, storage efficiency and search efficiency of data, utmostly to compensate for problem in the prior art
Upper raising software service ability and data service capabilities.
Obviously, the above embodiment of the present invention be only to clearly illustrate example of the present invention, and not be pair
The restriction of embodiments of the present invention may be used also on the basis of the above description for those of ordinary skill in the art
To make other variations or changes in different ways, all embodiments can not be exhaustive here, it is all to belong to this hair
The obvious changes or variations that bright technical solution is extended out are still in the scope of protection of the present invention.
Claims (10)
1. a kind of based on cold, warm, dsc data data management system, which is characterized in that including computer, at least one first clothes
Business device and at least one second server, wherein
The first server includes the first database for storing dsc data and the second database for storing warm data；
The second server includes the third database for storing cold data；
The computer real-time reception and the monitoring data for showing tested equipment, distinguish according to the data type of the monitoring data
Store the first database, the second database and third database.
2. data management system according to claim 1, which is characterized in that the first database is distributed memory number
According to library, second database is relevant database, and the third database is distributed column database.
3. data management system according to claim 1, which is characterized in that the first database include primary database and
At least one standby database, the standby database synchronize the data of the primary database.
4. data management system according to claim 1, which is characterized in that the first database is according to pre-seting the period
It is stored in the disk of the first server.
5. data management system according to claim 1, which is characterized in that
The first database is Redis database；And/or
Second database is MySQL database, including multiple tables of data；And/or
The third database is the Hbase column database for including structured storage cluster.
6. data management system according to claim 1, which is characterized in that the first database and the second database with
Key-Value structure is stored.
7. data management system according to claim 1 to 6, which is characterized in that the computer includes receiving
Unit, Data Management Unit and visualization, wherein
The receiving unit, for monitoring data described in real-time reception；
The Data Management Unit, for storing, retrieving, editing and deleting respectively according to the type of the monitoring data；
The visualization, for monitoring data described in real-time display.
8. data management system according to claim 7, which is characterized in that the computer further includes reading unit, institute
Stating reading unit includes the second interface and reading for reading the first interface of the first database, reading second database
The third interface of third database.
9. a kind of data managing method using the data management system of any of claims 1-8, which is characterized in that
Include:
Computer real-time reception and the monitoring data for showing tested equipment；
The computer is stored according to the type of the monitoring data: the dsc data in the monitoring data is stored in
Warm data in the monitoring data are stored in the second database of first server by the first database of one server, will
Cold data in the monitoring data is stored in the third database of second server.
10. data managing method according to claim 9, which is characterized in that the method also includes:
The computer further includes receiving unit, Data Management Unit, visualization and reading unit, passes through the reading list
Member and visualization are retrieved the data of the first database and/or the second database and/or third database, are compiled
It collects and deletes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910332128.0A CN110083649A (en) | 2019-04-24 | 2019-04-24 | It is a kind of based on cold, warm, dsc data data management system and data managing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910332128.0A CN110083649A (en) | 2019-04-24 | 2019-04-24 | It is a kind of based on cold, warm, dsc data data management system and data managing method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110083649A true CN110083649A (en) | 2019-08-02 |
Family
ID=67416414
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910332128.0A Pending CN110083649A (en) | 2019-04-24 | 2019-04-24 | It is a kind of based on cold, warm, dsc data data management system and data managing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110083649A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110888861A (en) * | 2019-11-12 | 2020-03-17 | 上海麦克风文化传媒有限公司 | Novel big data storage method |
| CN113420021A (en) * | 2021-07-14 | 2021-09-21 | 苏州峰之鼎信息科技有限公司 | Data storage method, device, equipment and medium |
| CN113687766A (en) * | 2020-05-19 | 2021-11-23 | 中移动信息技术有限公司 | Data storage method, device, equipment and storage medium |
| CN113704346A (en) * | 2020-05-20 | 2021-11-26 | 杭州海康威视数字技术股份有限公司 | Hbase table cold and hot data conversion method and device and electronic equipment |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106156331A (en) * | 2016-07-06 | 2016-11-23 | 益佳科技（北京）有限责任公司 | Cold and hot temperature data server system and processing method thereof |
| CN108512908A (en) * | 2018-03-13 | 2018-09-07 | 山东超越数控电子股份有限公司 | A kind of cloud storage fault tolerant mechanism based on Ceph and the web-based management platform based on Ceph |
| CN108520342A (en) * | 2018-03-23 | 2018-09-11 | 中建三局第建设工程有限责任公司 | Platform of internet of things management method based on BIM and its system |
| CN108549525A (en) * | 2018-04-04 | 2018-09-18 | 北京蓝杞数据科技有限公司天津分公司 | Data storage and access method, apparatus, electronic equipment and storage medium |
| CN108664565A (en) * | 2018-04-18 | 2018-10-16 | 吉旗（成都）科技有限公司 | A kind of Internet of Things timeliness data hierarchy storage method |
| US20180373722A1 (en) * | 2017-06-26 | 2018-12-27 | Acronis International Gmbh | System and method for data classification using machine learning during archiving |
| CN109379425A (en) * | 2018-10-16 | 2019-02-22 | 浪潮软件集团有限公司 | Distributed cluster deployment management method and device |
-
2019
- 2019-04-24 CN CN201910332128.0A patent/CN110083649A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106156331A (en) * | 2016-07-06 | 2016-11-23 | 益佳科技（北京）有限责任公司 | Cold and hot temperature data server system and processing method thereof |
| US20180373722A1 (en) * | 2017-06-26 | 2018-12-27 | Acronis International Gmbh | System and method for data classification using machine learning during archiving |
| CN108512908A (en) * | 2018-03-13 | 2018-09-07 | 山东超越数控电子股份有限公司 | A kind of cloud storage fault tolerant mechanism based on Ceph and the web-based management platform based on Ceph |
| CN108520342A (en) * | 2018-03-23 | 2018-09-11 | 中建三局第建设工程有限责任公司 | Platform of internet of things management method based on BIM and its system |
| CN108549525A (en) * | 2018-04-04 | 2018-09-18 | 北京蓝杞数据科技有限公司天津分公司 | Data storage and access method, apparatus, electronic equipment and storage medium |
| CN108664565A (en) * | 2018-04-18 | 2018-10-16 | 吉旗（成都）科技有限公司 | A kind of Internet of Things timeliness data hierarchy storage method |
| CN109379425A (en) * | 2018-10-16 | 2019-02-22 | 浪潮软件集团有限公司 | Distributed cluster deployment management method and device |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110888861A (en) * | 2019-11-12 | 2020-03-17 | 上海麦克风文化传媒有限公司 | Novel big data storage method |
| CN113687766A (en) * | 2020-05-19 | 2021-11-23 | 中移动信息技术有限公司 | Data storage method, device, equipment and storage medium |
| CN113704346A (en) * | 2020-05-20 | 2021-11-26 | 杭州海康威视数字技术股份有限公司 | Hbase table cold and hot data conversion method and device and electronic equipment |
| CN113420021A (en) * | 2021-07-14 | 2021-09-21 | 苏州峰之鼎信息科技有限公司 | Data storage method, device, equipment and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110083649A (en) | It is a kind of based on cold, warm, dsc data data management system and data managing method | |
| CN105549905B (en) | A kind of method that multi-dummy machine accesses distributed objects storage system | |
| US8909887B1 (en) | Selective defragmentation based on IO hot spots | |
| CN103106152B (en) | Based on the data dispatching method of level storage medium | |
| CN100571281C (en) | Great magnitude of data hierarchical storage method | |
| JP3786955B2 (en) | Data storage management system for network interconnection processor | |
| US6571261B1 (en) | Defragmentation utility for a shared disk parallel file system across a storage area network | |
| CN1559041B (en) | Sharing objects between computer systems | |
| CN100583832C (en) | Data management method and system | |
| CN100399301C (en) | Mass storage device and method for dynamically managing a mass storage device | |
| CN107291889A (en) | A kind of date storage method and system | |
| JP2002108662A (en) | Method for managing information | |
| CN102136003A (en) | Large-scale distributed storage system | |
| JP2006031668A5 (en) | ||
| CN101137981A (en) | Methods and apparatus for managing the storage of content in a file system | |
| CN103095805A (en) | Cloud storage system of data intelligent and decentralized management | |
| CN102197372A (en) | Partition management in a partitioned, scalable, and available structured storage | |
| CN102331986A (en) | Database cache management method and database server | |
| CN102667704A (en) | Hybrid storage subsystem | |
| CN102411542A (en) | Dynamic hierarchical storage system and method | |
| CN101488153A (en) | Method for implementing high-capacity flash memory file system in embedded type Linux | |
| US11880302B2 (en) | Optimized record placement in defragmenting graph database | |
| CN106610903A (en) | Tiered storage system, storage controller, and method for deduplication and storage tiering | |
| US20220350834A1 (en) | Optimized record placement in graph database | |
| CN104035925A (en) | Data storage method and device and storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190802 |