CN110019209A - A kind of big data emerging system and method based on Hydropower Enterprise ' business datum - Google Patents
A kind of big data emerging system and method based on Hydropower Enterprise ' business datum Download PDFInfo
- Publication number
- CN110019209A CN110019209A CN201711138975.0A CN201711138975A CN110019209A CN 110019209 A CN110019209 A CN 110019209A CN 201711138975 A CN201711138975 A CN 201711138975A CN 110019209 A CN110019209 A CN 110019209A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- emerging system
- big
- big data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 32
- 238000012545 processing Methods 0.000 claims description 21
- 238000003860 storage Methods 0.000 claims description 17
- 238000009826 distribution Methods 0.000 claims description 9
- 230000010354 integration Effects 0.000 claims description 9
- 238000004458 analytical method Methods 0.000 claims description 7
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 claims description 6
- 238000004519 manufacturing process Methods 0.000 claims description 5
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 238000000605 extraction Methods 0.000 claims description 4
- 238000011068 loading method Methods 0.000 claims description 4
- 230000011218 segmentation Effects 0.000 claims description 4
- 230000001360 synchronised effect Effects 0.000 claims description 4
- 238000012546 transfer Methods 0.000 claims description 2
- 230000004069 differentiation Effects 0.000 claims 1
- 230000004888 barrier function Effects 0.000 abstract description 6
- 230000004927 fusion Effects 0.000 abstract description 4
- 238000005457 optimization Methods 0.000 abstract description 4
- 238000005516 engineering process Methods 0.000 description 7
- 238000007726 management method Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 230000008859 change Effects 0.000 description 4
- 238000011161 development Methods 0.000 description 4
- 230000018109 developmental process Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 238000013480 data collection Methods 0.000 description 3
- 238000013500 data storage Methods 0.000 description 3
- 238000012423 maintenance Methods 0.000 description 3
- 238000011144 upstream manufacturing Methods 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000000547 structure data Methods 0.000 description 2
- 241001269238 Data Species 0.000 description 1
- 230000003698 anagen phase Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 239000003153 chemical reaction reagent Substances 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 239000006185 dispersion Substances 0.000 description 1
- 230000005611 electricity Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000002844 melting Methods 0.000 description 1
- 230000008018 melting Effects 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
- 238000005303 weighing Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/06—Energy or water supply
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- Economics (AREA)
- Water Supply & Treatment (AREA)
- Tourism & Hospitality (AREA)
- Strategic Management (AREA)
- General Business, Economics & Management (AREA)
- Primary Health Care (AREA)
- Marketing (AREA)
- Software Systems (AREA)
- Human Resources & Organizations (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention provides a kind of big data emerging system based on Hydropower Enterprise ' business datum, including TSDB time series database, KingbaseDB database, Redis memory database and Hadoop big data platform.The present invention also provides a kind of methods using big data emerging system.Big data emerging system and method provided by the invention based on Hydropower Enterprise ' business datum, optimization data space occupy, and data access speed is fast and data query operation is convenient；By the fusion of relational data engine and non-relational data engine, in the case of back-end data is stored separately, the universal data access interface of service-oriented application is provided, to simplify the exploitation of operation system；Hydropower Enterprise ' Various types of data resource effectively is integrated, gets through the barrier between data.
Description
Technical field
The invention belongs to big data processing technology fields, and in particular to a kind of big data based on Hydropower Enterprise ' business datum
Emerging system and method.
Background technique
With the fast development in digital information epoch, global digital information resources just entering one it is unprecedented
Rapid growth phase electric power big data is the necessary process that power industry technologies are reformed in energy revolution, and electric power big data is not only
Technological progress, be even more related to entire electric system under big data era idea of development, management system and in terms of
Major transformation, be that the next-generation intelligent electric Force system form of value under big data era rises to.
Currently, water power and relevant enterprise pass through the informatization of many years, Various types of data resource has begun to take shape, but above-mentioned
The data at the Enterprise Data center of enterprise are generally using the mode of dispersion, and memory space occupancy is larger, and data access speed is slow
And data query operation complexity etc., it is unable to satisfy the data supporting demand of the following enterprise efficiency promotion, ability transition etc..For
Reply big data era bring opportunities and challenges, and under reply future market environment, State Grid's system
The requirement of reform, how to design a kind of data acquisitions, integration, storage big data emerging system solve Enterprise Data center
Data concentration problem, become this field problem urgently to be resolved.
Summary of the invention
To solve the above-mentioned problems, the purpose of the present invention is to provide a kind of big datas based on Hydropower Enterprise ' business datum
Emerging system and method, optimization data space occupy, and data access speed is fast and data query operation is convenient；Offer face
To the universal data access interface of service application, to simplify the exploitation of operation system；Effectively integrate all kinds of numbers of Hydropower Enterprise '
According to resource, the barrier between data is got through.
To achieve the goals above, the technical solution adopted by the present invention are as follows:
A kind of big data emerging system based on Hydropower Enterprise ' business datum, including TSDB time series database, KingbaseDB number
According to library, Redis memory database and Hadoop big data platform；The TSDB time series database is used for the number of unstructured data
According to access；The KingbaseDB database is for structural data and the non-knot in part inputted through the TSDB time series database
The caching of structure data, and carry out depth and summarize, calculate, transfer data to the Redis memory database；The Redis
Memory database stores data, retains certain time, and carry out secondary operation to historical data.
It further, further include Oracle structured database, the Oracle structured database is used for storage organization
Change data, and carries out subsequent batch processing.
It further, further include cooperative module, the cooperative module provides unified SQL for executing stsndard SQL sentence
Engine.
Further, further include with KingbaseDB relational database carry out data manipulation distributed relational database and
Monitor supervision platform.
Further, the storage uses the data distribution based on Hash, and data and its backup are calculated by consistency Hash
Method is distributed evenly on the data store set group that all memory nodes are constituted.
Further, the structural data includes ERP, program plan, e-commerce platform, water power production management information
One or more of transaction data of system.
Further, the unstructured data includes in equipment operating data, sensing data and external social data
It is one or more.
The present invention also provides a kind of method using the above-mentioned big data emerging system based on Hydropower Enterprise ' business datum, tools
Body step are as follows:
1) by structural data, extraction synchronous by data, conversion, loading method enter the storage of Oracle structured database,
And carry out subsequent batch processing；
2) by unstructured data, TSDB time series database is entered using Stream Processing；
3) Kingbase relational database caches structural data and part unstructured data, carries out depth and summarizes, calculates,
Data are sent to Redis memory database；
4) Redis memory database uses the data distribution based on Hash, and data and its backup pass through consistency hash algorithm quilt
It is evenly distributed on the data store set group that all memory nodes are constituted and stores, and retain certain time；Described
Secondary operation is carried out to historical data in Redis memory database；
5) after Data Integration and processing, analysis and prediction address are exported for end user.
Further, the specific data access of the Stream Processing are as follows:
A. it pulls and is decoupled with push, data pre-fetching, queue keep in, pull and send parallel；
B. task segmentation is big Task-decomposing into small task, small task horizontal extension；
C. the fixed flow of the i.e. each task carrying of task standardization, flow increase then increase task quantity；
D. it dispatches, arrange more and develop simultaneously balance and squeezing machine performance, reduce delay, reach real-time access.
Further, the unstructured data updates every time, does not cover original version, generates a new version,
The new and old edition of data is distinguished by timestamp；Structural data updates operation hour according to directly being covered.
Big data emerging system and method provided by the invention based on Hydropower Enterprise ' business datum is compared with prior art, beneficial
Effect is:
1. big data emerging system of the invention and method, optimization data space is occupied, and data access speed is fast and data
Inquiry operation is convenient；
2. big data emerging system of the invention and method, pass through melting for relational data engine and non-relational data engine
It closes, in the case of back-end data is stored separately, provides the universal data access interface of service-oriented application, to simplify business
The exploitation of system；
3. data storage has been effectively ensured using the data distribution based on Hash in big data emerging system of the invention and method
Load balancing, reliability and consistency between node；
4. big data system and method for the invention is effectively integrated Hydropower Enterprise ' Various types of data resource, is got through between data
Barrier, the effective all kinds of problems improved in Enterprise Data central data collection and integration process.
In short, the invention proposes a kind of big data fusions based on Hydropower Enterprise ' business datum for being easy to use and safeguarding
System and method is with a wide range of applications in big data processing technology field.
Detailed description of the invention
Fig. 1 is the schematic diagram of big data emerging system of the present invention.
Fig. 2 is the flow diagram of big data emerging system of the present invention.
Wherein, the reference numerals are as follows:
1- big data emerging system, 2-TSDB time series database, 3-KingbaseDB database, 4-Redis memory database, 5-
Hadoop big data platform.
Specific embodiment
In order to make those skilled in the art more fully understand technical solution of the present invention, combined with specific embodiments below to this
Invention is described in further detail.It note that the embodiments described below is exemplary, for explaining only the invention, and
It is not considered as limiting the invention.Particular technique or condition are not specified in embodiment, according to the literature in the art institute
The technology or conditions of description are carried out according to product description.Reagents or instruments used without specified manufacturer, being can
With conventional products that are commercially available.
In the description of the present invention, it is to be understood that, term " first ", " second " are used for description purposes only, and cannot
It is interpreted as indication or suggestion relative importance or implicitly indicates the quantity of indicated technical characteristic.Define as a result, " the
One ", the feature of " second " can explicitly or implicitly include one or more of the features.In the description of the present invention,
The meaning of " plurality " is two or more, unless otherwise specifically defined.
In the present invention unless specifically defined or limited otherwise, term " installation ", " connected ", " connection ", " fixation " etc.
Term shall be understood in a broad sense, for example, it may be being fixedly connected, may be a detachable connection, or integral；It can be mechanical connect
It connects, is also possible to be electrically connected；It can be directly connected, can also can be in two elements indirectly connected through an intermediary
The interaction relationship of the connection in portion or two elements.It for the ordinary skill in the art, can be according to specific feelings
Condition understands the concrete meaning of above-mentioned term in the present invention.
The present invention provides a kind of big data emerging system and method based on Hydropower Enterprise ' business datum, by the system and
Method effectively integrates Hydropower Enterprise ' Various types of data resource, gets through the barrier between data, effective to improve in Enterprise Data
Calculation is according to all kinds of problems in acquisition and integration process.
As shown in Figs. 1-2, the big data emerging system 1 provided by the invention based on Hydropower Enterprise ' business datum, including
TSDB time series database 2, KingbaseDB database 3, Redis memory database 4 and Hadoop big data platform 5 etc..
Big data emerging system of the invention can carry out data processing to structured data source and unstructured data sources,
Unified data storage and management are carried out, have ensured subsequent analysis, consistent using bore.Big data emerging system of the invention
It in conjunction with Oracle structured database, can also constitute the Data Integration storage environment of complete set, certain Oracle structuring
Database also can be used as a part of big data emerging system of the present invention.
Big data emerging system of the invention, the biggish data of business relations, such as the basic information of data collection point
The time series data for being stored in traditional relevant database, while production equipment being generated in real time be stored in non-relational when
In sequence database, document and non-relational data are stored in big data platform.Big data emerging system of the invention needs to tie
The actual services demand of Heshui electricity operation monitoring system and newest non-relational database technology are found suitable data and are divided
Point.
For the structural data of traditional business system, such as ERP, program plan, e-commerce platform, water power production management
The transaction data of information system etc., extraction synchronous using traditional data, conversion, loading method enter Oracle structuring number
It is stored according to library, and carries out subsequent batch processing.
For the unstructured data of equipment operation, sensor and external social activity etc., when entering TSDB using Stream Processing
Sequence database, the flow efficiency of data are the key that such data access.Push and pull two is used when data access in the present invention
Kind model approach.Both model approach respectively have advantage and disadvantage, but common common situation is as follows: source data yield is stable forever
Slowly, the speed of data transmission is limited solely by the influence of the side in upstream and downstream for constant or variation, and unidirectional handling capacity=request is big
Small * number of concurrent, entire throughput=f(pull handling capacity, and link bearing capacity pushes handling capacity).
The step of specific data access are as follows:
A. upstream and downstream decoupling (pull and decouple with push), data pre-fetching, queue keep in, pull and send parallel；
B. task segmentation is big Task-decomposing into small task, small task horizontal extension；
C. the fixed flow of the i.e. each task carrying of task standardization, flow increase then increase task quantity；
D. resource utilization is promoted, i.e. the modes such as scheduling, balance, squeezing machine performance are parallel, make every effort to reduce delay, reach real-time
The effect of access.
In big data emerging system of the invention, the operation systems such as business supervision are closed by data/address bus and KingbaseDB
It is that database carries out data manipulation, transmission structure data；Unstructured data enters time series data by using Stream Processing
After the TSDB of library, the collected data of data of field instrument equipment therein enter the non-relationship memory database of Redis, file system
System uses Hadoop cluster expansion mode, remaining unstructured data is input in KingbaseDB relational database.
KingbaseDB relational database caches structural data and part unstructured data, carries out depth and summarizes, calculates, will count
According to being sent to Redis non-relational database.Redis non-relational database stores data, retains certain time, such as 5 years；Foundation
Business needs to change, and carries out secondary operation to historical data in Redis non-relational database.Wherein, the non-relationship memory of Redis
Database is calculated for the inquiry of complex page and the caching of field data, internal storage data for data, TSDB time series database
File system use Hadoop cluster expansion mode, advantage is there is preferable horizontal extensibility.
Big data emerging system of the invention, support three kinds of data access logic components: object stores (Key-Value), structure
Change storage, file storage；Using the data distribution based on Hash, data and its backup pass through consistency hash algorithm by equably
It is distributed on the data store set group that all memory nodes are constituted, load balancing, reliability and one between node has been effectively ensured
Cause property.
Big data emerging system of the invention, further includes cooperative module.The cooperative module, for executing stsndard SQL language
Sentence, provides unified SQL engine, facilitates use of the application development side to data.
Big data emerging system of the invention, wherein can also be by distributed relational database (remote disaster tolerance backup), monitoring
Platform (depth operation platform) and KingbaseDB relational database can carry out data manipulation, are easy to use and extend, provide
User-friendly control platform facilitates the maintenance work of user.
Big data emerging system of the invention is carried out by the way that the data in a variety of sources are entered water power big data platform
Unified storage and management have ensured subsequent analysis, consistent using bore.
The present invention also provides a kind of method using the above-mentioned big data emerging system based on Hydropower Enterprise ' business datum, tools
Steps are as follows for body:
1) by the structural data of traditional business system, such as ERP, program plan, e-commerce platform, water power production management information
The transaction data of system etc., extraction synchronous using traditional data, conversion, loading method enter Oracle structured database
Storage, and carry out subsequent batch processing；
2) equipment will be run, the unstructured data of sensor and external social activity etc., when entering TSDB using Stream Processing
Sequence database, wherein specific data access are as follows: upstream and downstream decoupling (pull with push decouple), data pre-fetching, queue be temporary,
It pulls and sends parallel；Task segmentation is big Task-decomposing into small task, small task horizontal extension；Task standardization is i.e. each to appoint
The fixed flow of business carrying, flow increase then increase task quantity；Resource utilization is promoted, that is, dispatches, balance, squeezing machine
Arrange such as energy is developed simultaneously more, is made every effort to reduce delay, is achieved the effect that access in real time；
3) Kingbase relational database caches structural data and part unstructured data, carries out depth and summarizes, calculates,
Data are sent to Redis non-relational database；
4) Redis non-relational database uses the data distribution based on Hash, and data and its backup pass through consistency hash algorithm
It is distributed evenly on the data store set group that all memory nodes are constituted and stores, and retain certain time；According to industry
Business needs to change, and carries out secondary operation to historical data in Redis non-relational database；
5) after Data Integration and processing, analysis and prediction address can be exported for end user, is used convenient for end user,
In can be policymaker, advanced level user, service-user and client, according to actual needs export trend analysis decision support report, it is quiet
State analyzes real time business analysis and prediction address, real-time business diagnosis and prediction address and simple visualization tool.
More flexible data access logic component can be used in method of the present invention.Support three kinds of data access logic components: object
Store (Key-Value), structured storage, file storage.Flexible storage model, the storage model provided include that timing is deposited
Store up engine and non-sequential storage engines.The data distribution based on Hash is also used, data and its backup pass through consistency Hash
Algorithm is distributed evenly on the data store set group that all memory nodes are constituted, and the load being effectively ensured between node is equal
Weighing apparatus, reliability and consistency.It can provide user-friendly control platform, facilitate the maintenance work of user.
Method of the present invention provides unified SQL engine, executes stsndard SQL sentence, facilitates application development side
Use to data.Due to can simultaneously processing structure and unstructured data, when unstructured database must simultaneously support
Sequence engine and non-sequential engine.The difference of the two is: for timing engine, each update will not be covered originally
Version, but a new version can be generated, the new and old edition of data passes through timestamp and distinguishes；For non-sequential engine,
Update operation hour directly be covered according to meeting.
Method of the present invention, by the fusion of relational data engine and non-relational data engine, in rear number of units
In the case of being stored separately, the universal data access interface of service-oriented application is developed, to simplify opening for operation system
Hair.In this way, effectively integrating Hydropower Enterprise ' Various types of data resource, the barrier between data is got through, it is effective to improve enterprise
All kinds of problems in industry grade data central data collection and integration process.
Big data emerging system and method provided by the invention based on Hydropower Enterprise ' business datum, optimization data storage are empty
Between occupy, data access speed is fast and data query operation is convenient；Pass through relational data engine and non-relational data engine
Fusion, in the case of back-end data is stored separately, provide service-oriented application universal data access interface, thus simplify
The exploitation of operation system；Using the data distribution based on Hash, load balancing between data memory node, reliable has been effectively ensured
Property and consistency；Hydropower Enterprise ' Various types of data resource effectively is integrated, the barrier between data is got through, effectively improves enterprise-level
All kinds of problems in the acquisition of data center's data and integration process.In short, the invention proposes a kind of easy to use and maintenance
Big data emerging system and method based on Hydropower Enterprise ' business datum have extensive in big data processing technology field
Application prospect.
In the present invention unless specifically defined or limited otherwise, fisrt feature in the second feature " on " or " down " can be with
It is that the first and second features directly contact or the first and second features pass through intermediary mediate contact.Moreover, fisrt feature exists
Second feature " on ", " top " and " above " but fisrt feature be directly above or diagonally above the second feature, or be merely representative of
First feature horizontal height is higher than second feature.Fisrt feature can be under the second feature " below ", " below " and " below "
One feature is directly under or diagonally below the second feature, or is merely representative of first feature horizontal height less than second feature.
In the description of this specification, reference term " one embodiment ", " some embodiments ", " example ", " specifically show
The description of example " or " some examples " etc. means specific features, structure, material or spy described in conjunction with this embodiment or example
Point is included at least one embodiment or example of the invention.In the present specification, schematic expression of the above terms are not
It must be directed to identical embodiment or example.Moreover, particular features, structures, materials, or characteristics described can be any
It can be combined in any suitable manner in a or multiple embodiment or examples.In addition, without conflicting with each other, the technology of this field
The feature of different embodiments or examples described in this specification and different embodiments or examples can be combined by personnel
And combination.
Claims (10)
1. a kind of big data emerging system based on Hydropower Enterprise ' business datum, including TSDB time series database, KingbaseDB
Database, Redis memory database and Hadoop big data platform；The TSDB time series database is for unstructured data
Data access；The KingbaseDB database is non-for structural data and the part inputted through the TSDB time series database
The caching of structural data, and carry out depth and summarize, calculate, transfer data to the Redis memory database；It is described
Redis memory database stores data, retains certain time, and carry out secondary operation to historical data.
2. big data emerging system according to claim 1, which is characterized in that it further include Oracle structured database,
The Oracle structured database is used for structured data, and carries out subsequent batch processing.
3. big data emerging system according to claim 2, which is characterized in that it further include cooperative module, the collaboration mould
Block provides unified SQL engine for executing stsndard SQL sentence.
4. big data emerging system according to claim 3, which is characterized in that further include and KingbaseDB relation data
The distributed relational database and monitor supervision platform of library progress data manipulation.
5. big data emerging system according to claim 1, which is characterized in that the storage uses the data based on Hash
Distribution, data and its backup are distributed evenly at the data store set that all memory nodes are constituted by consistency hash algorithm
On group.
6. big data emerging system according to claim 5, which is characterized in that the structural data includes ERP, planning
One or more of plan, e-commerce platform, the transaction data of water power Production MIS.
7. big data emerging system according to claim 6, which is characterized in that the unstructured data includes equipment fortune
It is one or more in row data, sensing data and external social data.
8. a kind of application is according to claim 1 described in any one of -7 based on the big data emerging system of Hydropower Enterprise ' business datum
Method, which is characterized in that specific steps are as follows:
1) by structural data, extraction synchronous by data, conversion, loading method enter the storage of Oracle structured database,
And carry out subsequent batch processing；
2) by unstructured data, TSDB time series database is entered using Stream Processing；
3) Kingbase relational database caches structural data and part unstructured data, carries out depth and summarizes, calculates,
Data are sent to Redis memory database；
4) Redis memory database uses the data distribution based on Hash, and data and its backup pass through consistency hash algorithm quilt
It is evenly distributed on the data store set group that all memory nodes are constituted and stores, and retain certain time；Described
Secondary operation is carried out to historical data in Redis memory database；
5) after Data Integration and processing, analysis and prediction address are exported for end user.
9. according to the method described in claim 8, it is characterized in that, the specific data access of the Stream Processing are as follows:
A. it pulls and is decoupled with push, data pre-fetching, queue keep in, pull and send parallel；
B. task segmentation is big Task-decomposing into small task, small task horizontal extension；
C. the fixed flow of the i.e. each task carrying of task standardization, flow increase then increase task quantity；
D. it dispatches, arrange more and develop simultaneously balance and squeezing machine performance, reduce delay, reach real-time access.
10. according to the method described in claim 9, not covering original it is characterized in that, the unstructured data updates every time
Version, generate a new version, the new and old editions of data passes through timestamp differentiation；Structural data updates operation hour evidence
Directly covered.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711138975.0A CN110019209A (en) | 2017-11-16 | 2017-11-16 | A kind of big data emerging system and method based on Hydropower Enterprise ' business datum |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711138975.0A CN110019209A (en) | 2017-11-16 | 2017-11-16 | A kind of big data emerging system and method based on Hydropower Enterprise ' business datum |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110019209A true CN110019209A (en) | 2019-07-16 |
Family
ID=67186483
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711138975.0A Pending CN110019209A (en) | 2017-11-16 | 2017-11-16 | A kind of big data emerging system and method based on Hydropower Enterprise ' business datum |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110019209A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111125132A (en) * | 2019-12-19 | 2020-05-08 | 紫光云(南京)数字技术有限公司 | Data storage system and storage method |
| CN112115150A (en) * | 2020-08-03 | 2020-12-22 | 上海金仕达软件科技有限公司 | Data management method, terminal device and medium for embedded memory database |
| CN112182081A (en) * | 2020-09-23 | 2021-01-05 | 金现代信息产业股份有限公司 | Commuting passenger vehicle performance analysis method and system |
| CN112559515A (en) * | 2020-07-08 | 2021-03-26 | 北京德风新征程科技有限公司 | Method and device for optimizing data storage space in matrix calculation and variation process |
| CN113961637A (en) * | 2021-12-23 | 2022-01-21 | 北京力控元通科技有限公司 | Database-based data fusion method and system and electronic equipment |
| CN114448959A (en) * | 2021-12-17 | 2022-05-06 | 国网河南省电力公司信阳供电公司 | Mobile internet-based quick transmission and sharing system for regulating and controlling operation information |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104021194A (en) * | 2014-06-13 | 2014-09-03 | 浪潮(北京)电子信息产业有限公司 | Mixed type processing system and method oriented to industry big data diversity application |
| CN104820670A (en) * | 2015-03-13 | 2015-08-05 | 国家电网公司 | Method for acquiring and storing big data of power information |
| CN105608144A (en) * | 2015-12-17 | 2016-05-25 | 山东鲁能软件技术有限公司 | Big data analysis platform device and method based on multilayer model iteration |
| CN107329982A (en) * | 2017-06-01 | 2017-11-07 | 华南理工大学 | A kind of big data parallel calculating method stored based on distributed column and system |
-
2017
- 2017-11-16 CN CN201711138975.0A patent/CN110019209A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104021194A (en) * | 2014-06-13 | 2014-09-03 | 浪潮(北京)电子信息产业有限公司 | Mixed type processing system and method oriented to industry big data diversity application |
| CN104820670A (en) * | 2015-03-13 | 2015-08-05 | 国家电网公司 | Method for acquiring and storing big data of power information |
| CN105608144A (en) * | 2015-12-17 | 2016-05-25 | 山东鲁能软件技术有限公司 | Big data analysis platform device and method based on multilayer model iteration |
| CN107329982A (en) * | 2017-06-01 | 2017-11-07 | 华南理工大学 | A kind of big data parallel calculating method stored based on distributed column and system |
Non-Patent Citations (1)
| Title |
|---|
| 单若琦: "一种基于opentsdb的海量实时数据存储系统" * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111125132A (en) * | 2019-12-19 | 2020-05-08 | 紫光云(南京)数字技术有限公司 | Data storage system and storage method |
| CN112559515A (en) * | 2020-07-08 | 2021-03-26 | 北京德风新征程科技有限公司 | Method and device for optimizing data storage space in matrix calculation and variation process |
| CN112115150A (en) * | 2020-08-03 | 2020-12-22 | 上海金仕达软件科技有限公司 | Data management method, terminal device and medium for embedded memory database |
| CN112115150B (en) * | 2020-08-03 | 2024-03-19 | 上海金仕达软件科技股份有限公司 | Data management method, terminal equipment and medium of embedded memory database |
| CN112182081A (en) * | 2020-09-23 | 2021-01-05 | 金现代信息产业股份有限公司 | Commuting passenger vehicle performance analysis method and system |
| CN114448959A (en) * | 2021-12-17 | 2022-05-06 | 国网河南省电力公司信阳供电公司 | Mobile internet-based quick transmission and sharing system for regulating and controlling operation information |
| CN113961637A (en) * | 2021-12-23 | 2022-01-21 | 北京力控元通科技有限公司 | Database-based data fusion method and system and electronic equipment |
| CN113961637B (en) * | 2021-12-23 | 2022-03-18 | 北京力控元通科技有限公司 | Database-based data fusion method and system and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110019209A (en) | A kind of big data emerging system and method based on Hydropower Enterprise ' business datum | |
| Munshi et al. | Data lake lambda architecture for smart grids big data analytics | |
| Gupta et al. | Cloud computing and big data analytics: what is new from databases perspective? | |
| US10198298B2 (en) | Handling multiple task sequences in a stream processing framework | |
| US10191768B2 (en) | Providing strong ordering in multi-stage streaming processing | |
| Bahga et al. | Analyzing massive machine maintenance data in a computing cloud | |
| Ramnarayan et al. | Snappydata: A hybrid transactional analytical store built on spark | |
| CN104021194A (en) | Mixed type processing system and method oriented to industry big data diversity application | |
| CN109344207B (en) | Big data platform of integrative frequency spectrum all over the sky based on big dipper scanning | |
| Osman et al. | Towards real-time analytics in the cloud | |
| Costa et al. | The SusCity big data warehousing approach for smart cities | |
| Gupta et al. | Smartbench: A benchmark for data management in smart spaces | |
| Lipčák et al. | Big data platform for smart grids power consumption anomaly detection | |
| Hartmann | Enabling model-driven live analytics for cyber-physical systems: The case of smart grids | |
| Gao | Frameworks for big data integration, warehousing, and analytics | |
| Hong et al. | Big Data Analysis System Based on Cloudera Distribution Hadoop | |
| Sun et al. | The implementation of air pollution monitoring service using hybrid database converter | |
| Fan et al. | Design and implementation of scientific research big data service platform for experimental data managing | |
| Bureva | Index matrices as a tool for data lakehouse modelling | |
| Wang et al. | Survey of time series data processing in industrial internet | |
| Sanaboyina | Performance evaluation of time series databases based on energy consumption | |
| Darius et al. | From Data to Insights: A Review of Cloud-Based Big Data Tools and Technologies | |
| Momtselidze | Hadoop Integrating with Oracle Data Warehouse and Data Mining | |
| Li et al. | Research on storage and processing method for renewable energy big data | |
| Xiao et al. | [Retracted] A Big Data Analysis Algorithm Designed for the Interactive Platform of the Intelligent Sensor Information System |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190716 |