CN107682395A - A kind of big data cloud computing runtime and method - Google Patents
A kind of big data cloud computing runtime and method Download PDFInfo
- Publication number
- CN107682395A CN107682395A CN201710736166.3A CN201710736166A CN107682395A CN 107682395 A CN107682395 A CN 107682395A CN 201710736166 A CN201710736166 A CN 201710736166A CN 107682395 A CN107682395 A CN 107682395A
- Authority
- CN
- China
- Prior art keywords
- data
- cloud computing
- module
- runtime
- server group
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of big data cloud computing runtime and method, including big data runtime and cloud computing server group, the runtime includes handling process integration module, historical data acquisition module, real-time data acquisition module, memory module, data sharing module, the handling process integration module, for focusing on data；The historical data acquisition module, for gathering nontransaction data；The real-time data acquisition module, for gathering the business datum occurred immediately in real time；The memory module, for storing the historical data and business datum of collection.Big data system of the present invention greatly reduces the use threshold of big data system, and enable organization internal big data related resource conveniently shared, it is a set of sustainable big data system accumulated, and cloud computing server group ensure that the correctness of data transmission procedure, the privacy of data, it ensure that the high reliability of server group normal work.
Description
Technical field
The present invention relates to computerized information storage and processing technology field, specially a kind of big data cloud computing runtime
And method.
Background technology
With the fast development of computer and Internet technology, we have been in the epoch of an information explosion, in order to locate
Substantial amounts of information is managed, the concept of big data occurs.So-called big data, referring to can not be in the time range that can be born with conventional
The data acquisition system that Software tool is caught, managed and handled, it is to need new tupe to have stronger decision edge, hole
Examine magnanimity, high growth rate and the diversified information assets for finding power and process optimization ability.Due to the mass property of data, people
Be only difficult to these analyses of these data, but in the lining using cloud computing as the technological innovation curtain of representative with one's own
Under support, these are difficult to collect and the data that use start easily to be utilized originally, by constantly bringing forth new ideas for all trades and professions,
Big data progressively creates more values for the mankind.
Cloud computing (Cloud computing), it is a kind of method of emerging shared architecture, can is by huge
System pond links together to provide various IT services.Cloud computing generally has point of narrow sense cloud computing and broad sense cloud computing.Wherein, extensively
Adopted cloud computing refers to delivery and the use pattern of service, refers to by network with service on demand, needed for easy extension way acquisition.
This service can be IT to software, internet is related or any other services.
But existing big data cloud computing system is in the service provided using different providers, especially with multiple small
During the cloud service that scale cloud computing provider provides, because the ability and interface between cloud service are incompatible, it is necessary to establish respectively
For the interface and platform of each big data system, inconvenience is caused, while also result in the waste of system resource, total system
The low problems above of efficiency just causes very big tired scratch to industry, then how to invent a kind of big data cloud computing operation
System and method, this, which turns into us, needs to solve the problems, such as.
The content of the invention
It is an object of the invention to provide a kind of big data cloud computing runtime and method, solves institute in background technology
The problem of proposition.
To solve the above problems, the present invention provides following technical scheme：A kind of big data cloud computing runtime and method,
Including big data runtime and cloud computing server group, the runtime includes handling process integration module, historical data
Acquisition module, real-time data acquisition module, memory module, data sharing module, the handling process integration module, for concentrating
Processing data；The historical data acquisition module, for gathering nontransaction data；The real-time data acquisition module, for reality
When gather the business datum that occurs immediately；The memory module, for storing the historical data and business datum of collection；Institute
Processing module is stated, for handling the historical data and business datum of collection, the data after being handled；The data sharing mould
Block, for sharing the data after the processing, the big data runtime and the cloud computing server group information intercommunication.
Preferably, the big data runtime includes following to data processing method：
S1, data format is changed, establish and be applied to the form that non-relational database is stored；
S2, basic data and expert data are splitted data into, data are entered using the strategy of centralized and distributed combination
Row storage, all data are all backed up during storage；The strategy of the centralized and distributed combination includes：For higher than default
The basic data of frequency uses centralised storage, is safeguarded by control data corporation is unified, for the professional number less than predeterminated frequency
According to using distributed storage, safeguarded respectively by each expert data center；
S3, corresponding data retrievad algorithm is established, quick-searching is carried out to data, the data retrievad algorithm uses catalogue
The mode that retrieval and search engine are combined is carried out, and is specifically included：Data directory is established, data are tentatively examined according to catalogue
Rope；Keyword is inputted in search engine, precise search is carried out to data；Search engine finds the number of matching according to certain mode
According to, and be ranked up according to the matching degree of data and keyword and feed back to user.
S4, composition is sequentially connected by data source, data sorting unit, data number unit, cloud database, the data are compiled
Number unit also includes：Data Position is numbered, data time numbering, data capacity numbering；The data sorting unit effectively will
Affiliated data source carries out the classification of classification, under different categorical datas, using the data number unit, to different classes of
Same type data carry out Data Position, data time, and the classification of data capacity is classified, and the data sorted out are stored in
Cloud database, so as to realize tree-shaped type textural classification system.
Preferably, the processing module includes SQL processing units, general-purpose algorithm processing unit and processing unit of increasing income；Institute
SQL processing units are stated to be used to support user directly to handle the historical data and transaction data using SQL statement.
Preferably, the cloud computing server group is provided with cloud computing processing module, the cloud computing processing module, for leading to
The cloud computing request that receiving module receives cloud computing requestor is crossed, request is therefrom parsed, selects the cloud computing requestor to carry
The cloud computing service and resource and functional mode of confession, perform cloud service operation.
Preferably, it is as follows to perform method for the cloud computing server group：
A, registered to middleware server；User is registered by client to middleware server, and sets the user can
Cloud computing server group；
B, the user that middleware server logs according to client, local data base is searched for, finding out active user can
The file of operation, available cloud computing server group；
C, user need the computation schema file that uploads locally selecting, transmitted after being divided into the data block being sized
To middleware server；
D, middleware server verify to the computation schema file of upload；If lack certain data block or do not pass through school
Test, then require that client where the user retransmits to the data block, if correctly, transferring data to the user and can use
Cloud computing server group.
Compared with prior art, beneficial effects of the present invention are as follows：
Big data system of the present invention greatly reduces the use threshold of big data system, and makes organization internal big data phase
Closing resource can be conveniently shared, is a set of sustainable big data system accumulated, and cloud computing server group ensure that number
According to the correctness of transmitting procedure, the privacy of data, the high reliability of server group normal work ensure that.
Brief description of the drawings
Fig. 1 is total system principle framework schematic diagram of the present invention；
Fig. 2 is processing module principle framework schematic diagram of the present invention.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clear, complete
Site preparation describes, it is clear that described embodiment is only part of the embodiment of the present invention, rather than whole embodiments.It is based on
Embodiment in the present invention, those of ordinary skill in the art are obtained every other under the premise of creative work is not made
Embodiment, belong to the scope of protection of the invention.
Fig. 1-2 is referred to, this practicality invention provides a kind of technical scheme：A kind of big data cloud computing runtime and side
Method, including big data runtime and cloud computing server group, the runtime include handling process integration module, history number
According to acquisition module, real-time data acquisition module, memory module, data sharing module, the handling process integration module, for collecting
Middle processing data；The historical data acquisition module, for gathering nontransaction data；The real-time data acquisition module, is used for
The business datum occurred immediately is gathered in real time；The memory module, for storing the historical data and business datum of collection；
The processing module, for handling the historical data and business datum of collection, the data after being handled；The data sharing mould
Block, for sharing the data after the processing, the big data runtime and the cloud computing server group information intercommunication.
The big data runtime includes following to data processing method：
S1, data format is changed, establish and be applied to the form that non-relational database is stored；
S2, basic data and expert data are splitted data into, data are entered using the strategy of centralized and distributed combination
Row storage, all data are all backed up during storage；The strategy of the centralized and distributed combination includes：For higher than default
The basic data of frequency uses centralised storage, is safeguarded by control data corporation is unified, for the professional number less than predeterminated frequency
According to using distributed storage, safeguarded respectively by each expert data center；
S3, corresponding data retrievad algorithm is established, quick-searching is carried out to data, the data retrievad algorithm uses catalogue
The mode that retrieval and search engine are combined is carried out, and is specifically included：Data directory is established, data are tentatively examined according to catalogue
Rope；Keyword is inputted in search engine, precise search is carried out to data；Search engine finds the number of matching according to certain mode
According to, and be ranked up according to the matching degree of data and keyword and feed back to user.
S4, composition is sequentially connected by data source, data sorting unit, data number unit, cloud database, the data are compiled
Number unit also includes：Data Position is numbered, data time numbering, data capacity numbering；The data sorting unit effectively will
Affiliated data source carries out the classification of classification, under different categorical datas, using the data number unit, to different classes of
Same type data carry out Data Position, data time, and the classification of data capacity is classified, and the data sorted out are stored in
Cloud database, so as to realize tree-shaped type textural classification system.
The processing module includes SQL processing units, general-purpose algorithm processing unit and processing unit of increasing income；At the SQL
Reason unit is used to support user directly to handle the historical data and transaction data using SQL statement.
The cloud computing server group is provided with cloud computing processing module, the cloud computing processing module, is received for passing through
Module receives the cloud computing request of cloud computing requestor, therefrom parses request, the cloud for selecting the cloud computing requestor to provide
Service and resource and functional mode are calculated, performs cloud service operation.
It is as follows that the cloud computing server group performs method：
A, registered to middleware server；User is registered by client to middleware server, and sets the user can
Cloud computing server group；
B, the user that middleware server logs according to client, local data base is searched for, finding out active user can
The file of operation, available cloud computing server group；
C, user need the computation schema file that uploads locally selecting, transmitted after being divided into the data block being sized
To middleware server；
D, middleware server verify to the computation schema file of upload；If lack certain data block or do not pass through school
Test, then require that client where the user retransmits to the data block, if correctly, transferring data to the user and can use
Cloud computing server group.
To sum up, big data system of the present invention greatly reduces the use threshold of big data system, and makes organization internal big
Data related resource can be conveniently shared, is a set of sustainable big data system accumulated, and cloud computing server group is protected
The correctness of data transmission procedure has been demonstrate,proved, the privacy of data, ensure that the high reliability of server group normal work.
Finally it should be noted that：The preferred embodiments of the present invention are the foregoing is only, are not intended to limit the invention,
Although the present invention is described in detail with reference to the foregoing embodiments, for those skilled in the art, it still may be used
To be modified to the technical scheme described in foregoing embodiments, or equivalent substitution is carried out to which part technical characteristic.
Within the spirit and principles of the invention, any modification, equivalent substitution and improvements made etc., it should be included in the present invention's
Within protection domain.
Claims (5)
1. a kind of big data cloud computing runtime and method, including big data runtime and cloud computing server group, it is special
Sign is：The runtime includes handling process integration module, historical data acquisition module, real-time data acquisition module, deposited
Store up module, data sharing module, the handling process integration module, for focusing on data；The historical data gathers mould
Block, for gathering nontransaction data；The real-time data acquisition module, for gathering the business datum occurred immediately in real time；Institute
Memory module is stated, for storing the historical data and business datum of collection；The processing module, for handling going through for collection
History data and business datum, the data after being handled；The data sharing module, for sharing the data after the processing,
The big data runtime and the cloud computing server group information intercommunication.
2. a kind of big data cloud computing runtime according to claim 1 and method, it is characterised in that：The big data
Runtime includes following to data processing method：
S1, data format is changed, establish and be applied to the form that non-relational database is stored；
S2, basic data and expert data are splitted data into, data are deposited using the strategy of centralized and distributed combination
Storage, all data are all backed up during storage；The strategy of the centralized and distributed combination includes：For higher than predeterminated frequency
Basic data use centralised storage, safeguarded by control data corporation is unified, adopted for the expert data less than predeterminated frequency
With distributed storage, safeguarded respectively by each expert data center；
S3, corresponding data retrievad algorithm is established, data are carried out with quick-searching, the data retrievad algorithm is retrieved using catalogue
Carry out, specifically include with the mode that search engine is combined：Data directory is established, preliminary search is carried out to data according to catalogue；
Keyword is inputted in search engine, precise search is carried out to data；Search engine finds the data of matching according to certain mode,
And it is ranked up according to the matching degree of data and keyword and feeds back to user.
S4, composition, the data number list are sequentially connected by data source, data sorting unit, data number unit, cloud database
Member also includes：Data Position is numbered, data time numbering, data capacity numbering；The data sorting unit is effectively by belonging to
Data source carries out the classification of classification, under different categorical datas, using the data number unit, to different classes of identical
Categorical data carries out Data Position, data time, and the classification of data capacity is classified, and the data sorted out are stored in into cloud number
According to storehouse, so as to realize tree-shaped type textural classification system.
3. a kind of big data cloud computing runtime according to claim 1 and method, it is characterised in that：The processing mould
Block includes SQL processing units, general-purpose algorithm processing unit and processing unit of increasing income；The SQL processing units are used to support user
Directly the historical data and transaction data are handled using SQL statement.
4. a kind of big data cloud computing runtime according to claim 1 and method, it is characterised in that：The cloud computing
Server group is provided with cloud computing processing module, the cloud computing processing module, is asked for receiving cloud computing by receiving module
The cloud computing request of person, therefrom parses request, the cloud computing service and resource and work(for selecting the cloud computing requestor to provide
Energy pattern, perform cloud service operation.
5. a kind of big data cloud computing runtime according to claim 1 and method, it is characterised in that：The cloud computing
It is as follows that server group performs method：
A, registered to middleware server；User is registered by client to middleware server, and sets the user available
Cloud computing server group；
B, the user that middleware server logs according to client, local data base is searched for, finding out active user can operate
File, available cloud computing server group；
C, user need the computation schema file that uploads locally selecting, and are divided into after the data block being sized in being transferred to
Between part server；
D, middleware server verify to the computation schema file of upload；If lacking certain data block or not validated,
Then require that client where the user retransmits to the data block, if correctly, it is available to transfer data to the user
Cloud computing server group.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710736166.3A CN107682395A (en) | 2017-08-24 | 2017-08-24 | A kind of big data cloud computing runtime and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710736166.3A CN107682395A (en) | 2017-08-24 | 2017-08-24 | A kind of big data cloud computing runtime and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107682395A true CN107682395A (en) | 2018-02-09 |
Family
ID=61135254
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710736166.3A Pending CN107682395A (en) | 2017-08-24 | 2017-08-24 | A kind of big data cloud computing runtime and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107682395A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108416067A (en) * | 2018-03-29 | 2018-08-17 | 重庆大学 | Mass data processing and the optimization of storing process execute evaluation method in industrial process |
| CN109063028A (en) * | 2018-07-09 | 2018-12-21 | 清远网博信息技术有限公司 | A kind of method, the system of tourism data push |
| CN109299213A (en) * | 2018-10-15 | 2019-02-01 | 临沂大学 | A kind of paper editor searching method |
| WO2019184618A1 (en) * | 2018-03-28 | 2019-10-03 | 腾讯科技（深圳）有限公司 | Method and device for storing data, server, and storage medium |
| CN110457281A (en) * | 2019-08-14 | 2019-11-15 | 北京博睿宏远数据科技股份有限公司 | Data processing method, device, equipment and medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102255933A (en) * | 2010-05-20 | 2011-11-23 | 中兴通讯股份有限公司 | Cloud service medium, cloud computing method and cloud system |
| CN102307233A (en) * | 2011-08-24 | 2012-01-04 | 无锡中科方德软件有限公司 | Cloud computing method for cloud computing server |
| CN104462585A (en) * | 2014-12-29 | 2015-03-25 | 芜湖乐锐思信息咨询有限公司 | Large data classification system |
| US20150341232A1 (en) * | 2014-05-22 | 2015-11-26 | Facebook, Inc. | Daily Counts and Usage Probabilities For a User of an Online Service |
| CN106203146A (en) * | 2016-08-30 | 2016-12-07 | 广东港鑫科技有限公司 | A kind of big data safety management system |
| CN106648672A (en) * | 2016-12-28 | 2017-05-10 | 北京云星宇交通科技股份有限公司 | Method and system for developing and running big data |
-
2017
- 2017-08-24 CN CN201710736166.3A patent/CN107682395A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102255933A (en) * | 2010-05-20 | 2011-11-23 | 中兴通讯股份有限公司 | Cloud service medium, cloud computing method and cloud system |
| CN102307233A (en) * | 2011-08-24 | 2012-01-04 | 无锡中科方德软件有限公司 | Cloud computing method for cloud computing server |
| US20150341232A1 (en) * | 2014-05-22 | 2015-11-26 | Facebook, Inc. | Daily Counts and Usage Probabilities For a User of an Online Service |
| CN104462585A (en) * | 2014-12-29 | 2015-03-25 | 芜湖乐锐思信息咨询有限公司 | Large data classification system |
| CN106203146A (en) * | 2016-08-30 | 2016-12-07 | 广东港鑫科技有限公司 | A kind of big data safety management system |
| CN106648672A (en) * | 2016-12-28 | 2017-05-10 | 北京云星宇交通科技股份有限公司 | Method and system for developing and running big data |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019184618A1 (en) * | 2018-03-28 | 2019-10-03 | 腾讯科技（深圳）有限公司 | Method and device for storing data, server, and storage medium |
| CN110309233A (en) * | 2018-03-28 | 2019-10-08 | 腾讯科技（深圳）有限公司 | Method, apparatus, server and the storage medium of data storage |
| US11275759B2 (en) | 2018-03-28 | 2022-03-15 | Tencent Technology (Shenzhen) Company Limited | Data storage method and apparatus, server, and storage medium |
| CN110309233B (en) * | 2018-03-28 | 2022-11-15 | 腾讯科技（深圳）有限公司 | Data storage method, device, server and storage medium |
| CN108416067A (en) * | 2018-03-29 | 2018-08-17 | 重庆大学 | Mass data processing and the optimization of storing process execute evaluation method in industrial process |
| CN109063028A (en) * | 2018-07-09 | 2018-12-21 | 清远网博信息技术有限公司 | A kind of method, the system of tourism data push |
| CN109299213A (en) * | 2018-10-15 | 2019-02-01 | 临沂大学 | A kind of paper editor searching method |
| CN110457281A (en) * | 2019-08-14 | 2019-11-15 | 北京博睿宏远数据科技股份有限公司 | Data processing method, device, equipment and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107682395A (en) | A kind of big data cloud computing runtime and method | |
| CN111435344B (en) | Big data-based drilling acceleration influence factor analysis model | |
| Ma et al. | Big graph search: challenges and techniques | |
| CN103631922B (en) | Extensive Web information extracting method and system based on Hadoop clusters | |
| CN109189782A (en) | A kind of indexing means in block chain commodity transaction inquiry | |
| CN109815254B (en) | Cross-region task scheduling method and system based on big data | |
| CN102722553A (en) | Distributed type reverse index organization method based on user log analysis | |
| CN107301197A (en) | A kind of business datum tracking processing system and method | |
| CN111552885B (en) | System and method for realizing automatic real-time message pushing operation | |
| CN102375837A (en) | Data acquiring system and method | |
| CN101446966A (en) | Data storage method and system | |
| CN104239377A (en) | Platform-crossing data retrieval method and device | |
| CN102200979A (en) | Distributed parallel information retrieval system and distributed parallel information retrieval method | |
| CN101739398A (en) | Distributed database multi-join query optimization algorithm | |
| CN105183809A (en) | Cloud platform data query method | |
| CN104615734B (en) | A kind of community management service big data processing system and its processing method | |
| CN106682206A (en) | Method and system for big data processing | |
| CN107046557A (en) | The intelligent medical calling inquiry system that dynamic Skyline is inquired about under mobile cloud computing environment | |
| CN102279891A (en) | Retrieval method, device and system for concurrently searching information technology (IT) logs | |
| CN108228787A (en) | According to the method and apparatus of multistage classification processing information | |
| CN115934856A (en) | Method and system for constructing comprehensive energy data assets | |
| CN103412883A (en) | Semantic intelligent information publishing and subscribing method based on P2P technology | |
| CN102981913A (en) | Inference control method and inference control system with support on large-scale distributed incremental computation | |
| CN100370458C (en) | Data integration and analysis system and method | |
| CN113806446A (en) | Rapid retrieval method for mass data of big data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180209 |
|
| RJ01 | Rejection of invention patent application after publication |