CN109240716A - A kind of big data platform version management and iteratively faster dispositions method and system - Google Patents
A kind of big data platform version management and iteratively faster dispositions method and system Download PDFInfo
- Publication number
- CN109240716A CN109240716A CN201811047946.8A CN201811047946A CN109240716A CN 109240716 A CN109240716 A CN 109240716A CN 201811047946 A CN201811047946 A CN 201811047946A CN 109240716 A CN109240716 A CN 109240716A
- Authority
- CN
- China
- Prior art keywords
- big data
- configuration
- image file
- version
- virtual machine
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
- G06F8/61—Installation
- G06F8/63—Image based installation; Cloning; Build to order
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/70—Software maintenance or management
- G06F8/71—Version control; Configuration management
Abstract
The present invention, which provides a kind of big data platform version management big data configuration is originally put into a virtual machine with iteratively faster dispositions method, to be run, and the virtual machine is saved as an image file later；Image file is uploaded to mirror image warehouse server, and generates corresponding timestamp；Mirror image warehouse server is connected to after the configuration version more new command of cluster management personnel, the image file of the entire cluster of corresponding big data is issued, and is updated in the iteration that the configuration version of entire cluster can be realized in each host node starting associated virtual machine process；The present invention also provides a kind of big data platform version managements and iteratively faster deployment system, and the upgrading iteration of the management configuration of entire cluster is reached by the iteration of image file.
Description
Technical field
The present invention relates to a kind of big data platform version managements and iteratively faster dispositions method and system.
Background technique
Big data platform, which copes with each business scenario, a set of big data software optimization configuration parameter, if used
Existing cluster management technology, then executing automation on hundreds and thousands of servers later with regard to needing to write a large amount of script
Configuration script, since script is write by personnel, workload is huge, and if in a certain script
It goes wrong, then it is larger to check difficulty, is unfavorable for user's use.
Summary of the invention
The technical problem to be solved in the present invention is to provide a kind of big data platform version management and iteratively faster deployment side
Method and system are realized each version of big data platform making a virtual machine image file, pass through the iteration of image file
Reach the upgrading iteration of the management configuration of entire cluster.
One of present invention is achieved in that a kind of big data platform version management and iteratively faster dispositions method, comprising:
Big data configuration is originally put into a virtual machine and is run by step 1, and the virtual machine is saved as a mirror later
As file；
Image file is uploaded to mirror image warehouse server by step 2, and generates corresponding timestamp；
Step 3, mirror image warehouse server are connected to after the configuration version more new command of cluster management personnel, will be corresponding big
The image file of the entire cluster of data issues, and entire collection can be realized in each host node starting associated virtual machine process
The iteration of the configuration version of group updates.
Further, the big data configuration originally includes: the environment of operating system version configuration and environment configurations, user
The configuration parameter of configuration and big data.
Further, the step 3 is further specifically: mirror image warehouse server is connected to the configuration version of cluster management personnel
After this more new command, corresponding image file is got, the image file is put into the virtual machine of configuration mirroring later
It is verified, if then being issued the image file by verifying, otherwise searching corresponding image file again, until logical
It is issued again after crossing verifying, and the configuration of entire cluster can be realized in each host node starting associated virtual machine process
The iteration of version updates.
Further, the mirror image warehouse server is used for the storage of image file, manages and is issued to big data and is each
The operational administrative of a node.
The two of the present invention are achieved in that a kind of big data platform version management and iteratively faster deployment system, special
Sign is: including:
Big data configuration is originally put into a virtual machine and runs, later saves as the virtual machine by generation module
One image file；
Image file is uploaded to mirror image warehouse server, and generates corresponding timestamp by uploading module；
Update module, mirror image warehouse server are connected to after the configuration version more new command of cluster management personnel, will correspond to
The image file of the entire cluster of big data issue, and can be realized in each host node starting associated virtual machine process whole
The iteration of the configuration version of a cluster updates.
Further, the big data configuration originally includes: the environment of operating system version configuration and environment configurations, user
The configuration parameter of configuration and big data.
Further, the update module is further specifically: mirror image warehouse server is connected to matching for cluster management personnel
After setting version updating instruction, corresponding image file is got, the image file is put into the virtual of configuration mirroring later
It is verified in machine, if then being issued the image file by verifying, otherwise searching corresponding image file again, directly
It is issued again to after through verifying, and entire cluster can be realized in each host node starting associated virtual machine process
The iteration of configuration version updates.
Further, the mirror image warehouse server is used for the storage of image file, manages and is issued to big data and is each
The operational administrative of a node.
The present invention has the advantage that
1) big data configuration text the mirror image quilt and configuration conformance of big data configuration: is realized by lightweight virtualization
The mirror image of part, and by the unified management of virtualization mirror image warehouse and version is issued, realize the configuration file of entire big data
Consistency, the problem of being not in version disorder.
2) the safety verification test of big data configuration version: each version is carried out in virtualization mirror image warehouse server empty
The correctness that configuration is verified in quasi- machine environment, avoids big data cluster abnormal problem caused by arrangement abnormalities.
3) by virtualizing mirror image warehouse server, configuration can quickly the iteratively faster of big data configuration version: be issued
Image release realizes the iteratively faster of cluster configuration version to all large data sets group node servers.
Detailed description of the invention
The present invention is further illustrated in conjunction with the embodiments with reference to the accompanying drawings.
Fig. 1 is the method for the present invention execution flow chart.
Specific embodiment
Big data platform version management of the present invention and iteratively faster dispositions method, comprising:
Big data configuration is originally put into a virtual machine and is run by step 1, and the virtual machine is saved as a mirror later
As file；Big data configuration originally includes: operating system version configuration and environment configurations, the environment configurations of user and big
The configuration parameter of data；
Image file is uploaded to mirror image warehouse server by step 2, and generates corresponding timestamp, the mirror image warehouse
Server is used for the storage of image file, manages and be issued to the operational administrative of each node of big data；
Step 3, mirror image warehouse server are connected to after the configuration version more new command of cluster management personnel, get correspondence
Image file, the image file is put into the virtual machine of configuration mirroring verifies later, if by verifying, then will
The image file issues, and otherwise searches corresponding image file again, until by being issued again after verifying, and each
The iteration that the configuration version of entire cluster can be realized in host node starting associated virtual machine process updates.
Big data platform version management of the present invention and iteratively faster deployment system, it is characterised in that: include:
Big data configuration is originally put into a virtual machine and runs, later saves as the virtual machine by generation module
One image file；Big data configuration originally include: operating system version configuration and environment configurations, the environment configurations of user with
And the configuration parameter of big data；
Image file is uploaded to mirror image warehouse server, and generates corresponding timestamp by uploading module, the mirror image storehouse
Library server is used for the storage of image file, manages and be issued to the operational administrative of each node of big data；
Update module, mirror image warehouse server are connected to after the configuration version more new command of cluster management personnel, get
The image file is put into the virtual machine of configuration mirroring verifies later by corresponding image file, if by verifying,
Then the image file is issued, otherwise searches corresponding image file again, until by being issued again after verifying, and
The iteration for the configuration version that entire cluster can be realized in each host node starting associated virtual machine process updates.
Can be embodied as through the invention each big data distribute rationally production one virtual image file, big data into
Journey operates in the unified optimization and upgrading configuration that entire cluster can be realized in the container that this image file creates.Realize big data
The iteratively faster of the management of platform image versioned and version optimizes version, changes the office for being previously required to management a pile complex script
Face, allows operation management flattening, changes traditional distributed script execution, becomes distributed container and be substituted.
Scenario-frame: the centralized management of the mirror image, configuration mirroring file of big data configuration, big data configure the peace of iteration
Full test verifying, big data configuration upgrading iteration switching.
Key step:
A: the virtual image of big data configuration:
The virtualization image file of one lightweight of this generation is configured for the big data of each, this mirror image includes below
Project: the operating system version configuration of operating system and environment configurations, the environment configurations of user, the configuration parameter of big data.
The kernel bottom layer code of operating system and the business datum of big data will use the dependency number of direct host
According to so that each version of big data configuration is realized virtualization mirror image.
B: the centralized management of configuration mirroring file:
The virtual image of big data configuration uploads to virtualization mirror image warehouse server, and thus server carries out depositing for mirror image
Storage, management and the operational administrative for being issued to each node of big data.
The version management of large data sets group configuration mirror image: the logical configuration mirroring to big data cluster carries out timestamp and version
Management combines (when generating mirror image, can all generate a corresponding version number), realizes the iteration management of version.
C: big data configures the safety test verifying of iteration:
Virtualization mirror image warehouse server can big data configuration file to each version, can be directly in configuration mirroring
Virtual machine environment in go verifying configure correctness, configuration whether it is risky lead to problems such as cluster appearance can not start, if
It is equipped with risk, then needs to regenerate image file, the safety of big data configuration is ensured by the above test operation.
D: big data configuration upgrading iteration switching:
Virtualization mirror image warehouse server is connected to after the configuration version more new command of cluster management personnel, can be started pair
The configuration mirroring file respective version of the entire cluster of big data issues, and each host node start associated virtual machine into
The iteration that the configuration version of entire cluster can be realized in journey updates.
Although specific embodiments of the present invention have been described above, those familiar with the art should be managed
Solution, we are merely exemplary described specific embodiment, rather than for the restriction to the scope of the present invention, it is familiar with this
The technical staff in field should be covered of the invention according to modification and variation equivalent made by spirit of the invention
In scope of the claimed protection.
Claims (8)
1. a kind of big data platform version management and iteratively faster dispositions method, it is characterised in that: include:
Big data configuration is originally put into a virtual machine and is run by step 1, and the virtual machine is saved as mirror image text later
Part；
Image file is uploaded to mirror image warehouse server by step 2, and generates corresponding timestamp；
Step 3, mirror image warehouse server are connected to after the configuration version more new command of cluster management personnel, by corresponding big data
The image file of entire cluster issues, and entire cluster can be realized in each host node starting associated virtual machine process
The iteration of configuration version updates.
2. a kind of big data platform version management according to claim 1 and iteratively faster dispositions method, it is characterised in that:
The big data configuration originally includes: operating system version configuration and environment configurations, the environment configurations of user and big data
Configuration parameter.
3. a kind of big data platform version management according to claim 1 and iteratively faster dispositions method, it is characterised in that:
The step 3 is further specifically: mirror image warehouse server is connected to after the configuration version more new command of cluster management personnel, is obtained
Corresponding image file is got, the image file is put into the virtual machine of configuration mirroring verifies later, if passing through
Verifying, then issue the image file, otherwise search corresponding image file again, until by carrying out down again after verifying
Hair, and updated in the iteration that the configuration version of entire cluster can be realized in each host node starting associated virtual machine process.
4. a kind of big data platform version management according to claim 1 and iteratively faster dispositions method, it is characterised in that:
The mirror image warehouse server is used for the storage of image file, manages and be issued to the operational administrative of each node of big data.
5. a kind of big data platform version management and iteratively faster deployment system, it is characterised in that: include:
Big data configuration is originally put into a virtual machine and runs, the virtual machine is saved as a mirror later by generation module
As file；
Image file is uploaded to mirror image warehouse server, and generates corresponding timestamp by uploading module；
Update module, mirror image warehouse server are connected to after the configuration version more new command of cluster management personnel, will be corresponding big
The image file of the entire cluster of data issues, and entire collection can be realized in each host node starting associated virtual machine process
The iteration of the configuration version of group updates.
6. a kind of big data platform version management according to claim 5 and iteratively faster deployment system, it is characterised in that:
The big data configuration originally includes: operating system version configuration and environment configurations, the environment configurations of user and big data
Configuration parameter.
7. a kind of big data platform version management according to claim 5 and iteratively faster deployment system, it is characterised in that:
The update module is further specifically: mirror image warehouse server be connected to cluster management personnel configuration version more new command it
Afterwards, corresponding image file is got, the image file is put into the virtual machine of configuration mirroring verifies later, if
By verifying, then the image file is issued, otherwise search corresponding image file again, until by being carried out again after verifying
It issues, and the iteration of the configuration version of entire cluster can be realized more in each host node starting associated virtual machine process
Newly.
8. a kind of big data platform version management according to claim 5 and iteratively faster deployment system, it is characterised in that:
The mirror image warehouse server is used for the storage of image file, manages and be issued to the operational administrative of each node of big data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811047946.8A CN109240716B (en) | 2018-09-10 | 2018-09-10 | Big data platform version management and rapid iterative deployment method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811047946.8A CN109240716B (en) | 2018-09-10 | 2018-09-10 | Big data platform version management and rapid iterative deployment method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109240716A true CN109240716A (en) | 2019-01-18 |
| CN109240716B CN109240716B (en) | 2022-10-25 |
Family
ID=65067613
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811047946.8A Active CN109240716B (en) | 2018-09-10 | 2018-09-10 | Big data platform version management and rapid iterative deployment method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109240716B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110647580A (en) * | 2019-09-05 | 2020-01-03 | 南京邮电大学 | Distributed container cluster mirror image management main node, slave node, system and method |
| CN111078302A (en) * | 2019-11-19 | 2020-04-28 | 许昌许继软件技术有限公司 | Automatic deployment method and terminal of distribution network monitoring platform system |
| CN112463205A (en) * | 2020-11-24 | 2021-03-09 | 史传波 | AI and big data based application program management method and artificial intelligence server |
| CN114968274A (en) * | 2022-07-29 | 2022-08-30 | 之江实验室 | Method and system for automatically and rapidly deploying front-end processor based on gray release |
| CN117240711A (en) * | 2023-09-15 | 2023-12-15 | 合芯科技有限公司 | Automatic updating method, device and equipment for cluster management tool configuration file |
Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102065411A (en) * | 2010-10-22 | 2011-05-18 | 上海交通大学 | Method for dynamically updating wireless sensor network |
| CN103051714A (en) * | 2012-12-24 | 2013-04-17 | 河海大学 | Implementation method of water conservation cloud platform |
| CN104102527A (en) * | 2014-07-10 | 2014-10-15 | 国云科技股份有限公司 | Method for updating software of virtual machine |
| CN104463012A (en) * | 2014-11-24 | 2015-03-25 | 东软集团股份有限公司 | Virtual machine image file exporting and importing method and device |
| CN105187252A (en) * | 2015-09-28 | 2015-12-23 | 浪潮(北京)电子信息产业有限公司 | Method and system for batch deployment of customized systems |
| US20160092261A1 (en) * | 2014-09-30 | 2016-03-31 | Sangfor Technologies Company Limited | Method and system for physical computer system virtualization |
| CN105786691A (en) * | 2014-12-25 | 2016-07-20 | 重庆重邮信科通信技术有限公司 | Automatic integration testing device, method and system of mobile terminal |
| CN106199696A (en) * | 2016-06-29 | 2016-12-07 | 中国石油天然气股份有限公司 | Earthquake data processing system and method |
| CN106445643A (en) * | 2016-11-14 | 2017-02-22 | 上海云轴信息科技有限公司 | Method and device for cloning and updating virtual machine |
| CN106528269A (en) * | 2016-11-08 | 2017-03-22 | 西安电子科技大学 | Light weight virtual machine access control system and method |
| CN106897093A (en) * | 2017-02-24 | 2017-06-27 | 郑州云海信息技术有限公司 | A kind of dispositions method and device of windows operating systems |
| WO2017166785A1 (en) * | 2016-04-01 | 2017-10-05 | 华为技术有限公司 | Method for deploying virtual machines, host machine, and system |
| CN107395762A (en) * | 2017-08-30 | 2017-11-24 | 四川长虹电器股份有限公司 | A kind of application service based on Docker containers accesses system and method |
| CN107783816A (en) * | 2016-08-31 | 2018-03-09 | 阿里巴巴集团控股有限公司 | The method and device that creation method and device, the big data cluster of virtual machine create |
| CN107807838A (en) * | 2016-09-08 | 2018-03-16 | 阿里巴巴集团控股有限公司 | Virtual machine processing method, device and equipment |
| WO2018086437A1 (en) * | 2016-11-09 | 2018-05-17 | 华为技术有限公司 | Accelerator loading method and system, and accelerator loading apparatus |
-
2018
- 2018-09-10 CN CN201811047946.8A patent/CN109240716B/en active Active
Patent Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102065411A (en) * | 2010-10-22 | 2011-05-18 | 上海交通大学 | Method for dynamically updating wireless sensor network |
| CN103051714A (en) * | 2012-12-24 | 2013-04-17 | 河海大学 | Implementation method of water conservation cloud platform |
| CN104102527A (en) * | 2014-07-10 | 2014-10-15 | 国云科技股份有限公司 | Method for updating software of virtual machine |
| US20160092261A1 (en) * | 2014-09-30 | 2016-03-31 | Sangfor Technologies Company Limited | Method and system for physical computer system virtualization |
| CN104463012A (en) * | 2014-11-24 | 2015-03-25 | 东软集团股份有限公司 | Virtual machine image file exporting and importing method and device |
| CN105786691A (en) * | 2014-12-25 | 2016-07-20 | 重庆重邮信科通信技术有限公司 | Automatic integration testing device, method and system of mobile terminal |
| CN105187252A (en) * | 2015-09-28 | 2015-12-23 | 浪潮(北京)电子信息产业有限公司 | Method and system for batch deployment of customized systems |
| CN107291525A (en) * | 2016-04-01 | 2017-10-24 | 华为技术有限公司 | A kind of method, host and system for disposing virtual machine |
| WO2017166785A1 (en) * | 2016-04-01 | 2017-10-05 | 华为技术有限公司 | Method for deploying virtual machines, host machine, and system |
| CN106199696A (en) * | 2016-06-29 | 2016-12-07 | 中国石油天然气股份有限公司 | Earthquake data processing system and method |
| CN107783816A (en) * | 2016-08-31 | 2018-03-09 | 阿里巴巴集团控股有限公司 | The method and device that creation method and device, the big data cluster of virtual machine create |
| CN107807838A (en) * | 2016-09-08 | 2018-03-16 | 阿里巴巴集团控股有限公司 | Virtual machine processing method, device and equipment |
| CN106528269A (en) * | 2016-11-08 | 2017-03-22 | 西安电子科技大学 | Light weight virtual machine access control system and method |
| WO2018086437A1 (en) * | 2016-11-09 | 2018-05-17 | 华为技术有限公司 | Accelerator loading method and system, and accelerator loading apparatus |
| CN106445643A (en) * | 2016-11-14 | 2017-02-22 | 上海云轴信息科技有限公司 | Method and device for cloning and updating virtual machine |
| CN106897093A (en) * | 2017-02-24 | 2017-06-27 | 郑州云海信息技术有限公司 | A kind of dispositions method and device of windows operating systems |
| CN107395762A (en) * | 2017-08-30 | 2017-11-24 | 四川长虹电器股份有限公司 | A kind of application service based on Docker containers accesses system and method |
Non-Patent Citations (2)
| Title |
|---|
| 刘晔: "云计算中虚似机镜像管理研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
| 戴青等: "云计算平台下基于内核的虚拟机使用评估", 《科技致富向导》 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110647580A (en) * | 2019-09-05 | 2020-01-03 | 南京邮电大学 | Distributed container cluster mirror image management main node, slave node, system and method |
| CN110647580B (en) * | 2019-09-05 | 2022-06-10 | 南京邮电大学 | Distributed container cluster mirror image management main node, slave node, system and method |
| CN111078302A (en) * | 2019-11-19 | 2020-04-28 | 许昌许继软件技术有限公司 | Automatic deployment method and terminal of distribution network monitoring platform system |
| CN112463205A (en) * | 2020-11-24 | 2021-03-09 | 史传波 | AI and big data based application program management method and artificial intelligence server |
| CN112463205B (en) * | 2020-11-24 | 2021-07-30 | 青岛日日顺乐信云科技有限公司 | AI and big data based application program management method and artificial intelligence server |
| CN114968274A (en) * | 2022-07-29 | 2022-08-30 | 之江实验室 | Method and system for automatically and rapidly deploying front-end processor based on gray release |
| CN117240711A (en) * | 2023-09-15 | 2023-12-15 | 合芯科技有限公司 | Automatic updating method, device and equipment for cluster management tool configuration file |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109240716B (en) | 2022-10-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109240716A (en) | A kind of big data platform version management and iteratively faster dispositions method and system | |
| US11797479B2 (en) | Integrating object-based data integration tool with a version control system in centralized and decentralized environments | |
| CN112585919B (en) | Method for managing application configuration state by using cloud-based application management technology | |
| US20200183896A1 (en) | Upgrade of heterogeneous multi-instance database clusters | |
| JP6490633B2 (en) | Systems and methods for private cloud computing | |
| CN114008605A (en) | Big data application lifecycle management | |
| CN109885316B (en) | Hdfs-hbase deployment method and device based on kubernetes | |
| CN111061817B (en) | Adaptive business construction system, method and computer readable medium | |
| US9558474B2 (en) | Systems management operational workflow templates | |
| CN106534291B (en) | Voltage monitoring method based on big data processing | |
| EP3258393A1 (en) | Data revision control in large-scale data analytic systems | |
| US20150020059A1 (en) | Migration services for systems | |
| WO2011014827A1 (en) | System and method to uniformly manage operational life cycles and service levels | |
| US11789912B2 (en) | Data analytic systems | |
| US20220391124A1 (en) | Software Lifecycle Management For A Storage System | |
| JP2023537083A (en) | Method and apparatus for updating cloud platform | |
| WO2022226400A1 (en) | Transition manager system | |
| CN103678240A (en) | Method and system for modularized management of distributed computation | |
| CN105678118A (en) | Generation method and device for software versions with digital certificate | |
| EP4296861A1 (en) | System and method for cloud infrastructure test automation | |
| US20230393825A1 (en) | Automated software deployment techniques | |
| US20220335318A1 (en) | Dynamic anomaly forecasting from execution logs | |
| JP2023541530A (en) | Cloud platform update method and device | |
| JP7177335B2 (en) | Information processing method, information processing device, and information processing program | |
| CN111897794A (en) | Database maintenance method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 350000 21 / F, building 5, f District, Fuzhou Software Park, 89 software Avenue, Gulou District, Fuzhou City, Fujian Province Applicant after: FUJIAN SINOREGAL SOFTWARE CO.,LTD. Address before: Floor 20-21, building 5, area F, Fuzhou Software Park, 89 software Avenue, Gulou District, Fuzhou City, Fujian Province 350000 Applicant before: FUJIAN SINOREGAL SOFTWARE CO.,LTD. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |