CN109240716B - Big data platform version management and rapid iterative deployment method and system - Google Patents
Big data platform version management and rapid iterative deployment method and system Download PDFInfo
- Publication number
- CN109240716B CN109240716B CN201811047946.8A CN201811047946A CN109240716B CN 109240716 B CN109240716 B CN 109240716B CN 201811047946 A CN201811047946 A CN 201811047946A CN 109240716 B CN109240716 B CN 109240716B
- Authority
- CN
- China
- Prior art keywords
- mirror image
- image file
- big data
- version
- configuration
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
- G06F8/61—Installation
- G06F8/63—Image based installation; Cloning; Build to order
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/70—Software maintenance or management
- G06F8/71—Version control; Configuration management
Abstract
The invention provides a big data platform version management and rapid iterative deployment method, which comprises the steps of putting a big data configuration version into a virtual machine for operation, and then storing the virtual machine as a mirror image file; uploading the mirror image file to a mirror image warehouse server, and generating a corresponding timestamp; after receiving a configuration version updating instruction of a cluster manager, a mirror image warehouse server issues a corresponding image file of the whole big data cluster, and starts a relevant virtual machine process at each host machine node to realize iterative updating of the configuration version of the whole cluster; the invention also provides a large data platform version management and rapid iteration deployment system, and the upgrade iteration of the management configuration of the whole cluster is achieved through the iteration of the image file.
Description
Technical Field
The invention relates to a large data platform version management and rapid iteration deployment method and system.
Background
The big data platform has a set of big data software optimized configuration parameters for each service scene, if the existing cluster management technology is used, a large number of scripts need to be written, and then automatic configuration scripts are executed on hundreds of servers.
Disclosure of Invention
The invention aims to solve the technical problem of providing a method and a system for managing and quickly deploying versions of a big data platform in an iterative manner, so that each version of the big data platform is made into a virtual machine image file, and the upgrade iteration of the management configuration of the whole cluster is achieved through the iteration of the image file.
One of the present invention is realized by: a big data platform version management and rapid iteration deployment method comprises the following steps:
step 1, placing a big data configuration version into a virtual machine for running, and then saving the virtual machine as a mirror image file;
step 2, uploading the mirror image file to a mirror image warehouse server, and generating a corresponding timestamp;
and 3, after receiving a configuration version updating instruction of a cluster manager, the mirror image warehouse server issues a corresponding mirror image file of the whole big data cluster, and starts a related virtual machine process at each host machine node to realize iterative updating of the configuration version of the whole cluster.
Further, the big data configuration version comprises: the operating system version configuration and the environment configuration, the user environment configuration and the configuration parameters of the big data.
Further, the step 3 is further specifically: after receiving a configuration version updating instruction of cluster management personnel, the mirror warehouse server acquires a corresponding mirror image file, then puts the mirror image file into a virtual machine configured with a mirror image for verification, if the mirror image file passes the verification, the mirror image file is issued, otherwise, the corresponding mirror image file is searched again, the mirror image file is issued until the mirror image file passes the verification, and the iterative updating of the configuration version of the whole cluster can be realized by starting a related virtual machine process at each host machine node.
Further, the mirror image warehouse server is used for storing and managing mirror image files and operating and managing the mirror image files issued to each node of the big data.
The second invention is realized by the following steps: a big data platform version management and rapid iteration deployment system comprises:
the generation module is used for putting the big data configuration version into a virtual machine for running and then storing the virtual machine as a mirror image file;
the uploading module uploads the mirror image file to the mirror image warehouse server and generates a corresponding timestamp;
and after receiving a configuration version updating instruction of a cluster manager, the mirror image warehouse server issues a corresponding mirror image file of the whole big data cluster, and starts a related virtual machine process at each host machine node to realize iterative updating of the configuration version of the whole cluster.
Further, the big data configuration version comprises: the operating system version configuration and the environment configuration, the user environment configuration and the configuration parameters of the big data.
Further, the update module further specifically includes: after receiving a configuration version updating instruction of cluster management personnel, the mirror image warehouse server acquires a corresponding mirror image file, then puts the mirror image file into a virtual machine of a configuration mirror image for verification, if the configuration version updating instruction passes the verification, the mirror image file is issued, otherwise, the corresponding mirror image file is searched again, the mirror image file is issued again until the verification passes, and the iterative updating of the configuration version of the whole cluster can be realized by starting a related virtual machine process at each host machine node.
Further, the mirror image warehouse server is used for storing and managing mirror image files and operating and managing the mirror image files issued to each node of the big data.
The invention has the following advantages:
1) Mirroring of big data configuration is consistent with configuration: the mirroring of the big data configuration file is realized through the lightweight virtualization, the version is uniformly managed and issued through the virtualized mirror warehouse, the consistency of the whole big data configuration file is realized, and the problem of version disorder is avoided.
2) And (3) safety verification test of the big data configuration version: the correctness of configuration is verified in the virtual machine environment of each version performed by the virtual mirror image warehouse server, and the problem of abnormal big data clusters caused by abnormal configuration is avoided.
3) Fast iteration of large data configuration versions: by the aid of the virtualized mirror image warehouse server, the configuration mirror image version can be rapidly issued to all large data cluster node servers, and rapid iteration of the cluster configuration version is achieved.
Drawings
The invention will be further described with reference to the following examples with reference to the accompanying drawings.
FIG. 1 is a flow chart of the method of the present invention.
Detailed Description
The invention discloses a big data platform version management and rapid iteration deployment method, which comprises the following steps:
step 1, placing a big data configuration version into a virtual machine for running, and then saving the virtual machine as a mirror image file; the big data configuration version comprises: version configuration and environment configuration of an operating system, environment configuration of a user and configuration parameters of big data;
step 2, uploading the mirror image file to a mirror image warehouse server, and generating a corresponding timestamp, wherein the mirror image warehouse server is used for storing and managing the mirror image file and issuing the mirror image file to the operation management of each node of the big data;
and 3, after receiving a configuration version updating instruction of cluster management personnel, the mirror warehouse server acquires a corresponding mirror image file, then puts the mirror image file into a virtual machine configured with a mirror image for verification, if the mirror image file passes the verification, the mirror image file is issued, otherwise, the corresponding mirror image file is searched again, the mirror image file is issued until the mirror image file passes the verification, and the iterative updating of the configuration version of the whole cluster can be realized by starting a related virtual machine process at each host machine node.
The invention discloses a big data platform version management and rapid iterative deployment system, which comprises:
the generation module is used for putting the big data configuration version into a virtual machine for running and then storing the virtual machine as a mirror image file; the big data configuration version comprises: version configuration and environment configuration of an operating system, environment configuration of a user and configuration parameters of big data;
the uploading module uploads the mirror image file to a mirror image warehouse server and generates a corresponding timestamp, and the mirror image warehouse server is used for storing and managing the mirror image file and issuing the mirror image file to the operation management of each node of the big data;
and the updating module is used for acquiring a corresponding image file after the image warehouse server receives a configuration version updating instruction of a cluster manager, then putting the image file into a virtual machine of a configuration image for verification, issuing the image file if the verification is passed, otherwise searching the corresponding image file again until the verification is passed, issuing the image file again, and starting a related virtual machine process at each host machine node to realize iterative updating of the configuration version of the whole cluster.
The invention can realize that a virtual image file is made for each big data optimization configuration, and the big data process runs in a container created by the image file to realize the uniform optimization upgrading configuration of the whole cluster. The large data platform mirror image versioning management and the quick iteration optimization version of the version are achieved, the situation that a pile of complex scripts need to be managed in the prior art is changed, the operation and maintenance management is flattened, the traditional distributed script execution is changed, and the distributed container is replaced.
The scheme structure is as follows: mirroring of big data configuration, centralized management of configuration mirror image files, safety test verification of big data configuration iteration and upgrading iteration switching of big data configuration.
The method mainly comprises the following steps:
a, virtual mirroring of big data configuration:
generating a lightweight virtualized image file for each big data configuration version, wherein the image comprises the following items: the operating system version configuration and environment configuration of the operating system, the environment configuration of the user and the configuration parameters of the big data.
The kernel bottom layer code of the operating system and the business data of the big data use the related data of the host machine directly, thereby realizing the virtualization mirror image of each version of the big data configuration.
B, centralized management of configuration mirror image files:
and uploading the virtual mirror image configured by the big data to a virtualized mirror image warehouse server, so that the server performs storage and management of the mirror image and operation management issued to each node of the big data.
Version management of big data cluster configuration mirror image: iterative management of versions is achieved by combining timestamp management and version management of configuration images of a big data cluster (when images are generated, a corresponding version number is generated).
And C, safety test verification of big data configuration iteration:
the configuration correctness of the big data of each version can be directly verified in the virtual machine environment of the configuration mirror image for the virtualized mirror image warehouse server, the cluster cannot be started due to the fact that whether the configuration is risky or not, and the like.
D, large data configuration upgrading iteration switching:
after receiving a configuration version updating instruction of a cluster manager, the virtualization mirror image warehouse server can start issuing corresponding versions of configuration mirror image files of the whole big data cluster, and start a related virtual machine process at each host machine node to realize iterative updating of the configuration versions of the whole cluster.
Although specific embodiments of the invention have been described above, it will be understood by those skilled in the art that the specific embodiments described are illustrative only and are not limiting upon the scope of the invention, and that equivalent modifications and variations can be made by those skilled in the art without departing from the spirit of the invention, which is to be limited only by the appended claims.
Claims (6)
1. A big data platform version management and rapid iteration deployment method is characterized in that: the method comprises the following steps:
step 1, putting a big data configuration version into a virtual machine for operation, and then saving the virtual machine as an image file, wherein the big data configuration version comprises the following steps: the method comprises the following steps of (1) configuring an operating system version and an environment, configuring the environment of a user and configuring parameters of big data software;
step 2, uploading the mirror image file to a mirror image warehouse server, and generating a corresponding timestamp;
and 3, after receiving a configuration version updating instruction of a cluster manager, the mirror image warehouse server issues a corresponding image file of the whole big data cluster, and starts a relevant virtual machine process at each host machine node to realize iterative updating of the configuration version of the whole cluster.
2. The big data platform version management and rapid iterative deployment method according to claim 1, characterized in that: the step 3 is further specifically: after receiving a configuration version updating instruction of cluster management personnel, the mirror warehouse server acquires a corresponding mirror image file, then puts the mirror image file into a virtual machine configured with a mirror image for verification, if the mirror image file passes the verification, the mirror image file is issued, otherwise, the corresponding mirror image file is searched again, the mirror image file is issued until the mirror image file passes the verification, and the iterative updating of the configuration version of the whole cluster can be realized by starting a related virtual machine process at each host machine node.
3. The big data platform version management and rapid iterative deployment method according to claim 1, characterized in that: the mirror image warehouse server is used for storing and managing mirror image files and operating and managing the mirror image files issued to each node of the big data.
4. A big data platform version management and rapid iteration deployment system is characterized in that: the method comprises the following steps:
the generation module is used for putting the big data configuration version into a virtual machine for operation, and then storing the virtual machine as an image file, wherein the big data configuration version comprises the following steps: the method comprises the following steps of (1) configuring an operating system version and an environment, configuring the environment of a user and configuring parameters of big data software;
the uploading module uploads the mirror image file to the mirror image warehouse server and generates a corresponding timestamp;
and after receiving a configuration version updating instruction of a cluster manager, the mirror image warehouse server issues a corresponding image file of the whole big data cluster, and starts a relevant virtual machine process at each host machine node to realize iterative updating of the configuration version of the whole cluster.
5. The big data platform version management and rapid iterative deployment system according to claim 4, wherein: the update module further specifically includes: after receiving a configuration version updating instruction of cluster management personnel, the mirror warehouse server acquires a corresponding mirror image file, then puts the mirror image file into a virtual machine configured with a mirror image for verification, if the mirror image file passes the verification, the mirror image file is issued, otherwise, the corresponding mirror image file is searched again, the mirror image file is issued until the mirror image file passes the verification, and the iterative updating of the configuration version of the whole cluster can be realized by starting a related virtual machine process at each host machine node.
6. The big data platform version management and rapid iterative deployment system according to claim 4, wherein: the mirror image warehouse server is used for storing and managing mirror image files and operating and managing the mirror image files issued to each node of the big data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811047946.8A CN109240716B (en) | 2018-09-10 | 2018-09-10 | Big data platform version management and rapid iterative deployment method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811047946.8A CN109240716B (en) | 2018-09-10 | 2018-09-10 | Big data platform version management and rapid iterative deployment method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109240716A CN109240716A (en) | 2019-01-18 |
| CN109240716B true CN109240716B (en) | 2022-10-25 |
Family
ID=65067613
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811047946.8A Active CN109240716B (en) | 2018-09-10 | 2018-09-10 | Big data platform version management and rapid iterative deployment method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109240716B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110647580B (en) * | 2019-09-05 | 2022-06-10 | 南京邮电大学 | Distributed container cluster mirror image management main node, slave node, system and method |
| CN111078302B (en) * | 2019-11-19 | 2023-08-11 | 许昌许继软件技术有限公司 | Automatic deployment method and terminal of distribution network monitoring platform system |
| CN112463205B (en) * | 2020-11-24 | 2021-07-30 | 青岛日日顺乐信云科技有限公司 | AI and big data based application program management method and artificial intelligence server |
| CN114968274B (en) * | 2022-07-29 | 2022-11-08 | 之江实验室 | Method and system for automatically and rapidly deploying front-end processor based on gray release |
| CN117240711A (en) * | 2023-09-15 | 2023-12-15 | 合芯科技有限公司 | Automatic updating method, device and equipment for cluster management tool configuration file |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102065411A (en) * | 2010-10-22 | 2011-05-18 | 上海交通大学 | Method for dynamically updating wireless sensor network |
| CN105187252A (en) * | 2015-09-28 | 2015-12-23 | 浪潮(北京)电子信息产业有限公司 | Method and system for batch deployment of customized systems |
| CN105786691A (en) * | 2014-12-25 | 2016-07-20 | 重庆重邮信科通信技术有限公司 | Automatic integration testing device, method and system of mobile terminal |
| CN106528269A (en) * | 2016-11-08 | 2017-03-22 | 西安电子科技大学 | Light weight virtual machine access control system and method |
| CN106897093A (en) * | 2017-02-24 | 2017-06-27 | 郑州云海信息技术有限公司 | A kind of dispositions method and device of windows operating systems |
| WO2017166785A1 (en) * | 2016-04-01 | 2017-10-05 | 华为技术有限公司 | Method for deploying virtual machines, host machine, and system |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103051714A (en) * | 2012-12-24 | 2013-04-17 | 河海大学 | Implementation method of water conservation cloud platform |
| CN104102527A (en) * | 2014-07-10 | 2014-10-15 | 国云科技股份有限公司 | Method for updating software of virtual machine |
| CN104298559B (en) * | 2014-09-30 | 2018-03-02 | 深信服科技股份有限公司 | The method and device of physical host system virtualization |
| CN104463012A (en) * | 2014-11-24 | 2015-03-25 | 东软集团股份有限公司 | Virtual machine image file exporting and importing method and device |
| CN106199696B (en) * | 2016-06-29 | 2019-01-18 | 中国石油天然气股份有限公司 | Earthquake data processing system and method |
| CN107783816A (en) * | 2016-08-31 | 2018-03-09 | 阿里巴巴集团控股有限公司 | The method and device that creation method and device, the big data cluster of virtual machine create |
| CN107807838B (en) * | 2016-09-08 | 2021-11-23 | 阿里巴巴集团控股有限公司 | Virtual machine processing method, device and equipment |
| CN108073423B (en) * | 2016-11-09 | 2020-01-17 | 华为技术有限公司 | Accelerator loading method and system and accelerator loading device |
| CN106445643B (en) * | 2016-11-14 | 2019-10-22 | 上海云轴信息科技有限公司 | It clones, the method and apparatus of upgrading virtual machine |
| CN107395762A (en) * | 2017-08-30 | 2017-11-24 | 四川长虹电器股份有限公司 | A kind of application service based on Docker containers accesses system and method |
-
2018
- 2018-09-10 CN CN201811047946.8A patent/CN109240716B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102065411A (en) * | 2010-10-22 | 2011-05-18 | 上海交通大学 | Method for dynamically updating wireless sensor network |
| CN105786691A (en) * | 2014-12-25 | 2016-07-20 | 重庆重邮信科通信技术有限公司 | Automatic integration testing device, method and system of mobile terminal |
| CN105187252A (en) * | 2015-09-28 | 2015-12-23 | 浪潮(北京)电子信息产业有限公司 | Method and system for batch deployment of customized systems |
| WO2017166785A1 (en) * | 2016-04-01 | 2017-10-05 | 华为技术有限公司 | Method for deploying virtual machines, host machine, and system |
| CN106528269A (en) * | 2016-11-08 | 2017-03-22 | 西安电子科技大学 | Light weight virtual machine access control system and method |
| CN106897093A (en) * | 2017-02-24 | 2017-06-27 | 郑州云海信息技术有限公司 | A kind of dispositions method and device of windows operating systems |
Non-Patent Citations (2)
| Title |
|---|
| 云计算中虚似机镜像管理研究;刘晔;《中国优秀硕士学位论文全文数据库 信息科技辑》;20160315;I137-47 * |
| 云计算平台下基于内核的虚拟机使用评估;戴青等;《科技致富向导》;20130720(第20期);248-249 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109240716A (en) | 2019-01-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109240716B (en) | Big data platform version management and rapid iterative deployment method and system | |
| CN107515776B (en) | Method for upgrading service continuously, node to be upgraded and readable storage medium | |
| CN108924217B (en) | Automatic deployment method of distributed cloud system | |
| US20220229649A1 (en) | Conversion and restoration of computer environments to container-based implementations | |
| US9280554B2 (en) | Using confidence values for synchronizing file systems | |
| CN109885316B (en) | Hdfs-hbase deployment method and device based on kubernetes | |
| US10430204B2 (en) | System and method for cloud provisioning and application deployment | |
| US10762075B2 (en) | Database interface agent for a tenant-based upgrade system | |
| US9471365B2 (en) | Techniques for performing virtual machine software upgrades using virtual disk swapping | |
| EP3387528B1 (en) | Updating dependent services | |
| US10585691B2 (en) | Distribution system, computer, and arrangement method for virtual machine | |
| US9405630B2 (en) | Methods and apparatus to perform site recovery of a virtual data center | |
| US9792321B2 (en) | Online database migration | |
| KR102047216B1 (en) | Replaying jobs at a secondary location of a service | |
| CN103226493B (en) | The dispositions method and system of multi-operation system service | |
| US10635473B2 (en) | Setting support program, setting support method, and setting support device | |
| CN108089913B (en) | Virtual machine deployment method of super-fusion system | |
| CN102937909B (en) | A kind of method of disposing and upgrading linux system | |
| US20080051921A1 (en) | Method for modifying configuration of business system | |
| CN107783816A (en) | The method and device that creation method and device, the big data cluster of virtual machine create | |
| US9959157B1 (en) | Computing instance migration | |
| US10338910B2 (en) | Multi-tenant upgrading | |
| CN117099079A (en) | System configuration freezing and change management of services deployed via continuous delivery configured on a data center in a cloud platform | |
| US20230393825A1 (en) | Automated software deployment techniques | |
| CN115145604A (en) | Containerized electric power marketing system deployment method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 350000 21 / F, building 5, f District, Fuzhou Software Park, 89 software Avenue, Gulou District, Fuzhou City, Fujian Province Applicant after: FUJIAN SINOREGAL SOFTWARE CO.,LTD. Address before: Floor 20-21, building 5, area F, Fuzhou Software Park, 89 software Avenue, Gulou District, Fuzhou City, Fujian Province 350000 Applicant before: FUJIAN SINOREGAL SOFTWARE CO.,LTD. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |